Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ON THE IMPACT OF SERVICE-ORIENTED PATTERNS ON SOFTWARE EVOLVABILITY: A CONTROLLED EXPERIMENT AND METRIC-BASED ANALYSIS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is very well written, organized and clear to read. Text is concise and cohesive. Moreover, authors did a good job in keeping the literature review closer to the scope of the paper. Some minor feedback on how to improve the tables, and include more visual presentation of the quantitative are discussed in the detailed review below.
Experimental design: There are two main problems with the design, further detailed below in the Comments to author section. Both problems are connected to ethical guidelines and data collection practices. Problem 1: Authors are not clear on the ethical guidelines that they follow. The text is not clear, whether the participants were given the option to opt-out. Even if the experiment is not graded, it seems that they "needed to participate" in order to pass a course. Even if they needed to participate, participants should be given the option to participate in the experiment (i.e., to complete the course), but ask that their data is not used for research or is not published. Problem 2: Collection of personal data. Authors collect and publish very personal data from the participants. Particularly for age and gender this would risk the anonymization of the data, given that some combinations of those are unique. Going through the paper, gender and age have no weigh in the conclusions. The data is used to claim consistency between both groups of participants, but authors can use the other data collected to argue for that (other rows of Table 6). There are also some concerns with the construct used for gender (0 for females, 1 for males), which violates recent guidelines to collect gender data (established from the field of gender studies, medicine and psychology, detailed in comments below).

Below, I present the detailed parts of the paper where I found both problems, along with a few suggestions for authors to consider.
Validity of the findings: The statistical analysis performed has several issues. Authors neglected to check assumptions of the test, and chose unsuitable test and correlation scores for their analysis. These checks have been widely discussed in the Empirical Software Engineering field. This is reflected on a lack of an analysis on conclusion validity threats. The authors did a great job with their data package, that allowed me to rerun all tests and even run with the right tests. Fortunately, the conclusions and findings remain overall the same, but the text should reflect and motivate these choices better. Moreover, the analysis with metrics (Section 5) is also subjective to validity threats (already mentioned by authors in the text) that should be further discussed.

Below I provide suggestions on how to fix both the statistical analysis and the metrics discussion.
Additional comments: Summary of the paper:
The paper evaluates the effect of four service-based patterns in an experiment with 62 students. The patterns are evaluated in terms of effectiveness and efficiency related to modifiability of the system. The analysis reveal that the use of patterns does not show a significant difference in modifiability (better or worse), except for a specific difference on one of the tasks performed (Task 3). Authors complement this evaluation doing an analysis on internal quality metrics of the system.

Strengths:
* Great effort in controlling the main threats when designing and running the experiment.
* Providing all data available enabled a thorough validation of the findings and the paper. This is certainly commendable and hopefully referenced by future empirical studies.
* Good contribution on the lessons learned (Section 8) to foster reproduction of the experiment and encourage more quantitative studies in the field.

Weaknesses:
* There are very pertinent open questions about the ethical protocols used by authors that require clarifications.
* Collection of personal data from participants is not properly motivated or justified.
* The statistical analysis were based on the wrong tests, introducing conclusion validity threats.
* The reporting and packaging of the experiment can be improved.

Summary of the review:
First and foremost, authors should be commended on the effort in making all the data available. Certainly, it allowed this review to go in a much deeper level, to identify and validate the findings (which is how science should be done). There are, unfortunately, major problems with the paper, but identifying and acting on these problems is not an issue, since the paper and the data (i.e., the research itself) allows that. Sharing the data allowed me to rerun the analysis and do "my own" check of data and conclusions, and I can say that, even though, the statistical analysis performed by authors is wrong, I checked that the conclusions would have been the same had the right statistical toolkit had been applied. In summary, I highlight three main problems with the paper: i) Ethical guidelines, ii) Anonymity of data, iii) Statistical analysis and iv) Metrics evaluation. Problems 'ii' and 'iii' are an easy fix, and I provided a variety of suggestions below; problem 'iv' is not that big, but authors should be aware of it and address it; problem 'i' is a bit more tricky and hopefully authors can state which ethical guidelines they followed, and clarify some aspects of the participant's consent. Since the paper is moving forward and encourages reproducible quantitative evaluation on the topic, the contributions are relevant to our field. Therefore, I suggest a "Major revision" so that authors can fix all 4 issues, even though the effort in fixing those are straightforward.

Problem 1: Ethical guidelines
Regarding student's particiation, authors state (Page 7/20): "Bachelor students that needed to participate in an experiment ...  while the experiment participation was not graded, they nonetheless had motivation to participate to pass the course." To me that statement is either conflicting or confusing. Did students *need* to participate in an experiment to pass the course? Perhaps I misunderstood, but does it mean that students *had to* participate, but were not graded based on their performance? Could they still pass the course *without* participating in the experiment? If the answer is 'yes', please clarify so. If the answer is 'no', refer to the comment below:
According to ethical guidelines [Badampudi 2017; Singer and Vision, 2002], one must ask participants for consent while *minimizing the possibility of coercion or undue influence*. Did participants sign any form? Students should be given the option to participate in the experiment (i.e., pass the course) but also ask researchers to *not* use their data (i.e. opt-out of the study). Was that done? If 'yes', please clarify so in the text. If 'no', then the experiment has severe ethical implications since students were coerced to participate and authors should consult with an ethical committee on how to proceed.
Specific parts supporting the arguments above are:
Singer and Vision, 2002; check Sections II, points 1 and 5;
Badampudi 2017; check pages: 1172-1773

Problem 2: Anonymity of collected data.
Authors do not motivate the purpose of collecting the participant's personal information (age, gender, etc.). There is no relevance in disclosing such information, and no need for it in this study, since it risks breaking anonymity and confidentiality of the data collected [Badampudi 2017; Singer and Vision, 2002].

Particularly, I am concerned when authors ask for gender; the survey file does not suggest any value for gender (which is good, in my opinion) so I am assuming authors did not provide "options". On the other hand, the R script calculates the % of male participants as a mean of "1" values. That construct is wrong in many levels. First, the script assumes there are only "0" (females) and "1" (males). That in itself is a bad construct, since that formula conveys the idea that "only males count". More important, two values does not capture the diversity of gender (cis-, trans-, etc.). There are numerous discussion on how to collect data on gender (identity) or (biological) sex (note that both are very different for survey purposes - see https://news.stanford.edu/2015/08/19/gender-sex-surveys-081915/). Particularly, a good summary of the discussion can be seen here (highly recommended read!): https://www.surveygizmo.com/resources/blog/how-to-write-survey-gender-questions/ or in this source: https://hrc.org/resources/collecting-transgender-inclusive-gender-data-in-workplace-and-other-surveys

Academic discussions on the topic are very much on the humanities field and a current topic in LGBTQ+ studies such as Reisner et al., available here: https://www.ncbi.nlm.nih.gov/pubmed/25030120. Moreover, I am assuming that students were given the option to not disclose their personal information (i.e., they could leave it blank). I have access simply to the questions, but not the web-form itself (or I could not find it in the provided packages); however, all students provided their personal information (all fields are filled with 0 or 1). Which is not a good indication since, again, it seems that they were *not* given the option to opt-out.

Finally, gender information is mostly relevant for studies focused on psychology or cultural aspects, whereas biological sex is an object of study for medical studies. Since this paper does not seem to fall on neither categories, I strongly suggest that authors remove the personal details (at least gender and age) of the students from the paper, the final dataset and from the tables altogether, since this is sensitive data that can be used to break anonymity of the data (unlikely, but not entirely impossible).

For instance, if the majority were male (Table 3), one can simply search for a 30 year-old female student at the 2 semester of the BSc. on Software Engineering. Even if many are available, that still narrows it down; or the only 19 year-old female who is from the "Other" program that participated in this experiment.

Problem 3: Statistical analysis.
In short, all choice of tests were not suitable for the data collected.

3.1. Correlation analysis:
Authors use Spearman's rho to analyze correlation, even though it is not appropriate for this analysis. So, Spearman uses the rankings of the observation. Albeit Spearman's rho is non-parametric, there are two main assumption, one of them being a monotonic relationship between the variables. If one does a scatterplot of the effectiveness over any of the other variables, one can see that the relationship is not monotonic.
More troubling for the effectiveness, is that Spearman's rho does not respond well to ties. For instance, authors state that there is a correlation between effectiveness and attendance to the intro lecture, but since there are only two categories on the "ordinal data" for the intro column, there will basically be two ranks, which R should calculate as the average rank of all repeated values (e.g., execute: rank(data$intro)). Nonetheless, there are basically only 4 possible values for effectiveness, which then again, will lead to many tied ranks in the data. I checked R's documentation for the corr.test method used, and it does seem they correct for ties (but only the source code can really tell).
I reran the tests using kendall's tau (a better choice since its less senstivitive to these discrepancies in ranks), and the results are similar for effectiveness vs. intro. So, the spearman implementation should be correct in R. However, the choice does not cater for the monotonic relationship.
In short, I suggest that authors revise the choice of correlation score. I suggest Kendall's tau, since it can lead to slightly similar inferences as Spearman, and is more permissive with assumptions. More importantly, I am not sure the choice of constructs (particularly effectiveness), is suitable for using these correlations (there aren't that many values that effectiveness can take), to an extent where I argue that the correlation found cannot lead to any discussion around causation. So, consider *also* emphasizing that in the validity threat section.
3.2 Hypothesis testing
Regarding the hypotheses testing, there are two severe problems that invalidates the statistical results of the test:

Problem 1: Authors used a Welch t-test for the hypothesis testing. That is the wrong choice for their data. Welch's t-test assumes that the samples are normally distributed which is not the case for effectiveness or efficiency. Doing a quick Shapiro-wilk test (or Anderson Darling) for normality, quicly reveals that the data are not normally distributted. Moreover, plotting a simple histogram shows that the distribution is, in fact, quite uniform for effectiveness. The visual distribution of the efficiency is slightly better, but it still fails the Shapiro-Wilk test.

Therefore, any parametric test or anything assuming normality can be quite risky to use. I suggest that authors revise the text. They should: i) properly motivate the choice of test making sure that the assumptions are met, hence introducing information on analysis of normality; ii) turn to non-parametric choices, that seems a better choice for their data. Perhaps a simple Mann-Whitney test could be enough for them, even though there will again be many ties in the ranks. I did a quick check, and for a two-sided test, the mann-whitney test indicates no differences for effectiveness (p-value = 0.09, so nothing changes), but with 0.1 alpha, there is a slightly significant difference between duration.

Problem 2: I checked the scripts, and the actual calculation *does not match the stated hypotheses*. The t-test method, in R, uses by default a two-sided test, often expressed as a "H1: a != b". Authors use a one-sided test, meaning that only half of the p-value should be accounted for the given level of confidence. For instance, for effectiveness, the p-value reported is: 0.915, when in fact, the "right" p-value should be: 0.54 (or 0.4574 if I misunderstood the script). This should be done by defining either "greater" or "less" on the alternative field in the t.test R function. So, on one side, the conclusions should not change since, still, there are no differences (high p-value); on the other side, authors don't need to bother, since the choice of Welch t-test was wrong in the first place. I suggest authors review the alternative hypotheses, and make sure the right calculations are being used by the R functions when choosing another test.

Checking assumptions behind parametric tests has been already discussed by [Arcuri and Briand, 2011], other risks raised by [Kitchenham et al., 2019] and the risks of using null hypothesis significance testing altogether was raised by [Sheppard, 2018] and [de Oliveira Neto et al., 2019]. I recommend authors the recent analysis done by [de Oliveira Neto et al., 2019] where its authors discuss the main pitfalls with statistical analysis in SE, but provide guidelines on how to avoid these pitfalls, along with a suggestion of tests that can be used.

Problem 4:
This is not as challenging as the other problems. On section 5, authors evaluate the different versions in terms of different internal quality metrics. This is tangential to the analysis with students, so it is appreciated to introduce a new perspective on the analysis. However, if I understood correctly, authors evaluate internal quality of artefacts created by themselves (since my understsanding is that the implementation of the services was done by researchers?!). Authors do acknowledge this issue under their external validity threat when they say: "experiment tasks are still [...] subjective bias".

Since the artefacts were developed to better fit the experiment context (as should be the case, since many aspects should be controlled in an experiment with people), the metric data, however interesting is not reliable or representative. I suggest that authors to i) re-run this analysis on perhaps an existing project (or more) that has such patterns. Even though I understand the difficulties of finding artefacts, *an alternative* should be to ii) have non-researchers or more senior students implement a few of these patterns and then evaluate them instead. In either case, this, unfortunately, means some revision of Section 5.

As a compromise, authors could try to keep section 5, but reduce the strength of their claims throughout the discussion, and explicitly highlight these limitations on this analysis (i.e. not representative) as a construction and external threat (i.e., expand a bit the current external validity text). The future work text can also build on this analysis to implement the suggestions 'i' or 'ii' above.

Minor comments:
M1 - Authors define that the purpose of their experiment is "modifiability" (P.6, L.68), but that is still a bit too open and not really a purpose (rather it's just the quality characteristic targeted). To further clarify the research objective, I suggest authors include the actual purpose in their description, such as one of the following: "for the purpose of characterizing/comparing/evaluating/predicting/controling modifiability" (as suggested by Wohlin et al., 2012 - Table 7.3).

M2 - Regarding the available system used in the experiment, I had trouble executing the "_scripts" folder at first on a MacOS. The README did not provid einstructions on how to make them work on a MacOS. My MacOS did not have the gnome-terminal, so I basically just commented out the first and last line. Also, the README suggests execution of the *srv-run.sh files, but the system needs to be built at least once so the target files can be found. I could not execute the WebUI also due to problems with npm.

I suggest that authors edit their README file, to provide pointers on how ot execute either on Windows or MacOS. If it works on MacOS, should work on Linux as well. Nonetheless, the scripts and analysis data were executable and editable, so regardless, the attitude and organization of the authors is commendable for fostering reproducibility.

M3 - Authors state on Page 4/20: "And second, if the patterns’ advantages materialize even with inexperienced developers that have little or no conscious knowledge of them, *their effect on evolvability must be really significant*.". That seems like a fair assumption, but it is misleading due to the use of "really significant". Certainly, would be more beneficial, but it is conjecture to assume the magnitude of this significance. I suggest authors rephrase the sentence to avoid overstating the magnitude of the effect.

M4 - Authors use the term (Page 7/20, line 204): "dependent response variables". The terminology is either "dependent" or "response" variable. Choose one.

M5 - Simply an aesthetic suggestion (feel free to ignore this one): reading through the formal hypotheses is not easy. Authors could format the text in a table (grid) with two columns (null and alternative hypothesis). Each row would be a pair of null/alternative hypotheses. In each cell (or different cells in a multi-row table in latex), authors could include the text + equation for the corresponding hypothesis. This would make it easier to visualize how there are 1 pair for effectiveness (also that this is a one-tailed test), and the distinction with efficiency per task (H1-3) and all tasks (H4)

M6 - When presenting the mean values in tables 3 and 4, authors do not clarify whether the 1-10 scale means. Does 10 mean very experienced on the topic? I suggest that authors clarify the scale both in the questionnaire file (zenodo) and the text or table caption.

M7 - On Table 4, the Mean Duration for all Tasks is ambiguous between i) the mean duration of each task (i.e., the mean is calculated over all task entries in the data), and ii) the mean duration for people who complete all tasks (i.e., how long on average it takes to do all 3 tasks).

I understand that authors and the script report on 'i', so I suggest authors clarify in the table (caption or footnote or row name) to remove the ambiguity. Our of curiosity, I calculated 'ii' in the scripts and the values were: version 1: 4530s; version 2: 3819s; (ca 12 minutes difference). I suggest authors to include this information. Certainly, the calculations only considered rows where effectiveness was 1 (i.e., all 3 tasks completed).

M8 - Authors state: "Roughly 30% were not able to solve any task.". The percentage should be complemented by the absolute numbers per version. For instance, I checked the data and it shows that the number is 10 for v1 and 11 for v2. I suggest that author add that information to the text, to show that there was no specific group that could not completely (it was virutally the same)

M9 - Authors state: "The only notable exception for this was task #3 which was perceived as about 40% less difficult by the pattern group #2." This is insightful, but I believe the statement is incomplete. Since only 25% of participants finished task 3 (10 for v2, and 7 for v1), perhaps those 40% are indeed not representative?! I personally cannot tell, I would have to analyse the data to assess what is this 40% difference in absolute terms (i.e., is it 4 points on the 1-10 scale easier?). I looked at the histograms of the ex3.diff of each version and that was more insightful. I suggest that authors elaborate on this difference providing more absolute terms to complement the current percentage value (40%). Perhaps introducing a section with some histograms of specific distributions would be more insightful than simply reading the numbers.

M10 - Page 10/20, lines 273,274 authors report on a multiple test for each task using Bonferroni correction. Also, authors report on a power analysis. I could not find thoses tests on the provided scripts or package. I suggest that authors add those tests to the script. It would be great to add, in the script as a comment, the corresponding page where or section where the test is being reported in the text.

M11 - As an aesthetic, but very relevant suggestion, Table 6 should be re-designed. The choice of colors is not good for many reasons. First and foremost, red and green is not a good combination for colorblind readers, and secondly, the distinction lies, currently on the hue, which is undistinguishable when printing on greyscale. I suggest that authors change to a colorblind friendly pallete that also have different luminosity/shading so that black-white prints can also highlight the differenes.

References:
F. G. de Oliveira Neto, R. Torkar, R. Feldt, L. Gren, C. A. Furia, Z. Huang. Evolution of statistical analysis in empirical software engineering research: Current state and steps forward. 2019. Available on arXiv: https://arxiv.org/abs/1706.00933

D. Badampudi, "Reporting Ethics Considerations in Software Engineering Publications," 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), Toronto, ON, 2017, pp. 205-210. doi: 10.1109/ESEM.2017.32

J. Singer and N. G. Vinson, "Ethical issues in empirical studies of software engineering," in IEEE Transactions on Software Engineering, vol. 28, no. 12, pp. 1171-1180, Dec. 2002. doi: 10.1109/TSE.2002.1158289

S. L. Reisner, K. Biello, J. G. Rosenberger, S. B. Austin, S. Haneuse, A. Perez-Brumer, D. S. Novak, M. J. Mimiaga. Using a two-step method to measure transgender identity in Latin America/the Caribbean, Portugal, and Spain. Arch Sex Behav. 2014 Nov;43(8):1503-14. doi: 10.1007/s10508-014-0314-2. Epub 2014 Jul 17.

M. Shepperd. Replication studies considered harmful. 2018. Available in: https://arxiv.org/abs/1802.04580

B. Kitchenham, L. Madeyski, and P. Brereton. 2019. Problems with Statistical Practice in Human-Centric Software Engineering Experiments. In Proceedings of the Evaluation and Assessment on Software Engineering (EASE '19). ACM, New York, NY, USA, 134-143. DOI: https://doi.org/10.1145/3319008.3319009


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ON THE IMPACT OF SERVICE-ORIENTED PATTERNS ON SOFTWARE EVOLVABILITY: A CONTROLLED EXPERIMENT AND METRIC-BASED ANALYSIS
Review round: 2
Reviewer: 1

Output:
Basic reporting: As written in my review for the original submission of this paper, I am very positive regarding the actual content of the paper.
Experimental design: I had only one major comment in my original review, asking for the discussion of standard deviations wherever appropriate. This has now been added.

As far as I can judge, the comments by the second reviewer have also been handled in a satisfying way. It should be noted that it seems as if the second reviewer has way more knowledge about this kind of studies than I do.
Validity of the findings: My major comment regarding the validity of the findings addressed that the developers participating in the study were quite junior. The according discussion (which was already fine before) has now been further extended.
Additional comments: Actually, I recommend not to use bold font to highlight single terms, since this decreases readability. Instead, italics should be used.

The authors should check again their references, since the level of detail given in the single entries is very heterogeneous. Just some examples - as written in my review to the original submission, the authors need to check all references carefully:
* Almost all articles are missing page numbers, a lot of journal articles are missing volume and issue information
* Uppercase and lowercase lettering of the titles does not seem to follow any pattern
* Some conference names include a "Proceedings of", but most don't. Some conference names are only given as abbreviations (e.g., Rud et al. 2006), while most are given completely.
* "Communications in Computer and Information Science" is given twice for the paper by Hegedüs et al., while the actual booktitle is  missing.
* Missing blank in Kassab et al. (2012)
* Publisher names are given in an inconsistent way
* etc. etc.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ON THE IMPACT OF SERVICE-ORIENTED PATTERNS ON SOFTWARE EVOLVABILITY: A CONTROLLED EXPERIMENT AND METRIC-BASED ANALYSIS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The new version is improved in terms of presentation with clearer tables and better description of formal hypotheses, data and statistical results.
Experimental design: The main hindrances have been addressed and carefully explained by authors. The design and the analysis now are more suitable to the data collected and hypotheses investigated.
Validity of the findings: Authors were very transparent with the main threats to validity. Also, the sharing of the experiment's data, scripts and artefacts significantly add to the reliability of the experiment since it allows other researchers to re-run the analysis done by authors.
Additional comments: Authors did a great job both in addressing all concerns I had regarding the paper and in providing detailed information regarding their changes and updates. Particularly, I appreciate that authors were thorough when describing the ethical concerns regarding their data collection in the new version of the paper. The new version of the analysis is also more suitable to the data collected, as well as the updates on the presentation. In summary, the paper was improved and addressed the main shortcomings, so I recommend an Accept to this paper without the need of further updates.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NOVEL TRAJECTORY CLUSTERING METHOD BASED ON DISTANCE DEPENDENT CHINESE RESTAURANT PROCESS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The text must be thoroughly revisited as there are several and recurrent typos, specially regarding prepositions, verb conjugation and false friends; while the overall ideas can be understood, the text should definitely be improved before publication. Authors should take special care with the paragraphs between lines 185-192 and 230-233 as I can't really understand most of the ideas presented in those lines.

Typos and other grammar errors can be found in lines:
29,31,38,43,48,50,101,140,143,149,152,195-196,200,207,208,219,223,226,249,271,272,276,305,325 and figure 5 caption.

The are no major flaws on the figures and tables but the values on the y axis for figures 3 and 4 could be confusing as they go beyond 1.0, please relocate the legend.

While the authors make an a effort to either provide a straight derivation or references for most of the relevant equations in the paper,  equations 8 and 9 cannot be trivially derived, and the authors provide no references for further reading.

The introduccion does a good job on highlighting the improvements of the proposed method vs other classic methods and its actual scope, but could be further improved by briefly mentioning that the paper addresses the trajectory analysis problem for  traffic intersections. The abstract can also be further enhanced with this information and some of the most relevant quantitative results.
Experimental design: Overall, I have three main concerns regarding the experimental setup, and the empirical validations.

First, I  would like to know why the authors choose 3 distance metrics (DTW,LCSS and MH), and 3 clustering algorithms (k-means,Spectral clustering and agglomerative clustering) after they present several more in the introduction, was the selection based on the performance of the methods?, its similarity?, another reason?. In other words, I would like to know better why the selected methods are an appropriate baseline given the author's proposal and the available data.

Second, I think the paper lacks on empirical validation to further assess the effectiveness of the proposed method, first of all I would like know why the authors use a fixed number of clusters (fixed at 19), how does the proposed method perform if that number varies?, is it optimal?,  how do the baselines perform if that parameter change?, also I would like to know more about the indicator function of equation 3, and its related threshold, these two parameter seem to be critical at selecting the number of clusters but are not reported for the experimental setup could the authors perform a sensitivity analysis on the threshold or perhaps try several functions? To complement the empirical validation of the methods, could the authors evaluate the capacity of their strategy to predict the actual amount of clusters in a dataset? is it possible to use the proposed strategy to classify the activities on the Lankershim datased, and compare against the baseline?

Third, in the introduction the authors discuss about ‘probabilistic reasoning’, where a soft-assignment could be obtained as probability distribution every sample and every cluster, this possible advantage of the proposed method is never explored or validated in the paper.

Finally, as stated in amigo at all 2009, the  purity metric penalizes the noise in clusters, an can yield very high values for trivial assignations, the authors should complement their empirical evaluation with metrics like Inverse purity or Van Rijsbergen F-Measure for a better understanding of the proposed method effectiveness
Validity of the findings: No comments
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NOVEL TRAJECTORY CLUSTERING METHOD BASED ON DISTANCE DEPENDENT CHINESE RESTAURANT PROCESS
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper presents a direct application of the distance dependent Chinese restaurant process to a tracking problem. There is no technical contribution, which is ok as long as the proposed application is novel and experimental results are convincing.

Yet, the article as it is has very poor quality in terms of writing (poorly written, unclear sentences, missing references), and experimental results (1 single dataset, insufficient comparisons with other state-of-the art methods). The article as it is needs considerable more work.

1) English: the text is very poorly written (all except the description of the DDCRP prior), I have annotated too many English typos/grammatical errors, which makes the article hard to read. Very often, there is no verb in the sentence.
Some examples:
obtained by keep tracking --> by keeping track
probabilistic reasoning can use to quantify --> can be used to quantify
which the knowledge of one helps --> ?
I suggest the authors to seek proof-reading from native English speakers.

2) The authors have missed important references, e.g., works that use particle filtering (one of the state-of-the-art techniques for trajectory modeling), as well as other Bayesian nonparametric models for tracking, such as:
A Bayesian Nonparametric Approach to Modeling Mobility Patterns (Joseph et.al, 2010)
Hierarchical Dirichlet processes for tracking maneuvering targets (Fox et.al, 2007)
Please include these references and compare your methods with some of these alternatives.

Minor:

- please number sections
- I have not found the raw data in the supplementary file. Also, demo.m does not run. Please document the raw data and code appropriately.
Experimental design: The authors use a multinomial likelihood to model continuous data, which requires quantization of the images using grids.

- How does the DDCRP compares against a (Joseph et.al, 2010)? This would be a very interesting comparison.

- How many MCMC chains were used? How was convergence assessed? Is 100 samples enough? Do the authors perform averaging over multiple samples?
The authors should run the models using different initializations and multiple chains.
Validity of the findings: - Results are inconclusive as they are. Since the main application is a direct application of the ddCRP to tracking, the authors should provide simulations with more than a single dataset. Also, the authors should compare with other probabilistic models like (Joseph et.al, 2010) to be fair and conclusive.

- Data is not provided in the supplementary (the folder /Software/demo/Lankershim/data/ is empty; there are some .mat files in Lankershim/ddcrp, but without any documentation.

Details about the dataset should be listed (how many images per label, number of pixels, etc...
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: NOVEL TRAJECTORY CLUSTERING METHOD BASED ON DISTANCE DEPENDENT CHINESE RESTAURANT PROCESS
Review round: 2
Reviewer: 1

Output:
Basic reporting: There are a few typos on the manuscript.
The authors provide a fair amount of related work along clear and concise explanations of the objectives, contributions and core methods used in the paper.
Some equations (7,8,9) are better suited for an appendix, while they present a detailed analysis of an step in the methodology they not part of the author contributions.
Experimental design: The authors evaluate in two datasets for trajectory analysis in intersections crosses. Key hyperparameters of the problem as sample size, grid size and cluster amount, are thoroughly evaluated. Several baseline methods are benchmarked and compared against the proposed methodology.
Figures provide interesting insights on the  trajectories discovered by the method.
Method explanations (although lengthy) provide valuable information for the replication of the proposed method.
Validity of the findings: My biggest concern on this paper is the result presentation, while the proposed method consistently outperforms the baselines across  operations regimes, sometimes the improvement looks minimal (e.g. 0.986 to  0.993), I think the authors could further highlight why it is important to show improvements in a problem whose baseline is so close to perfection.
I also would like to know why the abnormal trajectories are discarded.
Other than this the conclusions are, in my opinion,  validated by the presented results.
Additional comments: I think the article presents a clear methodology to improve the baseline on the problem of trajectory discovery and clustering, it is interesting that the proposed DDCRP automatically finds a number of clusters that produce improved results.
There are some minor flaws that must be revisited, but I'm leaning towards acceptance of the manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUTOMATED LANGUAGE ESSAY SCORING SYSTEMS: A LITERATURE REVIEW
Review round: 1
Reviewer: 1

Output:
Basic reporting: The manuscript brings an excellent review on automated scoring systems, providing the most relevant items in the literature put into context. There is also a fair amount of discussion about the strengths and limitations of existing systems. Overall, the manuscript is likely to be useful for anyone investigating intelligent systems associated with text analytics.
Experimental design: The study was well designed with a systematic search of automated scoring systems, which were then divided into broad categories to facilitate readability. The review is also organized in a logical manner.
Validity of the findings: The findings were reported in a straightforward manner, making it easy for the readers to grasp the most important developments in the field.
Additional comments: This is an excellent review, and my only recommendation is a careful proofreading of the manuscript. There are a few typos that need to be corrected, in addition to inadequate usage of some terms (e.g. "In the other hand", rather than "On the other hand").


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUTOMATED LANGUAGE ESSAY SCORING SYSTEMS: A LITERATURE REVIEW
Review round: 1
Reviewer: 2

Output:
Basic reporting: The topic of the article is important, and the evaluation of the different approaches/systems is required in the presence of huge use of AES systems especially in English Language Assessment and standardized exam all over the world.

The article has a good introduction and background to demonstrate the importance of the work and how it fits into the field of knowledge. Also, the structure of the article is logical and in an acceptable format.

In general, the language of the article is clear and the ideas are well organized. Yet, there are few English mistakes (See below). I suggest reviewing the article by a native speaker.  The tables and figures are clear and useful as well.

In line 185, “the correlations average” to “the average of the correlations”
In line 235, “makred” to “marked”
In line 249, “learn” to “learns”
In line 313, “consider” to “considers”
In line 332, “convert” to “converts”
Line 336, ambiguous sentence. Please rewrite
In line 338, “represent” to “represents”
In line 352, “took” to “taken”
In line 380, “On contrary of” to “On the contrary to”
In line 391, delete “and” from “…… automatically, and apply them ……”
In line 435, delete “to AES systems” in “The third challenge to AES systems is measuring ….“
Experimental design: The article content is within the aims and scope of the journal. The method is well described and is sufficient for the purpose and nature of the article. It can be reproduced. The survey method is unbiased. The review is organized logically into coherent sections.

It seems that the focus of the article is AES systems that evaluate English language writings only. I think it will be an enrichment if it includes also systems for other languages such as Arabic; (For example Al-Jouie, M. F., & Azmi, A. M. (2017). Automated Evaluation of School Children Essays in Arabic. Procedia Computer Science, 117, 19–22. https://doi.org/10.1016/j.procs.2017.10.089). Otherwise, it should be stated clearly as a limitation of the article. It is up to the authors.

I also propose to include another system from China: Yubing, Q. (2016). Pigai Smart Essay Scoring System and Its Implications for Teaching English Writing. Journal of Applied Science and Engineering Innovation, 3(6), 217–219. Retrieved from http://www.jyb.cn/high/gdjyxw/
Validity of the findings: The comparison made is logical and data-oriented. The conclusions are well stated. It will be better if authors propose future work/an approach to overcome the third challenge or enhance AES systems.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUTOMATED LANGUAGE ESSAY SCORING SYSTEMS: A LITERATURE REVIEW
Review round: 1
Reviewer: 3

Output:
Basic reporting: The article is generally written in a clear, coherent and professional English. However, there are some segments that need to be corrected because they are not well understood or they are incorrectly written. It is necessary to improve the writing at lines:
- 41 to 43:  they lack the sense of the rater as a  person ....  (improve the writing)
- 54 to 55: The extended-response, such as essays, … (it is not clear, improve the writing)
- 60 to 61: It occurs that the score of an essay scored  …  (improve the writing)
- 80:  The most common models found for the AES systems are Natural Language Processing … (NLP is not a model,  use another term to refer to NLP)
- 102-103: To end, a score is determined by estimating coefficients (β weights) from the training stage…   (how the coefficients are calculated?)
- 234: It consists of 12.976 150-to-550 word-essays… (it is not clear, improve the writing)
- 298 to 299:  1->2, 3->4, 5->6 and 7->8, where 1->2 denotes prompt one as source domain and prompt.  (I did not understand this notation, try to explain better)
- 336: Dasgupta and his colleagues employed in their experiments is that the ASAP… (improve the writing)
- 350 to 351: Recently, computer technologies, especially NLP and AI … (use a more appropriate term for NLP and AI)
- 421 to 423: … examined whether a computer could be deceived and assign a lower … (it is not clear, improve the writing)
- 425 to 426: one of the most recent technologies, which is Neural Networks… (use a more appropriate term for Neural Networks and improve the writing)

Figures:
- Figure 2: I think the figure is redundant. Maybe the information the figure contains could be explained in a simple list inside the paper content.
- Figure 3: The elements of the figure should be better explained.
- Figure 4: Both figures are equal and they do not show something special. You should use other example figures for a better understanding.

Table:
- Table 1: Spelling error of word “scorers” at the beginning of the table:  Correlation with human scorers
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The article is well written and structured.  The Introduction and background are correct because they reveal the problems that need to be addressed and the objectives of the paper are clearly explained. Also,  the methodology of this work is well developed.  The conclusions are well established and they discuss the relevance of this research for the task of automate language essay scoring.

Some recommendations for the improvement of this paper are detailed in the basic reporting section.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUTOMATED LANGUAGE ESSAY SCORING SYSTEMS: A LITERATURE REVIEW
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors addressed the issues raised, and the manuscript can now be accepted for publication.
Experimental design: No comment
Validity of the findings: No comment
Additional comments: The authors addressed the issues raised.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: AUTOMATED LANGUAGE ESSAY SCORING SYSTEMS: A LITERATURE REVIEW
Review round: 2
Reviewer: 2

Output:
Basic reporting: -
Experimental design: -
Validity of the findings: -
Additional comments: -


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAY ATTENTION AND YOU WON’T LOSE IT: A DEEP LEARNING APPROACH TO SEQUENCE IMPUTATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: In this paper, the authors proposed the use of attention mechanisms for the imputation of values in discrete sequences using LSTMs. According to the authors, this is the first time that this type of imputation mechanisms are used in discrete sequences and they argue this fact.
They use the architecture of Vaswani's transformer model and test the results in the CAN data set.

The paper is well written and well understood. The results compared to an LSTM baseline are surprising in terms of accuracy and time reduction. However, I am not clear about the novelty of the idea with respect to an accepted paper written by the same authors in IJCNN this year.

I'm sure there are differences, beyond the implementation of the model in Keras instead of TensorFlow, but I think that the authors should record each and every one of the novelties that this new study represents with respect to the previous publication. I am referring to the differences in the mechanism of attention and architecture, as well as the extension of experimental results and tests that are carried out in this paper. A new section (or subsection) should be included for this purpose.

From my point of view, with a clear justification of the novelty, the paper could be accepted, as it represents an interesting advance in the pre-processing of discrete sequence data.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAY ATTENTION AND YOU WON’T LOSE IT: A DEEP LEARNING APPROACH TO SEQUENCE IMPUTATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: Line 40: I’d still delete “only” from here so it reads “very few attempts” rather than “only very few attempts”.

I’d prefer a legend on Figure 2 that didn’t rely on underscores and clarified the names a little more, particularly since these names are not defined in the manuscript (I appreciate that the configuration  names are clear enough in intent, but they could definitely be improved).

Line 181: I don’t think this comma is needed (or another matching comma is missing from earlier in the sentence).

Lines 206–207: Perhaps “In the context of…” (adding “the”) here and a comma after “systems”.

Line 248: I don’t think the first comma (after “model”) is needed (or another matching comma is missing from earlier in the sentence).
Experimental design: No comment.
Validity of the findings: No comment.
Additional comments: The revised version of the manuscript addresses my primary concern from last time (the reliance on an anonymous unpublished manuscript) and addresses most of the other comments I made also. I have only a few minor comments above, one which is a repeat from the previous version of the manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAY ATTENTION AND YOU WON’T LOSE IT: A DEEP LEARNING APPROACH TO SEQUENCE IMPUTATION
Review round: 1
Reviewer: 3

Output:
Basic reporting: This paper studies the use of LSTMs for data restoration. The current state-of-the-art performs a large number of sequential operations, slowing the training down. The aim of this paper is to alleviate such issues by means of attention mechanisms. This is inspired in the paper of Vaswani 2017, and the authors use a version of ‘Transformer with modified hyper-parameters. As such, the novelty is not extremely high, but I believe that it is definitely of interest for the deep learning community.

Overall, the paper is well-written and is easy to follow. I believe, however, it requires to extend a few sections to contextualise better their proposal.
Experimental design: - The authors are using data collected from a CAN of an SUV. Following the description in section 3.1, it is unclear what kind of machine learning problem the authors are tackling. I don’t understand the meaning of 43 unique messages Ids. The authors need to clarify what is going to be input into the proposed model.

- The training/test split seems to be a bit too simplistic. Why should you apply a fold-cross validation?

- Why table 4 doesn’t report the time for 3000 epochs?
Validity of the findings: The algorithm seems to be quite competitive in comparison with the previous authors’ proposals. However, this is only applied on a single dataset; the authors should apply this on more datasets to check if the proposed method actually restores well in other scenarios.
Additional comments: The introduction is missing general references on data preprocessing, data cleaning, obtaining ‘smart data’ from big data. I would also extend the introduction to motivate better the problem of restoration and when it is applied.

I very much like the discussion about Deep learning not being used for data-preprocessing or cleaning. I wonder if the authors could elaborate a bit more on that regard. For example, data augmentation, GANs could be considered preprocessing as well (as oversampling technique). But it is definitely true that it is frequently not being used for data filtering or noise detection. I think that would also be a good opportunity to contextualise better the proposed method.

Is restoration a missing values imputation mechanism? If yes, it is unclear in the introduction if the proposed method is devised to tackling images/videos or ‘any kind’ of data. In the case of deep learning, the authors should also mention in-painting techniques as related work, and for standard classification/regression/time series problems, there are many missing values imputation mechanisms which are related. I slightly disagree that the classical techniques for missing values imputation revolve around local and global means (they are of course the basic), but classical techniques also include techniques such as the k-nearest neighbours.

The description of the proposed ‘restorer’ method is a bit shallow. The authors have assumed lots of background information which could be solved by adding some additional information in the background. Figure 1 should belong to section 2.2.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PAY ATTENTION AND YOU WON’T LOSE IT: A DEEP LEARNING APPROACH TO SEQUENCE IMPUTATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have satisfactorily addressed all my comments and now the paper can be accepted for publication.
Experimental design: The authors have satisfactorily addressed all my comments and now the paper can be accepted for publication.
Validity of the findings: The authors have satisfactorily addressed all my comments and now the paper can be accepted for publication.
Additional comments: The authors have satisfactorily addressed all my comments and now the paper can be accepted for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CITATION.JS: A FORMAT-INDEPENDENT, MODULAR BIBLIOGRAPHY TOOL FOR THE BROWSER AND COMMAND LINE
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article gives a detailed overview of the Citation.js software with impressive results. The article is a more a report than a research paper and it provides a practical solution to the relevant problem of accessing, and converting bibliographic data for citation. The background section should slightly be extended with references to related works (see comments on validity of the findings). Apart from that there are only minor issues, listed here in their order of appearance :

- subsection "output formatting" (line 104) could make more clear that the most of the formatting is done via the existing library citeproc

- subsection "CLI use" (line 152) could be reduced as it contains general information common to most command line application

- figure 3 is too large

- there seems to be a "not" missing between line 215 and 216

- in the same subsection one might mention that extension of the crosswalk format CSL-JSON might be a solution

- the subsection on GraphQL (line 235-242) should be removed as it its currently not used in Citation.js and the specific relevance is not made clear

- in section line 249-252 one might mention RDF as it is better suitable for merging data from multiple sources that other formats
Experimental design: no comment
Validity of the findings: The results are relevant and created in the best form by modern software development, but the research question has been addressed before. The article in should be extended by references to existing solutions and related works. In particular:

- mention library catalogues as central stores of bibliographic data (line 29) with library-specific formats such as MARC

- mention existing tools: At least Chris Putnam's Bibutils is relevant here: Citation.js seems like a modern successor to Bibutils

- add a subsection about the problem of converting between bibliographic data formats in the background section (with reference to subsection starting at line 198 for details).  The topic of schema/metadata crosswalks is relevant both in general information integration and in metadata management in particular. The article does not need to give a full review (see https://en.wikipedia.org/wiki/Schema_crosswalk for a starting point) but it should briefly describe where crosswalks between bibliographic formats already exist (for instance XSLT has played a major role).
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CITATION.JS: A FORMAT-INDEPENDENT, MODULAR BIBLIOGRAPHY TOOL FOR THE BROWSER AND COMMAND LINE
Review round: 1
Reviewer: 2

Output:
Basic reporting: This paper introduces citation.js, which seem to be a very useful and important tool to map between the different formats currently used to represent citation information. The paper is relatively easy to read, but unfortunately not written and structured in a very scientific way and lacks crucial elements. The reader is given many details about the technical details of the tool, but unfortunately virtually nothing is said about its position with respect to the existing scientific literature on the topic of citations, and very little in terms of evaluation results.

Specifically, the main sections Background, Results, and Discussion are all very unconventional and lack important scientific content:

- The Background section only introduces technologies and libraries that were used to build the tool, but not a broader background in a scientific sense. I expected to read something about other work on analyzing citation networks, on the usage and impact of reference managers, on the properties and uptake of ontologies on scholarly communication, just to mention a few examples from the top of my head.

- The Results section contains mostly details about the implementation of the tool, which are not results in the scientific sense. These implementation details should rather go into a section called Approach or Implementation or Methods.

- The performance evaluation is the only real result in the Results section, but performance is in my view a secondary problem here. More importantly, I would like to see some results on whether the tool is accurate, useful, and/or actually making an impact.

- The Discussion section is also much too technical, whereas I would expect here some more high-level context and interpretation of the presented results.

In general, the paper seems to be more of the kind of a Resource Paper as some conferences and journals have them (e.g. here http://iswc2018.semanticweb.org/call-for-resources-track-papers/), but these papers are typically required to demonstrate and quantify adoption, or at least to provide "evidence and motivation for claiming potential adoption". PeerJ doesn't seem to have an explicit call for such resource papers, but this paper would in any case not meet such typical Resource Paper requirements either.

Therefore, in summary, I think that this paper shouldn't be accepted. I have the impression that citation.js is indeed a very useful and important tool, but the omissions of the paper as explained above are too serious to warrant Major Revision as a decision.
Experimental design: No experimental design is reported.
Validity of the findings: No scientific findings are shown, except for performance measures, which are of secondary nature.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CITATION.JS: A FORMAT-INDEPENDENT, MODULAR BIBLIOGRAPHY TOOL FOR THE BROWSER AND COMMAND LINE
Review round: 1
Reviewer: 3

Output:
Basic reporting: This is a well written presentation of an important and impressive open-source software contribution and as such I'd recommend it for publication in PeerJ. Most of my review is contained as comments in the uploaded PDF, but some general comments:

I don't think the article is well-served by following a structure designed for more "traditional" empirical articles. E.g., it wasn't clear to me why some things were "Results" and some were "Discussion". I would revise this structure, and my reading of PeerJ guidelines allow, if not encourage, authors to do so.
Experimental design: no comment (does not apply)
Validity of the findings: no comment (see PDF)
Additional comments: The biggest advantage of a journal publication for a software package over simply providing the DOI for the source code itself (which is already the case for citations.js) is that it allows authors to explain their choices and practices in developing the software and to provide a "bigger picture". The present article fully delivers on the former, but I'd like to see some more big picture. For one, a clearer exposition of what exactly the tool does, right at the start of the paper, would help a lot. Secondly, a list of current implementations would be helpful -- I know there's at a minimum Wikicite, but I figure there are others? As it currently stands, the articles requires the reader to be very familiar with the metadata/citation landscape to understand the tool's contribution. I think this can be explained in a way that most readers familiar with academic referencing more broadly could be convinced of its importance.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CITATION.JS: A FORMAT-INDEPENDENT, MODULAR BIBLIOGRAPHY TOOL FOR THE BROWSER AND COMMAND LINE
Review round: 2
Reviewer: 1

Output:
Basic reporting: With its focus on a scientific tool, the article is more a report than a research paper. It provides a practical solution to the relevant problem of accessing, and converting bibliographic data for citation. The article gives a detailed overview of the Citation.js software with impressive results. The software is well described with motivation, capabilities and compared with existing solutions.
Experimental design: The article does not describe a scientific experiment but the outcome of engineering. Motivation and goals are clearly described and the article includes some technical evaluation. Additional user studies may be appropriate if the software was more end-user oriented but it is not required in this case.
Validity of the findings: The results are relevant and created in the best form by modern software development. The software documentation is better than normal at current research software so the article can also help to increase awareness for other kind of scientific software reports. The documented mapping table between CSL JSON and other bibliographic formats is valuable as research data in its own right.
Additional comments: Thanks for the thorough extension of the article.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CITATION.JS: A FORMAT-INDEPENDENT, MODULAR BIBLIOGRAPHY TOOL FOR THE BROWSER AND COMMAND LINE
Review round: 2
Reviewer: 2

Output:
Basic reporting: Quite well written. Background needs to be improved (see below).
Experimental design: Evaluations make sense, but need to be explained better (see below).
Validity of the findings: The findings are quite technical but seem valid (see below).
Additional comments: This paper has greatly improved from the last revision. It is, however, still not written in a sufficiently scientific way, with too much focus on technical details and not enough high-level explanations and motivations.

Major comments:

- The Background section has now a different content than before, partly copied from other places of the previous manuscript, but it is still not a proper background section. It is very technical and very narrow, and thereby fails to give a general and broad overview of this research field. Specifically, this comment from my previous review still holds: "I expected to read something about other work on analyzing citation networks, on the usage and impact of reference managers, on the properties and uptake of ontologies on scholarly communication, just to mention a few examples from the top of my head."

- As with the Background, also the Approach section lacks a high-level description. The content is all very low-level and technical, which is OK for such an Approach section, but only if preceded by a high-level introduction and explanation. The Approach section start with pointing out what version control system was used, instead of introducing the high-level Citation.js approach.

- I like the addition of the accuracy reports in the Results section. This really strengthens the paper! However, "accuracy" is highly confusing as a term, both intuitively and technically (because of its very specific meaning in statistics). Reporting accuracy levels of 27% makes your tool look performing very poorly when the common interpretation of "accuracy" is applied. I suggest to rename "accuracy" with a more appropriate term, e.g. "coverage". Moreover, the text could make it a bit clearer to what extent these numbers come from a lack of expressiveness of the studied formats or a lack of implemented features on the side of Citation.js (I think it's already in the text to an extent but I found it difficult to understand).

- The designs of the evaluations presented in the Results section should moreover be introduced better (either in a previous section such as Approach, or before the actual results are described in Results). Now, these designs are only described implicitly together with the results. It is good style to separate the two when describing evaluations, i.e. first describing the evaluation's design and only then its results.

More minor comments:

- I like Figure 1! But it's a bit unclear which parts make up the Citation.js tool (all the arrows? only the arrow pointing from "Input" to "Output"?)

- I like the addition of Impact in the Results section! I think it would help further if Figure 5 was quickly summarized in a sentence (like "Overall, our package was downloaded X times since its creation in October 2016").


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVENT-DRIVEN INDUSTRIAL ROBOT CONTROL ARCHITECTURE FOR THE ADEPT V+ PLATFORM
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper presents an event-based system/architecture for the control of Adept robots. The system is written in Python and is based on Python coroutines and the asyncio library.

The introduction and background sections of the paper is nicely written and give a good overview of event-based vs. periodic control, events in computational systems, event-based middleware, and related work on software for building complex robotic systems, e.g., ROS and YARP.

The language is good throughout the paper. However, some minor points need to be addressed: Words like 'the', 'a', and 'an' are missing in many sentences. A native English reader should go through the paper.
Experimental design: An application use case of the system is presented. A simple vision-based inspection system is implemented using OpenCV, where the robot should pick up an object and present it in front of the camera.

The main part of interest in the experimental section is the presented listings in the section “coroutines”. This way to program robots is quite nice.

The section on time measurements show that the system is capable of non/soft real-time control, however, the lack of comparison with a similar implementation in, e.g., ROS, makes it impossible to see if the proposed system performs better or worse than that.

I would have liked to see a comparison of the presented system with an implementation of the system in ROS. With the authors' obvious knowledge of robots and Python programming, I suspect that this should not take them longer than a couple of days. However, this is not a requisite for publication, in my opinion.
Validity of the findings: The presented way to program robots using Python coroutines is as mentioned quite nice. However, I do not see why researches should switch from using ROS to this system as the paper stands now. This could have been clarified with a direct comparison with an implementation in ROS.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVENT-DRIVEN INDUSTRIAL ROBOT CONTROL ARCHITECTURE FOR THE ADEPT V+ PLATFORM
Review round: 1
Reviewer: 2

Output:
Basic reporting: This manuscript proposes an architecture that aims to facilitate the interaction between a robotic system, control modules and other external systems. The architecture (methodology) is clearly justified based on the state-of-the-art, the authors provide a strong background supported by literature. A case study is presented as an example of how to implement the architecture in an industrial environment. The language used is appropriate and the content is technically correct.
However, there is a heavy use of acronyms in the manuscript, which is understandable based on its content. I suggest that the authors include a list of abbreviations at the beginning of the manuscript to make it easier for the reader to understand some of the concepts, I suggest some changes in order to help the flow of the paper (SEE GENERAL COMMENTS).
Experimental design: The research conducted follows the Aims and Scope of PeerJ Computer Science. The introduction is strong and clearly describes the research problem that is addressed in this paper. It also describes the knowledge gap and the need for an architecture that solves the problem.
The conducted research is reproducible and the source code is accessible online. Additionally to this manuscript, I encourage the authors to add relevant documentation along with the code. A user’s guide with a case study in tutorial format with examples would be useful for anyone who might be interested in using this Python library. It does not affect this publication but I wanted to take the opportunity to remind the authors to follow good practices when sharing code and the library documentation is missing in this case.
Validity of the findings: The architecture provides a structure that connects multiple elements while easing the communication between them in a robotic environment.
See minor comments on the “system architecture” and “application use case” in general comments. The weakest element is found in the lack of context for the 32 test runs; I highly encourage the authors to document the experiment with a detailed context.
I also suggest trying to restructure the “discussion” and “summary and further work” since I found that some concepts are repetitive and could be merged in one single section: “Conclusions and further work”. An intuitive structure of this section would be: (1) Summary, (2) Architecture highlights, (3) Achieved improvements from existing platforms, and (4) Further work
Additional comments: 1. Introduction:
i.	There is not need to introduce ROS in the introduction since it will not be used in the architecture.
ii.	Make the objectives and contributions clearer. For example, line 95-96 are contributions and should go with the text that finishes in line 74
2. Background:
i.	Line 97: reference the software
ii.	Lines 101-108: a figure that comes along with this paragraph would be helpful to understand the programming approach
iii.	Lines 109-119: again, a figure would be useful. It is hard to follow the text without a diagram that shows how the elements are interconnected
iv.	Lines 135-141 and algorithm 1: some variables are not defined. I suggest to at least give a reference in the text or add comments on the algorithm of what each line does
v.	Line 148: please add reference to the AsyncIO Python library documentation
vi.	Line 198: a reference to the conducted study is missing
vii.	Line 202: there is no need to introduce ROS in the introduction since it is described here. I would also remove the reference to ROS2 since is not used in the paper. The mention of ROS2 in future work should be enough
3. System architecture:
i.	Line 257: please define what those 6 arguments are. I assume that they are x, y, z and roll, pitch, yaw but they have to be defined with the example similar to lines 271-273.
ii.	I would move the examples in page 6 under “Components and general principles” (after line 253) to the “communication protocol” section. It makes more sense since the commands are explained there.
4. Application use case
i.	Line 395: typo: it is Figure 4
ii.	Lines 412-421: please relate to the corresponding objects and function calls from Figure 6 when explaining how the algorithm works.
iii.	Line 422: although Figure 8 might seem relevant to support the algorithm introduced in Figure 6, it is not relevant for the purpose of the paper. The graphs axes are not labeled and I cannot see how it demonstrates the sharpness measurement routine. Please consider to explain the figure in more detail or remove it.
iv.	In case that the authors decide to keep Figure 8, it should be placed before Figure 7 in the manuscript.
v.	Page 15: the equations must be numbered on the right side of the line. Please indicate what each element in the equations mean; most of them are not formally defined.
vi.	Line 525: the conducted experiment needs more context: why 32 runs? how was the experiment conducted? randomly? what are the ranges of appr_speed and delta_z?
vii.	Line 528: please indicate what variable is each
viii.	Line 532: Please indicate what variable is each
ix.	Figures 11 and 12 need more explanation. What does that particular distribution mean to the overall architecture?
x.	Line 537: I suggest  moving the “composition of coroutines” after the coroutines are defined and before the time measurements study
5. Discussion
i.	Line 555: I think it would be make more sense to place “similar work” under the background section as a subsection. The last paragraph can be added to conclusions to highlight the improvements made to existing research


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVENT-DRIVEN INDUSTRIAL ROBOT CONTROL ARCHITECTURE FOR THE ADEPT V+ PLATFORM
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: EVENT-DRIVEN INDUSTRIAL ROBOT CONTROL ARCHITECTURE FOR THE ADEPT V+ PLATFORM
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: The research concepts and conducted work were valid and strong from the initial version. These minor changes help the reader better understand the contents of the paper. I considered valid the authors' decision of no re-structuring the last part of the manuscript; it's just a writing style preference.
I highly appreciated that the authors followed my recommendation of adding documentation to the Python libraries; it didn't affect the manuscript but I believe it was relevant for the significance of their research.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPARISON OF THE PERFORMANCE OF SKIP LISTS AND SPLAY TREES IN CLASSIFICATION OF INTERNET PACKETS
Review round: 1
Reviewer: 1

Output:
Basic reporting: 1. Some typos, e.g., Line 59 freat--> great. A spell check should be applied to the document.

2. The paper talks about comparing Splay trees to skip lists for packet classification. The original paper by Sleator and Tarjan [1985, Self-adjusting binary search trees] that invented Splay trees should be cited.
Experimental design: 1. It seems like the Splay tree is always worse than Skip List. Are there any scenarios, perhaps artificially constructed, when they may be more suitable?

2. If Skip lists are always better, perhaps a theoretical analysis will prove this.

3. What if the authors used a static binary tree (no rotations)? This will demonstrate whether the problem is with rotation rule of the Splay trees, or with using a tree data structure in the first place.

4. Generally, skip lists act as the same as binary trees, so in case the Splay rotation rule is what causes the inferior performance, what about simpler rules, like move to root?
Validity of the findings: No comment.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: COMPARISON OF THE PERFORMANCE OF SKIP LISTS AND SPLAY TREES IN CLASSIFICATION OF INTERNET PACKETS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors present a comparative analysis of two decision tree implementations — splay trees and skip lists in order to determine which of these two are better suited for classification of internet packages. They find that skip lists perform better than splay trees in terms of package classification speed and memory accesses. They, however, note that the memory used by skip lists is higher than splay trees. Although based on Figure 5, it seems that they are not drastically different.

The paper is decently written, and is easy to follow. I would perhaps structure the tables and figures in a different way; at this point, they are at the end of the paper and completely detached from the experimental results section, which means I had to go back and forth, making it harder to read. Also, I do not see any reason why the table labels and the table itself have to be on different pages, given that both pages are more than half empty. Although, I am no expert in this area, the literature review does seem to be thorough. I find the citations to be a little strange, where the author names appear and again within parenthesis the last names with the year are given; I would think that providing just the year within the parenthesis ought to be enough. But these are all a matter in exposition and I would not dwell on them further.
Experimental design: My main concern is that there is no bench-marking with respect to the existing techniques for package classification. I believe that this being an important problem, there must be other well established techniques. Therefore I feel such a comparison analysis is almost imperative.
Validity of the findings: The other issue is the lack of novelty in the paper. It simply compares two decision tree implementations based on memory footprint and classification time.

In most practical scenarios skip lists often outperform most balanced search trees, and that “worst case” scenarios are typically infrequent in probabilistic data structures (such as skip list). Hence, I do not see why we need to use package classification to differentiate between the two. Is there some structure to package classification that I am missing? If there is, it would be beneficial if the authors could elucidate that clearly.

Additionally, splay lists attain O(log n) in the amortized sense. Have the authors considered this for the splay tree structure? Have they considered other form of (strictly) balanced search trees?

Lastly, I see that the authors have talked about parallelization of the algorithms. Skip lists are better for parallel implementation and writing multi-threaded programs (than most balanced search trees) as one has to only obtain the lock locally when it comes to modification or search. In most (balanced) search trees, locking is painful mainly due to re-balancing/splaying. Hence, I do not see why the outcomes over there would be any different (i.e., skip lists would still vastly outperform splay trees). Having said that it would have been a welcome addition to the papers as it would have boosted the contribution.
Additional comments: Overall, I feel that for the paper to be accepted, the authors must address the above concerns.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SAMPLE-LEVEL SOUND SYNTHESIS WITH RECURRENT NEURAL NETWORKS AND CONCEPTORS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Overall, the English use is good. The main problem is the readability of the paper. It is very lengthy and more like an experimental report. The paper needs to be reorganized and needs to be more concise.
Experimental design: I have major concerns about the metrics used in the paper, as well as about the lack of baselines. Please see my comments below.
Validity of the findings: I think the applications can be described more clearly.
Additional comments: In the Manuscript, the Author studies generative audio modeling using recurrent neural networks under conceptor control. I think the material is interesting and novel, but there are various fundamental missing points for the paper to be accepted. Below are my detailed comments:

- The Abstract is very vague and general. Please state contributions more clearly and quantitatively, e.g. emphasizing on some of the results on synthesis accuracy.
- I couldn't understand what imposes sparsity in the weight matrix W, please explain.
- Please formulate the operation of scaling with the described factors precisely.
- Please describe the washout superscript in L.
- It is common to solve regression using gradient-descent based methods to avoid the high complexity of matrix inversion. I suggest the Author to consider and discuss this aspect.
- Please define the notations for linear algebra operations like transpose explicitly.
- For sound, mean-squared error on the raw time domain waveform has a very low correlation with the perceptual quality. For example, phase errors causing a shift in the waveform can significantly increase this error, but does not change the perceptual quality much. I suggest the Author to consider this point and evaulate the synthesis quality with other metrics that are commonly used in the literature, like the spectral errors defined on spectrograms.
- There are no baselines for the results, so the efficacy of the proposed RNN architecture is not convincing. For example, can a standard RNN without conceptors achieve a similar quality?
- The results section is very lengthy, both in terms of figures and text. For example, what is the significance of Fig. 3? I suggest the Author to reorganize the section in a more concise way.
- Sound morphing is described but the quality of the morphed waveforms is not evaluated. So it is not convinving whether the proposed technique can synthesize realistic artificial sounds.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SAMPLE-LEVEL SOUND SYNTHESIS WITH RECURRENT NEURAL NETWORKS AND CONCEPTORS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The basic reporting meets the standards for PeerJ.

This paper presents two related sound synthesis methods based on "conceptors", a mechanism for learning and then manipulating subsets of the reservoir used in  RNNs to create different attractors, and this different pattern generators. The author calls the two methods are conceptular synthesis, and "the conceptillator"

This is an interesting direction of research since recent and ongoing developments in neural networks naturally hold great promise for contributing to sound synthesis. The references and landmarks cited to orient the current work are thorough and appropriate as is the background discussion of Reservoir Computing and Echo State networks.

The organization needs some work, and there are a few issues that need to be addressed to improve clarity (see General Comments to the Author).
Experimental design: The 5 experiments reported are in line with the objectives of the paper. The goals and scope are clear, and the work is carried out well, source code is provided, and (very importantly) sound examples are provided.
Validity of the findings: I think the paper is pretty forthcoming about both the potential and limitations of the synthesis technique. I really look forward to hearing some morphs between complex sounds (even just the "oo" from the spoken "two" and square wave would be interesting).
Additional comments: The paper could be improved for clarity, although some of the questions that arise in the mind of the reader early on in the paper do get clarified later and on subsequent readings, but it would be possible to significantly increase clarity for the first time reader by moving some of the explanation of terms and the discussion of the how the system is used to generate sound up towards the front of the paper.

The clarity in the equation section, "Basic Models" could be improved:

Why the extra text of parens in both parts of the sum in equation 2?

The word "target" in eqs 1-3 is unexplained. I guess it just means the value of the state nodes prior to their alpha decay, but why should your reader guess? Also, "target" is so commonly used to signify a desired output for training, that it is a little confusing here if left unexplained.

123: The nodes are sparsely connected with a probability of 10.0/N in weight matrixW (of size N x N).
   Why 10/N? It's a strange and arbitrary number which also doesn't really make sense as N gets small anyway. And how big is N typically? Is it chosen arbitrary? Is this what you later refer to as "the number of nodes in the RNN"?

135, 138: "loaded in to the model" - does this mean "used to train the model"?

144: What is L? What does "washout" mean? How big is L^(train) typically? What is the significance of different values of this parameter?

158: eq(6) is L superscript(train) different from L subscript(train)?

160: S^j is "modified"
     Why? Intuition?

161: \alpha  is the aperture of the conceptor
	Is this different from the \alpha used for the leaky integration from line 131 and eq(2)? If so, better use different notation.
	Can you give some words to help with the understanding of the term "aperture" here?

eq(12) - are C and alpha both subscript on 'a' here? So for any C (based on the SVD) and aperture \alpha, you get an 'a', a minimum value of attention which is used ..... how and where?

192: [The percussion sequencer website] On my fast university connection, it too about 12 minutes for the web page to load. Then with the default 32 steps, and 36ms step length, I hit render and waited, but gave up after about an hour (from the stack-of-empty-boxes "curser" I think it was about 30%% done by then. Maybe rendering a few examples on your side an then showing the pre-rendered samples on the web page would work better?

213: "number of nodes in the RNN" - do you mean in the reservoir? Is this the "N" referred to line 123? The first time you mention N in the equations, you could note that you will refer to this number as the size of the network throughout the rest of the paper.
...

218: "loaded in" - I am unclear on what this means exactly.
   A high-level talk through would help. First the system is initialised by "loading in" (once we know what that means) M grains that define the space of grains that can be generated. Then the reservoir is manipulated (how exactly - or is loading grains the way they are manipulated) in order to control which attractors are used when the RNN generates it's next grain. Then the reservoir is manipulated again to deliver another grain and so on until we have the number of grains desired to synthesise a sound of a certain length" (or something of that nature).

231: Which parameters are hand-tuned? (I think this becomes clear only later in the paper)

239: "The first 100 audio signal in set a were used to calculate a model" - What are the rest of the samples in the set used for? Or is this because you are representing a single of length 100*mu = 1500?

239: I don't understand the purpose of Figure(3), and line 239 doesn't say anything about it other than that it exists.

Equation 13 for \alpha.sub(scaled) is strange - why not simply say that alpha can be changed over time to control the weight between the current and previous target.
Equation 13 also has an unnecessary layer of parenthesis.

306: "The weight scaling scaleW parameter is a multiplier"
	I don't see a scale.sub(w) anywhere - what are you referring to?

336: Are you sure you want to overload the parameter mu (already used for sample length above)?

398:  I'm confused how L(washout) and L(train) are expressed in terms of mu, because mu isn't constant for this example. Are L(washout) and L(train) also not constant?

455: A phase shift hardly seems like a problem since it would by inaudible in an A/B comparison between the two signal.

457: "driving audio signals begin to alternate between two frequencies"
	Alternate? In time? Clearly the voiced signal is harmonic and made up of multiple frequencies, but I don't understand what you mean by alternating frequencies. Do you mean that your zero-crossing algorithm started alternating between two different signal lengths for segment modelling? Ah - that must be it. I think this could be explained more clearly.

491: the value of scale.sub(alpha) is meaningless without knowing what the original \alpha was - and did \alpha stay inside [0,1] with the changing scale parameter?

Figure 22: The x-axis needs labelling
Figure 22:  - The heat map for these spectrograms doesn't correspond very well to what one hears - nor what what one sees in the wave file (S13). A different colour map (of the equal-luminance variety?) may correspond better to perception.

511: "tone"
	timbre?

511: variation in Wscaling.
	Wscaling in what equation?

526, 548: Scale.sub(w) from what equation?

608-609:  generative generation ....

It's great that you provide visual and audio examples to help the reader asses the output.