Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEGER PARTICLE SWARM OPTIMIZATION BASED TASK SCHEDULING FOR DEVICE-EDGE-CLOUD COOPERATIVE COMPUTING TO IMPROVE SLA SATISFACTION
Review round: 2
Reviewer: 1

Output:
Basic reporting: 1.	Previous comments regarding the abstract have been incorporated, however, it is recommended to start your abstract from an introductory sentence (as in the previous one) instead of a sentence like “In this paper”.
2.	Authors have provided quantitative results in the abstract and in the contribution section, however, the improvements shown in the different sections are different. Moreover, in the abstract, the term “x” is used while in the contribution section the symbol “%” is used, correct them or justify if it is different?
3.	In the last paragraph of section 1 “introduction”, Section 6 has mentioned two times?
4.	The symbol “Z” and “U” used in equations 17 and 19 have not been explained in the text.
5.	In response to previous comments “Do you think inertia weight has a role in PSO algorithm? Which variant of inertia weight you have used for your algorithm. Detail about inertia weight and other terms like r1, r2, c1, c2 not given (lines 212-213)”, it is mentioned that it is out of scope of this paper.  I agree up to some extent that you are not working the inertia weight strategies but Inertia weight is one of the most important control parameters for maintaining the balance between the global and local search of the PSO, which can affect the performance of the PSO.  Moreover, the values of constants like c1 and c2 also matter.
Therefore, it is suggested to use the latest variant of inertia weight strategy or at least refer/cite some latest papers in the text so that readers can explore them if interested to follow the details, some of the latest papers in this regard are given below.
https://doi.org/10.1007/s11227-021-04062-2.
https://doi.org/10.1007/s10586-019-02983-5
Experimental design: N/A
Validity of the findings: N/A
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: INTEGER PARTICLE SWARM OPTIMIZATION BASED TASK SCHEDULING FOR DEVICE-EDGE-CLOUD COOPERATIVE COMPUTING TO IMPROVE SLA SATISFACTION
Review round: 2
Reviewer: 2

Output:
Basic reporting: The language used in the paper is very poor. The language of the whole paper still needs to be improved.
Experimental design: The reviewer has not incorporated changes to the article based on my first comment
 "The authors claimed that his approach has better resource efficiency. However, the results
concerning resource efficiency are not meaningful. The authors are required to obtain results
concerning Average resource utilization which will be between 0 to 1 or 0 to 100 %.  "
Validity of the findings: The authors have claimed almost 10 times improvement against the state-of-the-art approaches which seems incorrect. The authors are required to perform the experiments again and see whether the improvement claimed is same
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ACCELERATING COVERING ARRAY GENERATION BY COMBINATORIAL JOIN FOR INDUSTRY SCALE SOFTWARE TESTING
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is very difficult to understand due to many spelling and grammar mistakes:

Spelling mistakes on line 24 (compbinatorial-> combinatorial), line 144 (chat->that), line 617 (movers-> moves), and line 834 (Bbut->but).

The listings in the paper are not numbered and referred. A good way is to cross-reference the listings such as use ”Listing X shows something” instead of “in the listing bellow”.

On line 930, the actual percentages are missing and instead question marks are shown. It could be that the authors forgot to write the actual numbers.

Why is “weaken-product based combinatorial join” in quote marks?

“There are three streams of methods 47 have been studied” should be “There are three categories of methods that have been studied”.

 Many of the figures are also too small to be readable (e.g., Figure 1, Figure 14, 15 etc.).

The obtained data is provided, but I expected a more thorough experimental setup to be provided, including how the data should be obtained from these tools, the input models used and so on. In this form, the results are not really replicable.
Experimental design: The experimental study relies only on some rather simplistic examples. Hence, whether the results can be generalized remains open to some extent. An extension to include more examples/higher strengths and input space models of larger size would be highly recommended. I am missing the results for PICT in Section 5. Causality and dependence between different variables is not thoroughly investigated.

Research questions should be rewritten since these contain mistakes and I am more confused after reading them. For example, the first research question actually contains two sub-questions. The first question is binary in nature, which seems quite simplistic given the limitations of such an investigation. No relation between these proxy measures of test efficiency and actual cost measures are given. Some useful advice can be found here: http://www.cse.chalmers.se/~feldt/advice/guide_to_creating_research_questions.pdf

The structure of the evaluation methodology and results sections could benefit if these are explicitly aligned with the structure proposed in the guidelines for conducting case study research in software engineering by Runeson et al. who considers for instance sections on the data collection procedure and the data analysis procedure (which seem to be implicitly mentioned in the paper).

The threats to validity section needs to be extended with regard to:

Conclusion validity. Does the treatment/change we introduced have a statistically significant effect on the outcome we measure?

Internal validity. Did the treatment/change we introduced cause the effect on the outcome? Can other factors also have had an effect?

Construct validity. Does the treatment correspond to the actual cause we are interested in? Does the outcome correspond to the effect we are interested in?

External validity. Transferability Is the cause and effect relationship we have shown valid in other situations? Can we generalize our results? Do the results apply in other contexts?

Credibility. Are we confident that the findings are true? Why? Dependability Are the findings consistent? Can they be repeated?

Confirmability. Are the findings shaped by the respondents and not by the researcher?

The relationship between the independent and dependent variables is not really explained in the context of the research questions. Would be good to explain the stated hypotheses on test efficiency and effectiveness. How about the time needed to generate the existing covering arrays that are used for the combinatorial join?

The experiment seems planned in a very ad-hoc manner. For example, the execution time measures are rather unscientific, and there is no possibility for any causality or actionable results. Potentially, the algorithm could be useful and the questions are interesting. However, IMHO it's unfortunately back to the drawing board for designing and reporting the experimental evaluation.
Validity of the findings: The generation of covering arrays is an interesting area of research in combinatorial testing. This paper is proposing an algorithm to accelerate the generation of covering arrays using joins. The algorithm takes two existing covering arrays of the same size and a strength t, to perform the join “operation”. First, the strength of the input covering arrays is reduced, and a cartesian product is taken between them. The final covering array with unused rows is then merged to obtain the output. The algorithm is then evaluated based on the time taken, size of covering array, constraints handling capability, and capability of variable strength covering array generation. The paper also discusses the potential oracle reuse, enabled by the proposed algorithm.

Covering array generation is an open area of research. Authors have done a good job of explaining how the approach can be used with a diverse set of CIT tools for the input covering arrays generation. The evaluation includes most generation scenarios. Generally, the paper covers well the background of the area. The authors explain quite well the approach, but there are major points that must be addressed.

While RQ1 shows promising results, the answers to the other research questions are not convincing. For example, RQ2 clearly shows that if t is 2, the size of the covering array generated by the proposed approach is at least 47% larger than the other approaches. If t is 3, the size increase is dramatic (around 264% at least). The authors’ argument in explaining this finding is that test execution is automated anyway. Even if the approach reduces the generation time by 11 to 88% (which would be a few minutes), this could add hours of test execution time.

Section 5.3 discusses how this approach can help in reusing test oracles for three kinds of faults. Concrete examples are needed here for each of the defects and for which oracle can be reused. Also, the answer to RQ3 is not justifiable unless some results are provided demonstrating that reuse of test oracle using combinatorial join has indeed caught new bugs.

The algorithm assumes that the covering arrays have the same number of columns, which makes the approach impossible for many realistic scenarios, e.g., regression testing scenarios or evolving software. The practicality of this algorithm needs to be discussed.

The creation of basic and basic+ baselines is questionable. For example, how did the authors end up with formulae 16 and 17 for defining the constraints? How are these constraints really representing the constraints in some real applications? The algorithm also does not handle inter-covering array constraints that might limit the applicability of the approach.

The running example for the demonstration of the algorithm considers two covering arrays with the same values and parameters. It makes sense to use a better example with different values in the covering array.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ACCELERATING COVERING ARRAY GENERATION BY COMBINATORIAL JOIN FOR INDUSTRY SCALE SOFTWARE TESTING
Review round: 1
Reviewer: 2

Output:
Basic reporting: 1) This paper seems to be an extension of the authors’ previous paper (Ukai et al. 2019), but the difference is unclear. The authors should better discuss and highlight the novelty of this study.

2) There is no background of covering array (e.g., definition, notation, example, etc.), as well as the constraints in combinatorial interaction testing. This paper should include them to make it self-contained.

3) This paper discusses the different modelling languages used by different tools to describe constraints. I do not think the proposed algorithm makes a contribution on this aspect.

4) Figures 1 and 2 are hard to read; the p_{i,k} notation in Equation (16) is unclear.

5) This paper is too long. For example, the experiment design is too verbose; Figures 3, 4, 5, and 6 are unnecessary.
Experimental design: 1) The largest problem instance used in the experiment has only 200 parameters, which is far away from the “industry scale” that might have thousands of parameters.

2) There are many real-world benchmarks for covering array generation, but the experiment includes synthetic problem instances only.

3) The experiment investigates coverage strengths two and three. The reduction of covering array generation time is more important and beneficial for higher strengths, say five and six.

4) As discussed in related works, there are also algorithms that seek to reuse existing covering arrays to generate new arrays. The proposed algorithm is not compared with them in the experiment.
Validity of the findings: 1) The proposed algorithm assumes that there is no constraint across pre-constructed covering arrays. This is a very strict assumption, so the applicability of the proposed algorithm in the real world is limited.

2) The results are not very promising. Although the proposed algorithm can reduce the time cost of covering array generation, it produces much larger arrays, and accordingly, much higher testing cost.

3) There is no data to support the claim of the answer to RQ3.
Additional comments: This paper addresses a very important topic in Combinatorial Interaction Testing (CIT). The proposed algorithm, which seeks to reuse existing covering arrays, could be an interesting direction in the field of CIT. However, this paper is too long; further improvements are also needed for the experiment design and discussion of results.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ACCELERATING COVERING ARRAY GENERATION BY COMBINATORIAL JOIN FOR INDUSTRY SCALE SOFTWARE TESTING
Review round: 2
Reviewer: 1

Output:
Basic reporting: Even if the README file is useful, I could not find the suggested zip file, but only a latex project. I was unable to replicate the results. The README should also include details about the CASA benchmark and how one could use your approach on these.

Since the last review, several readability improvements have been made and some of the comments have been satisfactorily addressed. Would be good to emphasise in the summary of contributions the results related to RQ2 as you do for RQ1, RQ3, RQ4. Still, the listings in the paper are not numbered (e.g., Listing 1) and references in the text.

There are still places where I found writing issues. Another thorough proofreading would be helpful for fixing these issues (articles, punctuation). Figures 6 to 11 can be subfigures part of the same figure on scratch generation. The same for the other figures related to VSCA generation and incremental.

Just a few examples:

“superrior” should be “superior”

“ with a focus in their 186 different characteristics in defining constraints” should be  “with a focus on their 186 different characteristics in defining constraints”.

“can be expressed using the < and negate operator” should be “can be expressed using the < and negate operators”.

“based combinatorial 595 join approach.” should be “based combinatorial 595 join approach:”
Experimental design: Improvements have been made to include input space models of larger size and the results for PICT. Substantial changes have been regarding the research questions. Substantial evaluation experiments have been made using the CASA benchmark. Unfortunately, not too many details about this benchmark are given. Also, the link in Shaowei Cai (2020) does not work. Shaowei Cai (2020). A description of this benchmark and some “size” metrics should be given in order to assess how realistic these benchmarks are.

In Section 4 one should give more details on the data sets used in terms of size, how relevant they are, variability and so on. A specific section on the selection of the data sets should be included.

Several improvements have been done in the threats to validity section.
Validity of the findings: Good clarifications are given and I appreciate the improvements made to the algorithm as well as the discussion related to a trade-off analysis between test execution time and the number of test cases. The examples of oracles classes are useful in the discussion of fault detection capability.

Would be good to clarify how the two approaches outlined in 4.3.1 Generation Scenario are useful in practice. When would an engineer using your approach benefit from generating a CA from scratch incrementally? Clarify the statement “Our tool makes reuse of oracles with higher possibilities”.

Section 5 could benefit from better explanations and interpretations of the results shown in each Table and Figure.
Additional comments: I notice with satisfaction that most of the comments made in my first review have been taken into account. Nevertheless, there are still some issues that remain to be resolved.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ACCELERATING COVERING ARRAY GENERATION BY COMBINATORIAL JOIN FOR INDUSTRY SCALE SOFTWARE TESTING
Review round: 2
Reviewer: 2

Output:
Basic reporting: The presentation of the revised version is now clear and self-contained.
Experimental design: The experiment design is improved. Especially, real-world problem instances are now included.
Validity of the findings: The potential threats to validity have been well discussed.
Additional comments: The authors have paid good efforts to address my previous comments, and I think this paper is now ready for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ACCELERATING COVERING ARRAY GENERATION BY COMBINATORIAL JOIN FOR INDUSTRY SCALE SOFTWARE TESTING
Review round: 3
Reviewer: 1

Output:
Basic reporting: Since the last review, several improvements have been made.
Experimental design: I notice with satisfaction that most of the comments related to the experiment design have been taken into account.
Validity of the findings: The paper now appears as a well-rounded piece of work.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROBUSTNESS OF AUTOENCODERS FOR ESTABLISHING PSYCHOMETRIC PROPERTIES BASED ON SMALL SAMPLE SIZES: RESULTS FROM A MONTE CARLO SIMULATION STUDY AND A SPORTS FAN CURIOSITY STUDY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The article under review tries to compare the performances of the PCA and autoencoders in the field of questionnaire analysis.

 Although the work seems serious and well done
I have some reservations about the general form of the article that made its reading difficult.
Plus the exact definitions of the ento-encoders (see bellow).

Some definitions are not clear and a mathematical clear definition:
-  What is communality and non-cummunality (see references problem bellow)
- The formula of line 325 (and in general all the metrics used) should be clarified as one wonders if the sum is running across the data-set or across the dimensions of the data.
- In section 2.4: what does the term "increased by" refer to? n is the number of samples, what is there to increase.

 In general a clearer mathamtical presentation is needed.

 I'm also concerned by the exact definitions of the autoencoders: In the colab

   First many references are not listed in the reference section (for example Forcino (2012)  Manjarres-Martinez et al. (2012) can not be found in the reference section). Some citation follow the form [number] and other follow the form "Author (Year)".

 It is important to fix this citation problem.

 Likewise, some figures are mis-referenced (Figure 8 in line 403 is inexistent and surely the authors meant figure 6 in line 390).

 There is a bulletpoint in line 295 that has no text.
Experimental design: My main concern is about the definition of the auto-encoders: In the code submitted the activation are "linear" this means that the autoencoders boils down to an affine transform. How can an affine transform do better than the PCA in termes of MSE? I think this is because non linear auto-encoders overfit on the small number of samples used for training.

 Judging by the code submitted, what is called deep-autoencoder has only one layer more than the autoencoder. This is not a really deep architecture.  In the code submmited the deepautoencoder has two times 3-size layers while the text mesion one layer with dimension 6.

 I also encourage the authors to disclose the data generation code (for synthetic data)
Validity of the findings: Pending the experimental clarification about the architectures I can not comment on the validity of the findings.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROBUSTNESS OF AUTOENCODERS FOR ESTABLISHING PSYCHOMETRIC PROPERTIES BASED ON SMALL SAMPLE SIZES: RESULTS FROM A MONTE CARLO SIMULATION STUDY AND A SPORTS FAN CURIOSITY STUDY
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper compares PCA and Autoencoder as dimentionality  reduction techniques, aiming at showing the superiority of autoencoders in case of small sample sizes  (and departures from normality)  in the context of psychometric survey data
As a preliminary observation, the title of the  paper seems not appropriate since, its main content is a MC simulation study and only an application to psychometric data is presented.  Moreover, this  real data example is based on a single random sample  draw from the original data set .  The authors should either consider more real data  sets or resampling several times form the single one considered  averaging on the results also,  the results on the whole data set (400obs) should be reported.
When applying the PCA, since the items are ordinal (the indication of the scale is missing…) I suggest to employ Polychoric correlations.

In general, the paper lacks definitions and details. While PCA is a very well known multivariate method , NN are in general considered black boxes , several structures are possible and none of the considered architecture are displayed in the paper, making very difficult to understand them and their  differences.

Note that there’s no figure 8 in the paper and that citations need to be fixed as the follow different citation standards. Also, on. Line 295 there’s an extra bulletpoint
Experimental design: The three correlation structures are designed to mimic a 3 factor solution organizing 15 items into block of 5 showing the same correlation. For the high communality for example, the value of 0.8 and then always 0.1 for cross blocks items. I think that this design is too extreme and far from any real situation.
In section 2.4 (sample size)  don’t understand what the sample size increments are meant for.
Validity of the findings: The paper is not clear enough to let me judge  on the validity of the findings.

Architectures of the  NN should be displayed and suggestions about the real data application followed to facilitate the reading and the evaluation
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: ROBUSTNESS OF AUTOENCODERS FOR ESTABLISHING PSYCHOMETRIC PROPERTIES BASED ON SMALL SAMPLE SIZES: RESULTS FROM A MONTE CARLO SIMULATION STUDY AND A SPORTS FAN CURIOSITY STUDY
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have satisfactorily responded to all my questions .
From my point of view, the paper is now suitable for publication.
Experimental design: The authors have modified  the experimental design according to my remarks
Validity of the findings: In this new version the validity of the findings emerges  in a clearer way.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BOTS IN SOFTWARE ENGINEERING: A SYSTEMATIC MAPPING STUDY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The introduction provides a good, generalized background of the topic Bots in Software Engineering that gives an appreciation of the wide range of applications for this technology. However, the  motivation for this study need to be made clearer.
Experimental design: The survey methodology is consistent with a comprehensive coverage of the subject.
Validity of the findings: It may be worthwhile to mention the tradeoffs involved in choosing the inclusion/exclusion criteria  as opposed to other criteria to filter out irrelevant articles to the mapping study
Additional comments: Overall, this is a clear study manuscript that has implications for the development of  Bots in Software Engineering


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BOTS IN SOFTWARE ENGINEERING: A SYSTEMATIC MAPPING STUDY
Review round: 1
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: In this Paper complete information related to bots with empirical data provided.
it is a good source of information for developing new bots.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: BOTS IN SOFTWARE ENGINEERING: A SYSTEMATIC MAPPING STUDY
Review round: 1
Reviewer: 3

Output:
Basic reporting: In general, the article is written in a clear and reproducible manner. There is an overall storyline, which supports the reader. Besides few minor issues, however, the report lacks information in two key aspects.

Minor issues:
- There is an inconsistency: the cover page abstract talks about a systematic literature review, while the rest of the paper talks about a systematic mapping study (more on that in the research design section).
- Since this article does not have any page restrictions, authors should avoid the heavy use of abbreviations, e.g., SE, NL PR etc. Furthermore, it happens fairly often that, when abbreviations are used, no spaces have been left. For example, p. 2, l. 55: “study(SMS)”. Authors should thus carefully read the text and fix those issues.
- In section 1 (introduction), authors should provide a bit more structure. A structured introduction with subsections “objective”, “problem statement”, “context”, “contribution” would help better support the reader getting key information quickly.
- Also, in section 1, authors should provide a subsection “outline” in which they guide the reader in terms of what can be found where in the article.
- Fig. 2 is of limited use as the authors do not provide numbers at the bars. Hence, this whole figure can be used as a trend, yet, authors should add the numbers to the bars and also to the trendline in order to provide details required to understand/follow the text.
- Fig. 3 and Fig. 4: Authors should revise these figures. In their current form, especially with the color scheme applied, the figures are (in the best case) hard to read (in a gray-scaled printout they are unreadable).  If I may, I’d suggest converting Fig. 3 into a tree to better visualize the topics and hierarchies. Fig. 4 should be converted into simple tables.

Critical key aspects: Even though the general reporting is of good quality, the article suffers in two key aspects:
1) Related work: the related work should be revised substantially. About a third of the related work is only concerned with pointing to an ICSE workshop and dropping the 3 categories discussed on this workshop. It continues with collecting further terms, but the related work does not provide a detailed description. I wonder if something like Fig. 5 would be beneficial here. In particular, I miss a clear approach in setting the scene in the related work. Few aspects are mentioned in the introduction. However, I would expect a clear background and terminology building in this section, i.e., authors should not only name, e.g., “code bots”, but should also explain what they are, which tasks they do, etc. Long story short, I consider the related work insufficient and suggest the authors to create the two subsections “background” and “related work”, such that the scene is set, the terminology is introduced, and the state of the art is properly presented. Finally, authors should discuss gaps in the field to provide an argument why the study is necessary (more on that in the study design section).

2) I apologize for the clear and hard statement, but the research design is poor. Everything about the research design seems to be located in section 3. This section needs to be completely revised. Details on this in the study design section of this review.
Experimental design: The study design is the weak point of this paper. In the submission-system abstract, the authors state to report a systematic literature review (SLR). In the actual paper, authors talk about a systematic mapping study (SMS). Unfortunately, the paper does not allow for judging which of these study types have been applied. The results section reads like an SLR. However, if the authors conducted a mapping study, key aspects are missing, notably:
- A classification schema
- A collection of maps
To make my point clear, SMS and SLR both have different targets. An SMS is aimed to explore a domain of interest and to provide an overview of the field by classifying the publications, while an SLR explicitly aims to generate new knowledge from analyzing a publication body. In the current version, neither of these goals is properly addressed.

Moreover, the research design is weak missing essential parts. Let’s start with the research questions. On page 2, l. 98, authors state “We surveyed the research articles pondering upon these three research questions” – however, even with a text search I was not able to find the research questions. This is the show stopper. Since I don’t know what the research questions are, I just cannot evaluate the paper properly and, therefore, I cannot provide much of a review of the findings (see respective section).

Furthermore, the research design lacks important information:
- Rationale for the study
- Search queries – the construction procedure/rationale is missing
- Search strategy: incomplete and inconsistent. Google Scholar is still subject to debate whether or not it is a proper (meta) search engine. An argument why “traditional” literature databases have been excluded from the search is, however missing. The search strategy is also inconsistent described since the snowballing is kind of located after the in-/exclusion, which is kind of “creative”. This decision was, unfortunately, not explained.
- Paper selection: while in-/exclusion criteria are listed, the actual paper selection procedure is not explained.
- The data extraction procedure is not explained.
- Validity procedures and the actual threats to validity are not discussed

Therefore, I suggest that the authors first clarify which kind of study the are reporting and, second, after having selected a study type, I suggest consulting the major references for executing and reporting these studies, i.e.:
- Kitchenham et al.: Evidence-Based Software Engineering and Systematic Reviews
- Petersen et al.: Systematic mapping studies in software engineering

I apologize again for my hard statement. Without this information, the article just cannot be accepted since the requirements for transparent and reproducible research are not met.
Validity of the findings: Well, I have to say the text in the results section is quite informative and, so, authors did a lot of work. However, since the research questions are missing, I just cannot seriously review this section. Since I just don’t know the exact questions, every review would be grounded in my personal interpretation. I kindly ask the authors for understanding that I cannot do this.
Additional comments: In general, the overall writing of the paper is good and I have no further comments to be added to my list in the basic reporting section.

My only general comment is that authors need to revise and probably rethink this article. It might happen that, after deciding on an actual study type, a considerable amount of rework is necessary, maybe including another data collection stage. Therefore, authors should critically discuss and revise their paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP CONVOLUTIONAL NEURAL NETWORKS FOR REGULAR TEXTURE RECOGNITION
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper provides a good and sufficient background of Deep CNN  that can classify regular and irregular texture images.  Though the approach is relatively simple to implement, yet it is quite powerful.

No significant limitations were discussed. It may be worthwhile to mention the tradeoffs involved in choosing the Deep CNN as opposed to some other technique

Professional English used.  No ambiguity
Experimental design: The experimental setup is quite standard, and is appropriate for the study, especially the evaluation and comparison of several different classical CNN models, but is the sample size taken for testing sufficient to represent a large database images
Validity of the findings: I think the motivation for this study need to be made still clearer. In particular, the connection between
(a) the necessary of recognizing and locating regular textures by giving  applications, and (b) the necessity of choosing Regular Texture Recognition
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP CONVOLUTIONAL NEURAL NETWORKS FOR REGULAR TEXTURE RECOGNITION
Review round: 1
Reviewer: 2

Output:
Basic reporting: - This paper proposed a new texture database that includes 2800 images (1300 regular and 1500 irregular images). The author proposed using five state-of-the-art CNN models; AlexNet, InceptionV3, ResNet50, InceptionResNetV2, and texture CNN (T-CNN), to examine the texture database. The result showed that the InceptionResNetV2 obtained the highest accuracy on the texture database with an accuracy of 97.0%.

- However, this paper did not show any novel method or did not show any complex experiments, such as experiments on insufficient data (10%, 20%, 30% of training set), proposed a new CNN method; Fusion CNNs, 1D-CNN, etc., or show the parameter tuning.
Experimental design: This paper doesn't show any new experiments. It presented only which CNN model performs better in the proposed texture dataset and show only one experiment (see Table 3). The author should do more experiments, at least three experiments. The author should concern about the novel method or exciting experiments.
Validity of the findings: This paper showed only which CNN model performs the best on the proposed database. Then, the author should provide a discussion section on why the InceptionResnetV2 outperformed other CNNs.
Additional comments: - The author should concern more about the novelty of this work. Provide only a new texture database was not sufficient to publish in the outstanding journal.
- The abstract section is relatively short. It did not include any new information about the proposed deep convolutional neural networks (CNNs). However, this paper provides a new regular texture database and uses only standard CNN in the experiments.
- The author should do more experiments, at least three experiments. The author should concern about the novel method or exciting experiments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP CONVOLUTIONAL NEURAL NETWORKS FOR REGULAR TEXTURE RECOGNITION
Review round: 1
Reviewer: 3

Output:
Basic reporting: There were no significant constraints discussed. It's probably worth mentioning the tradeoffs involved in using deep CNN instead of another approach
Experimental design: This article does not much new experiments over traditional. Only one CNN model performs better in the proposed texture dataset and show only one experiment. It should be compare with more methods see table 3 . The author should do some more experiments, atleast three experiments.
Validity of the findings: Entire article discussed only which CNN model performance the best on the proposed database. Then, the author should summarise why inception ResnetV2 outperformed other CNNs.
Additional comments: The author should concern more about the novelty of this work. Provide only a new texture database not much technical strength to publish in the outstanding journal.
The abstract section is relatively short should be only 300 words
The authors should do more experiments at least three methods.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP CONVOLUTIONAL NEURAL NETWORKS FOR REGULAR TEXTURE RECOGNITION
Review round: 2
Reviewer: 1

Output:
Basic reporting: The revised manuscript submitted  provided a good and sufficient background of Deep CNN that can classify regular and irregular texture images.
 The author also mentioned the tradeoffs involved in choosing the Deep CNN as opposed to some other technique
Experimental design: The new manuscript designed provided more complex experiments using 10-fold cross- validation and is sufficient enough to validate the approach
Validity of the findings: (a) the necessary of recognizing and locating regular textures by giving
applications, and
 (b) the necessity of choosing Regular Texture Recognition

The above 2 points has now been made clearer in the revised manuscript
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP CONVOLUTIONAL NEURAL NETWORKS FOR REGULAR TEXTURE RECOGNITION
Review round: 2
Reviewer: 2

Output:
Basic reporting: In this paper, the author proposed a new texture database collected from Flickr and google images, including 1230 regular textures and 1230 irregular textures. The author proposed to use fisher vector pooling-CNN (FV-CNN) to extract the local feature from the images. In the experimental results, three CNN architectures were evaluated; ResNet-50, InceptionV3, and Inception-ResNetV2. The result showed that the Inception-ResNetV2 outperformed all CNNs with an accuracy of 96% with 10-fold cross-validation. Figure 6, however, could the author shows both train and validation losses and train and validation accuracy. The author showed the corrected classification of regular and irregular textures in Figures 7 and 9. However, could the author show the confusion matrix? So, the audience could see all the correct and incorrect numbers. In addition, the result of the Inception-ResNetV2 and FV-CNNreg-IncepResv2 were 96% and 97% accuracy, respectively. Is it a significant result?
Experimental design: The author provided the experimental results of the FV-CNN. Also, a 10-fold cross-validation method was proposed. The author also compared the FV-CNN with the FV-SIFT method. However, the experimental result is not enough. The author should do more experiments as suggested before.
Validity of the findings: In the new manuscript, the author proposed the FV-CNN to classify the texture images. In the experimental result, the author showed that the FV-CNN outperformed the original CNNs and FV-SIFT method. So, could the author report the kernel of the SVM used in the experiment and report the hyperparameter values. Could the author add the "Fisher Vector" into the framework (Figure 5)? In the framework, it was unclear how the author attached the FV into the CNN?
Additional comments: - I suggest the author do more experiments, such as data augmentation techniques, tuning hyperparameters, etc.
- Would you please provide the Discussion section?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEEP CONVOLUTIONAL NEURAL NETWORKS FOR REGULAR TEXTURE RECOGNITION
Review round: 3
Reviewer: 1

Output:
Basic reporting: More concerning points are as follows.
- The author responded that the confusion matrix was added as Table 4. But, Table 4 showed the misclassified images by Inception-ResNet-v2.
- The author presented training and validation in Figures 6, 10, and 11. However, the quality of the Figure is relatively low resolution and see some noises. The legend should not overlap on the graph.
- In Figures 7, 8, and 9, what are the numbers under each pattern? The author should describe in the caption or in the text.
- Up-to-date references (2019-2021) are required.
Experimental design: The author clearly explained the experimental results. The experiments are cover the contribution.
Validity of the findings: In Figure 5, please check the block of Fisher vector pooling. Somethings appeared before the word "pooling."
Additional comments: - The author responded by adding the experiments of data augmentation. I found the words "data augmentation features" in line 234. However, I cannot see the compare results. Could the author recheck it?
- The discussions section was presented in this revised manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTIFACTS AND CITATIONS IN COMPUTER SYSTEMS PAPERS
Review round: 1
Reviewer: 1

Output:
Basic reporting: 1.1. Clear and unambiguous, professional English used throughout.
I find the English acceptable for publication. However, I suggest adding some definitions and unifying some concepts. I found it difficult to understand whether some expressions were used interchangeably or not, which turned out to make it difficult to realize the scope of the paper regarding the type of software included in the study. In particular,
* Is “computer systems” used as a synonym of “computer science”, “software”, “system software”, some of them or none?
* Does system software correspond to software used as a platform to develop other software including operating systems, software languages and/or utility software?
* Is the paper about system software in general or only system software created as part of a research process? Is the paper about software artifacts or research artifacts (which also include, for instance, data)?
* What is the field the author refers to? Is it computer science, computer systems, system software, research software?

Some typos:
* Line 11, “research’s results”, possibly no need of the possessive form
* Line 60, “are valued an are an important contribution”, possibly “and” rather than “an”
* Line 443 and 444, “many—-f not most—”, possible “if” rather than “f”
I suggest a full proof-reading to catch and fix those issues.

1.2. Literature references, sufficient field background/context provided.
The article is accompanied with a good number of references. However, the author should double check the citation information. For instance
* Line 472, reference ACM, URL not working (https://www.acm.org/publications/policies/artifact-review473 and-badging-current).
* Whenever possible, even for web pages, please include information about the author and published date.
* The reference used for FAIR applied to research software (line 319, Katz et al., 2021) corresponds to a (partial) outcome of a Research Data Alliance Working Group and could be updated to the now first formal output produced by this group (see DOI:10.15497/RDA00065 and corresponding PDF https://www.rd-alliance.org/system/files/FAIR4RS_Principles_v0.3_RDA-RFC.pdf)
* I suggest also including a reference to the code supporting this publication. I know it is provided as supplementary material but getting a permanent identifier together with citation data for this particular release is also possible (and slowly becoming a common practice). I would also like to see the GitHub pages working and linked to from the publication (possibly with a note regarding differences that might appear as the repo evolves post-publication).

1.3. Professional article structure, figures, tables. Raw data shared.
The article structure is appropriate as well as figures and tables.
Raw (and processed) data is included in the supplementary materials.

1.4. Self-contained with relevant results to hypotheses.
It was possible to read the article on its own.
Results are aligned to hypotheses.

1.5. Formal results should include clear definitions of all terms and theorems, and detailed proofs.
Code to process raw data and analyze results is shared, including a set of instructions on how to use it. The presentation of results is clear, detailed and supported by the presented data.
Experimental design: 2.1. Original primary research within Aims and Scope of the journal.
The article aligns to the aims and scope of the journal.

2.2. Research question well defined, relevant & meaningful. It is stated how research fills an identified knowledge gap.
Yes

2.3. Rigorous investigation performed to a high technical & ethical standard.
It looks like.

2.4. Methods described with sufficient detail & information to replicate.
It looks like.
Validity of the findings: 3.1. Impact and novelty not assessed. Meaningful replication encouraged where rationale & benefit to literature is clearly stated.
Replication possible thanks to the available code and basic set of instructions on how to run it

3.2. All underlying data have been provided; they are robust, statistically sound, & controlled.
Raw data has been provided.

3.3. Conclusions are well stated, linked to original research question & limited to supporting results.
I would say yes.
Additional comments: I find this article of broad interest for the research community. Sharing software and in general research artifacts is slowly becoming a common practice aligned to Open Science and the FAIR principles. I hope to see more research on this topic.

As the authors stated, one of the main challenges was the curation of the data which was mainly manual. Although out of the scope of this paper, I hope research in this area will motivate not only conferences but also preprints and journals to be more proactive regarding the metadata accompanying research artifacts.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTIFACTS AND CITATIONS IN COMPUTER SYSTEMS PAPERS
Review round: 1
Reviewer: 2

Output:
Basic reporting: [Introduction]
The list of detailed research questions presented in the Introduction contrasts with the regular scope of an initial section. These questions should be properly justified and connected, which should be done in a further section.  Alternatively, I suggest the authors expanding the "interesting" question (line 66) by exemplifying other possible indicators that would be influenced by artifact sharing.

I suggest the authors summarize the main findings of the study in the Introduction.

[Background and Related Work]
The author cites several references addressing open science and initiatives promoting artifact sharing, which address the research background. However, I missed a section discussing related work on the topic. For instance, previous investigations/discussions on the maturity of open science in the field.

[Figures/Tables]
I suggest presenting the Figure 1 data using tables or grouping conferences by frequency. Yet regarding Figure 1, it is not clear to me the relevance of identifying the "Organization" of the conferences. For instance, several conferences continuously alternate between ACM and IEEE.

I understand Table 1 should adopt a better-contextualized criterion than alphabetical order for listing conferences.  For instance, the author may use the number of papers or the acceptance rate.

Each conference has a variable time of running/publication, which may even lead to its publishing in the next year. I would like to understand how the author reached "42 months of publication" for all publications and why he opted by following this gold number.

[Verifiability]
I was expecting a user-friendly dataset available from a research paper addressing the artifacts' availability in research papers.
Experimental design: Before reading about the study results (Section 3), I was expecting a Section reporting the study design. After reading the entire section of "Data and Methods" (Section 2), I could not find any reference to the research goals/ hypotheses.  Besides, the author should properly address the study population and sample investigated.

The text of section 2 is too monolithic and hard to follow. It does not address the basic content expected from a study design section. After reading it, I only got a vague idea about what the authors had done (execution) and the main data types they collected.

It is not clear the criteria followed by the author to reach the "hand-curated collection" of 56 peer-reviewed "systems conferences." Even if the sample was established by convenience from google scholar, the authors should let it clear in the paper.

I could not reach what does the author mean by "systems conferences."  Besides, I am afraid about the lack of representativeness of the addressed in the study. For instance, I could found only one relevant conference from the Software Engineering field.

It is not clear the methodology followed for identifying artifacts in the papers. Besides, I understand that artifacts available in research papers should also address their quality. I recommend the authors planning and performing an evaluation in this direction.
Validity of the findings: Section 3.2 illustrates my difficulty in understanding the research methodology. It is a subsection about results. However, it starts by presenting a new research question: "does the open availability of an artifact affect the citations of a paper in computer systems?". Besides, this RQ is presented as "the main research question of this paper." Contrastingly, several of the eight research questions presented in the paper Introduction are barely (or clearly) addressed in the paper.

The findings of the paper are not discussed. For instance, what are the possible implications of these findings for the practice of sharing artifacts? Which are possible ways for filling the gaps observed? The section 4.1 "Discussion" basically summarize the several statistical tests and analysis performed (4.1). However, even here I could not see a clear connection among them. Besides, without a proper research plan, I could not get why all these tests and analyses were performed.

Only at the end of Section 5, I could observe some links between the study results and some implications.
Additional comments: Unfortunately, the lack of properly reporting basic content expected from a research paper hampered me from performing a more accurate review and indicating further suggestions for improvement. Despite that, I recognize the relevance of the research topic. Therefore, I encourage the author to keep on with this research. Maybe the author would rethink the design of this study as a systematic review addressing other research artifacts beyond the papers.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTIFACTS AND CITATIONS IN COMPUTER SYSTEMS PAPERS
Review round: 1
Reviewer: 3

Output:
Basic reporting: This article is well-written and addresses statistically the very important question if sharing of software artifacts has an effect on citation metrics of computer science, as well as the need for proper archiving of software artefacts for longevity.

The argument is well put forward and supported by a thorough data investigation for the given sample set.

A couple of smaller typos were found

As the article points out, the availability of software artefacts is important - however this article itself does not have an explicit "Data and software availability" section - which would be very valuable given the data quality and reproducibility provided for this manuscript.

I would also expect the data to be archived in Zenodo with a DOI rather than be a fluctual GitHub reference - I know the author have also uploaded a snapshot of said GitHub repository along with the article - but this would not have been detected by their own measures.

As the data seems to be used for multiple articles under review, a separate version-less Zenodo DOI would be able to bring these articles together for describing different aspects of the same data (e.g. using Related Identifiers mechanism in Zenodo).Only one of these other articles are cited from this manuscript.

Kelly McConville is acknowledged for assisting with the statistical analysis - this article is quite strong on statistics - I might have added McConville as co-author to recognize this fact.
Experimental design: This research is rigourous and the experimental design is well explained.
Validity of the findings: I have not assessed the choices of statistical methods, but the authors have been thorough throughout, e.g. providing p-values and describing method choices. The actual values and figures are also calculated dynamically by the R markdown for the manuscript, and so can be reproducibly verified.

In order to verify the reproducibility I had some challenges in getting the R environment set up - however the author quickly responded to my GitHub issue  and they added detail installation instructions which helped me.

The findings are based on a large subste of the CS System conferences - perhaps more could be added on why these particular conferences (e.g. practicallity of access, notability, familiarity to the authors). The authors have done a good selection time-wise (all considered data in 2017) in order to consider citations building over time.

The data set can be hard to navigate and use as it is shared across multiple publications and corresponding R markdown environments. README files have been provided, but the manuscript itself does not detail which of the data is relevant to this particular manuscript. For instance it is possible for someone proficient in R to analyse the R Markdown of pubs/artifact/artifact.Rmd where the statistical analysis is embedded.

The data/ CSV files are also well documented in that they each have a little file describing their schema - some of these state their provenance, which is nice.
Additional comments: Hi, I am Stian Soiland-Reyes http://orcid.org/0000-0001-9842-9718 and believe in open reviews.

This article is a very welcome addition to the CS field, where sharing of software artefacts have only recently emerged as a practice, catcing up on fields like bioinformatics.

The key message from this article for me is showing evidence for CS authors that sharing artefacts is beneficial for science and for their article's to be cited.

I therefore think this is a very important article for PeerJ CS to publish.

My recommendation would have been Accept except for that the article should more clearly promote its Data/Software Availability as well as archiving those with their own DOI at Zenodo or similar repository.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTIFACTS AND CITATIONS IN COMPUTER SYSTEMS PAPERS
Review round: 2
Reviewer: 1

Output:
Basic reporting: I appreciate most of the improvements made by the author to the paper. Now, the paper reading flows better. The main research goal and study hypothesis are clear. Some design decisions are now better contextualized. However, there are still relevant remaining issues to be addressed. In the following, I summarize my key recommendations for this paper.

1- The research steps should be completely described, especially in the case of following a systematic design (which is not still clear to me).
2- The author should contextualize the detailed research questions and corresponding data analysis procedures, considering main study goals and hypotheses.
3- Discussions should focus on interpreting more the implications of the study findings rather than adding more ad hoc data analysis and statistical tests.
4- The author should significantly improve the discussions about threats to validity.
Experimental design: The author does not report systematic steps followed for gathering papers' data. For instance, "....several search terms were used to assist in identifying artifacts, such as “github,” “gitlab,” “bitbucket,”, “sourceforge,” and “zenodo” for repositories; variants of “available,” “open source,” and “download” for links; and variations of “artifact,” “reproducibility,” and “will release” for indirect references." Besides, it is not clear how the author checked out whether a paper received an award.

At the beginning of the paper, several research questions are presented without a proper justification. These detailed RQ should be addressed and contextualized in the proper section.

I understand that the authors followed a partially ad-hoc process for gathering papers' data without double-checking, instead of following a systematic one. If true, the author should explicit these relevant threats to validity.  Otherwise, the author should report step-by-step how they reached each data to led the package reproducible.
Validity of the findings: I could not find any mention to double-checking on the gathered data. It is relevant at least fo support data gathering through non-systematic steps.

What was the normality test applied? p-value? Yet about normality, the authros state that the data was normalized. However, after that the author argue that "...omit the 49 papers with zero citations to improve the linear fit with the predictors." Outliers? And what about those with >2000 citations?

"...It is worth noting, however, that most of the source-code repositories in these
artifacts showed no development activity—commits, forks, or issues—after the publication of their paper, suggesting limited impact for the artifacts alone". I disagree about this example of poor impact. Please note that commits and forks are typically made for source code, which is not necessarily the case of research data.

I miss a discussion of the findings towards causality (which seems to be unfeasible) and more statistical tests (that does not help to understand the original results. Some of the complementary analysis seems to address irrelevant papers' features (at least without proper contextualization). For instance: "Incidentally, papers with released artifacts also tend to incorporate significantly more references themselves (mean: 32.31 vs. 28.71; t = 5.25, p < 10−6 )." I definitively could not see why this information is relevant in the context of the research. The same for "using colons" and "paper length," among others.

"... In contradistinction, some positive attributes of artifacts were actually associated with fewer citations. For example, the mean citations of the 573 papers with a linked artifact, 47, was much lower than the 71.3 mean for the 102 papers with artifacts we found using a Web search (t = −2.02, p = 0.04; W = 22865, p < 10−3247 ). Curiously, the inclusion of a link in the paper, presumably making the artifact more accessible, was associated with fewer citations. This finding strengthens the need for going beyond a quantitative analysis over a single sample established by convenience.

For instance, I did not see some treatment about self-citations.

"Of the 2439 papers, 292 91.8% displayed at some point an accessible link to the full text on GS. What is the impact of these 9.2% over the citations analysis found? Accessibility is definitively relevant in this context.

"These textual relationships may not be very insightful, because of the difficulty to ascribe any causality them, but they can clue the paper’s reader to the possibility of an artifact, even if one is not linked in the paper." I am afraid I could not get the point here.

"As a crude approximation, a simple search for the string “github” in the full-text of all the papers yielded 900 distinct results. Keep in mind, however, that perhaps half of those could be referring to  their own artifact rather than another paper’s, and that not all cited github repositories do indeed represent paper artifacts." I could not see how this analysis is relevant to support the discussions.
Additional comments: Figure 1 is not cited. Besides, I  understand that this figure could be omitted from the paper.

I suggest combining the analysis presented in Figure 2 and Figure 3 into a single one.

Table 1: please standardize the number of decimal places

2439 papers=> 2,439 papers

"...when skimming these paper*s*..."

The zenodo package is not cited in the paper.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTIFACTS AND CITATIONS IN COMPUTER SYSTEMS PAPERS
Review round: 2
Reviewer: 2

Output:
Basic reporting: This article is well-written and addresses statistically the very important question if sharing of software artifacts has an effect on citation metrics of computer science, as well as the need for proper archiving of software artefacts for longevity.

The argument is well put forward and supported by a thorough data investigation for the given sample set

Overall language, background and methodology details have been significantly improved  in response to peer review.
Experimental design: This research is rigourous and the experimental design is well explained.

The author has responded well to my change suggestions in GitHub and have now added a Zenodo DOI with archive of the data and software.
Validity of the findings: I have not assessed the choices of statistical methods, but the authors have been thorough throughout, e.g. providing p-values and describing method choices. The actual values and figures are also calculated dynamically by the R markdown for the manuscript, and so can be reproducibly verified.

The data set can be still be a bit hard to navigate as it is intermingled with the data, but the strength of that is that the the data results are (for this paper) reproducible. The install instructions have been improved.
Additional comments: Thank you for improving this submission, this is now a very strong contribution.

Repeating from my previous review:

This article is a very welcome addition to the CS field, where sharing of software artefacts have only recently emerged as a practice, catcing up on fields like bioinformatics.

The key message from this article for me is showing evidence for CS authors that sharing artefacts is beneficial for science and for their article's to be cited.

I therefore think this is a very important article for PeerJ CS to publish.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTIFACTS AND CITATIONS IN COMPUTER SYSTEMS PAPERS
Review round: 3
Reviewer: 1

Output:
Basic reporting: The article is presented in a clear way, with definitions and references useful to the reader. I still would suggest a quick proof-reading to double check for typos and so. For instance, an article seems to be missing in the sentence starting as “An additional goal of is an exploratory”.

First mention of a research artifact (including data, publication, software, etc.) should be accompanied by a reference, even if detailed later (this makes it easier for readers interested in that element to directly go for it rather than looking for the reference somewhere else in the text). This is not the case of the dataset produced together with this study, first mentioned on line 115.

Regarding the sharing of data and software. For any future occasion, I suggest separating data from software as they might have different licenses (unless using a data version for both is indeed intended).

Some conferences might have the same acronym. I suggest expanding the name and adding a link to their websites (if still available).
Experimental design: I suggest making clear from the beginning that the only research artifacts analyzed in this study are in the form of software (or systems software) excluding data, workflows and others.

Step 6 (line 200) needs more detailed explanation. I (randomly) checked the file https://github.com/eitanf/sysconf/blob/master/data/papers/ASPLOS.json from the GitHub repo corresponding to this study, it includes some “citedBy” information. The first paper “cherupalli2017determining” appears with 21 in Google Scholar but that does not seem to coincide with the “citedBy” in that file or to the “outCitations” in the file https://raw.githubusercontent.com/eitanf/sysconf/master/data/s2papers.json. Based on the description on the GitHub repo “papers/: A collection of JSON files, one per conference, with Google Scholar statistics on each paper in the conference”, I was expecting to find 21 “citedBy” items there for the mentioned paper but I saw more. Again, this was a random and quick look but still it might be worth to add some more information on this step (and maybe the others).
Validity of the findings: I find this study of value as we need to understand better how research artifacts beyond the paper itself are shared and what impact they represent in terms of recognition (e.g., in the form of citations) and reproducibility.

One of the limitations of this paper that is not discussed at all (it is maybe outside the scope) is exclusive focus on software as research artifact. Sharing and citing data has been encouraged longer than sharing and citing software. One question that arises is whether the analysis on citation count would change across papers only sharing data, only sharing software or sharing both. One of the premises in this study is that paper sharing software is more reproducible and therefore it would be preferred by researchers, i.e., traducing in more citations. I find it difficult to reproduce a paper with data but not software or vice versa.
Additional comments: Thanks to the author for taking the time to address the reviewers’ comments. I find the current version an improvement regarding the previous one.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTIFACTS AND CITATIONS IN COMPUTER SYSTEMS PAPERS
Review round: 3
Reviewer: 2

Output:
Basic reporting: I understand the author made considerable improvements in the manuscript, especially those addressing the study's characterization and reproducibility. Therefore, I recommend accepting the paper.

Besides, I invite the author to check the grammar of the following sentences:

“These textual relationships may not be very insightful, because of the difficulty to ascribe any causality them, but they can clue the paper’s reader to the possibility of an artifact, even if one is not linked in the paper.”

“…Using a search function on each document (“pdfgrep”) on each of search terms listed above…”, please check the grammar.

Yet regarding the last sentence, please note that the paragraph where the terms are listed reports that “...several search terms were used to assist in identifying artifacts, *such as* “github,” “gitlab,” “bitbucket,”, “sourceforge,” and “zenodo”….” Here, I am afraid the use of the expression “such as” may not let clear whether all terms applied in the searchers are listed (as expected). It may address a reproducibility issue to fix.
Experimental design: ok
Validity of the findings: ok
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTIFACTS AND CITATIONS IN COMPUTER SYSTEMS PAPERS
Review round: 4
Reviewer: 1

Output:
Basic reporting: No comments
Experimental design: No comments
Validity of the findings: No comments
Additional comments: Thanks for taking into account all the comments raised by reviewers.

Two comments on this new version.

In line 202 “to” is duplicated, see “closest to to the”.

One more comment related to the rebuttal letter.
Comment by reviewer: I suggest making clear from the beginning that the only research artifacts analyzed in this study are in the form of software (or systems software) excluding data, workflows and others.
Response by the author: That is not quite the case. I looked at all the research artifacts I could find in repositories and digital libraries. A minority of those were pure data or configuration files (even when hosted on github.com). They are treated equally to the code artifacts in this study.

The article is titled “Software artifacts and citations in computer systems papers”. The abstract mentions software again “The availability of these software artifacts is critical” and the Introduction section mentions code “an important step towards this goal is the sharing of artifacts associated with the work, including computer code” and then the rest of the article mentions “artifacts” in general. It puzzles me that the title is about software artifacts but the article is about research/experimental artifacts.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: RESEARCH ARTIFACTS AND CITATIONS IN COMPUTER SYSTEMS PAPERS
Review round: 5
Reviewer: 1

Output:
Basic reporting: Thanks for the revised version. I think all points raised by reviewers have been now addressed.
Experimental design: No comment
Validity of the findings: No comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A MATHEMATICAL FORMULATION AND AN NSGA-II ALGORITHM FOR MINIMIZING THE MAKESPAN AND ENERGY COST UNDER TIME-OF-USE ELECTRICITY PRICE IN AN UNRELATED PARALLEL MACHINE SCHEDULING
Review round: 1
Reviewer: 1

Output:
Basic reporting: The language used in the study is very clear and unambigous. In the introduction, the topics are discussed regularly. There are enough figures to make understandable. Especially the  part experiment and test has been prepared in detail and with care. Necessary definitions that make the equations more understandable are given in tables.
Experimental design: It is a study in accordance within Aims and scope of the journal.
Validity of the findings: The conclusion part is discussed in detail and explanatory. The curves obtained prove the effectiveness of the proposed method.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A MATHEMATICAL FORMULATION AND AN NSGA-II ALGORITHM FOR MINIMIZING THE MAKESPAN AND ENERGY COST UNDER TIME-OF-USE ELECTRICITY PRICE IN AN UNRELATED PARALLEL MACHINE SCHEDULING
Review round: 1
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: In this study, which optimizes the scheduling problem of a new bi-objective unrelated parallel machines with sequential setup times, it is aimed to minimize the makespan and the total energy cost.

The motivation, importance and flow of the article is quite good. The contribution of the article appears to have met the standards for the "PeerJ Computer Science" journal.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A MATHEMATICAL FORMULATION AND AN NSGA-II ALGORITHM FOR MINIMIZING THE MAKESPAN AND ENERGY COST UNDER TIME-OF-USE ELECTRICITY PRICE IN AN UNRELATED PARALLEL MACHINE SCHEDULING
Review round: 1
Reviewer: 3

Output:
Basic reporting: The abstract should not be expressed in general sentences. Instead, it should be explained as much as possible about the work done and the conditions under which the work takes place. Effects, values, results, etc. it will be much more convenient and useful for the reader/researcher to provide information containing answers to questions.

The introduction section should be developed more clearly and concretely for those who are going to read this article. There are decouples between paragraphs. It should be rewritten more fluently.
Experimental design: Similar articles in this field should be summarized and tabulated for readers at the end of the literature review section. A flowchart should be attached to the article that shows the general structure of the work. The sections proposed in the flowchart can be shown in more detail.

The dataset studied and given as an example should be specified in the article and added as a reference. The success of the NSGA-II algorithm should be compared with other basic algorithms in this field. The advantages and disadvantages should be revealed by examining the results obtained.
Validity of the findings: It should be compared with similar results obtained in similar studies conducted earlier. The effective aspects of the study should be highlighted. Considering the situations arising as a result of this study, new ideas should be offered to readers.
Additional comments: no comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A MATHEMATICAL FORMULATION AND AN NSGA-II ALGORITHM FOR MINIMIZING THE MAKESPAN AND ENERGY COST UNDER TIME-OF-USE ELECTRICITY PRICE IN AN UNRELATED PARALLEL MACHINE SCHEDULING
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors correctly understood the changes I recommended and responded appropriately to all of them. Considering the necessary explanations, the article was revised as desired.
Experimental design: The authors correctly understood the changes I recommended and responded appropriately to all of them. Considering the necessary explanations, the article was revised as desired.
Validity of the findings: The authors correctly understood the changes I recommended and responded appropriately to all of them. Considering the necessary explanations, the article was revised as desired.
Additional comments: The authors correctly understood the changes I recommended and responded appropriately to all of them. Considering the necessary explanations, the article was revised as desired.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYNTAX AND PREJUDICE: ETHICALLY-CHARGED BIASES OF A SYNTAX-BASED HATE SPEECH RECOGNIZER UNVEILED
Review round: 1
Reviewer: 1

Output:
Basic reporting: In investigating if and how syntactic information can be used to de-bias hate speech recognizers, this paper presents negative results: even when just syntactic features are considered, prejudice is captured from training corpora and make the application to new datasets not very fruitful.
The advantage of Kermit, the model proposed by the authors, is that it enables a visual, post hoc analysis of the results by exploring the syntactic information used in the decision process of a neural network. Consequently, it is possible as future work to define ad-hoc rules for mitigating the bias, even if the authors do not clarify how this feedback can be incorporated in the model.

The paper is sometimes not very clear or incomplete. In my opinion, the authors should provide

- a short definition for hate speech recognizers in the introduction
- in the related work section, they should quote works about gender-based bias in neural models

A native speaker should check the paper; here is a list of several mistakes:

Line 41- 42 There is a repetition "aiming in transformers aiming to put trigger words in contexts"
Line 90 - 91 "learned models fail to capture another important fact: learned models can "right from the wrong reasons" -> "learned models fail to capture another important fact: learned models can be "right from the wrong reasons"
Line 100 - 101 "help in shedding the light" -> "help in shedding light"
Line 113 - 114 "Zanzotto et al. (2020)" is not reported correctly (add a verb)
Line 130 "Devlin et al. (2018)" is not reported in the correct way
Could you standardize the name using a letter for the model Kermit with a thunder symbol?
Experimental design: line 35-36 The authors wrote "Ethically-charged biases are extremely more dangerous than simple probabilistic biases, which can lead to a conclusion utilizing wrong premises." However, I think that even the first type of bias is probabilistic in nature, in the sense that the model created on training data acquires their bias probabilistically. In other words, I don't see this dichotomy between probabilistic vs. ethical bias since even ethical bias is acquired through a probabilistic process. Statistical bias becomes ethical when we recognize them as such.

Kermit, a syntax-based hate speech recognizer, is trained on a possibly unbiased training set and tested on two newly created datasets. The description of the datasets is not clear, and sometimes they are reported as corpus. I do not think that the way the new datasets have been created made them meaningful for the reported experiments. For example, the authors suppose that Black lives matter corpus is composed by tweets and comments from Instagram commenting on a movie produced by users from the black community. However, comments and tweets about the movie can be produced by users from different ethnicity. Since the authors can not be sure that the majority of the content came from the black community, the experiment is flawed.
Validity of the findings: Since the datasets created on purpose to investigate the research questions are not well designed, it is not clear what the experiments are showing.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYNTAX AND PREJUDICE: ETHICALLY-CHARGED BIASES OF A SYNTAX-BASED HATE SPEECH RECOGNIZER UNVEILED
Review round: 1
Reviewer: 2

Output:
Basic reporting: - The structure of the paper is clear and reasonable.

- The form of the sentences needs to be corrected, the beginning of a sentence must be capitalized and the end of a sentence must have a period, e.g, lines: 52, 53, 106-111, ...

- I don't understand the "SaettaKermit" concept because it only appears in Table 2.

- Sentences and grammar need to be sent to proof editing for correction to improve.
Experimental design: - You need to describe more clearly the process of creating the dataset, how to annotate it, how to calculate inter-annotator agreement? It is necessary to analyze and compare datasets on many different linguistic aspects. For example, this paper (https://arxiv.org/pdf/2103.11528.pdf) describes how to create a dataset for HSD.

- The evaluation indicators have not been clearly described.

- There should be an ablation test to clarify the role of syntax integration.

- The perturbation contribution needs to be further analyzed with clearer data. I don't know what the number is and how much your method solved.
Validity of the findings: Integrating syntactic information into models is essential. This paper has proved its effectiveness on the HSD task. But you need to clarify my suggestions to increase the persuasion.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYNTAX AND PREJUDICE: ETHICALLY-CHARGED BIASES OF A SYNTAX-BASED HATE SPEECH RECOGNIZER UNVEILED
Review round: 1
Reviewer: 3

Output:
Basic reporting: - p.1 : what the authors mean with the phrase "ethically charged" can be misleading. The expression in itself can make one think of something that bear a positive connotation (i.e. something that is supposed to do some good and has a positive societal impact), but from what is stated in the immediately following sentences, it would seem to indicate the exact opposite, that is a type of bias that brings potential representational harms to certain groups

 - nodes in the parse trees should be made more visible; I understand that the purpose is to provide a visual representation of the different  activation values, by foregrounding the active nodes, but in some figures the other nodes (especially the black ones) are barely readable, while I think it's important to give a clear picture of the whole sentence, also showing the nodes that remained inactive (the color scale used by the authors is indeed helpful in this)

 - p.3, ll. 82-3: is it "unintended" bias, instead of "untended"? I don't seem to find the reported quotation in Hardt et al. 2016

- p.8. ll.304-5: I don't fully understand what the authors mean when they say that Davidson et al.'s dataset is not sensitive to syntax; they motivate this claim commenting on the same accuracy reported for the three BERT-based models, but they should probably try to elaborate more on this, maybe explaining in particular what BERTreverse and BERTrandom  models are useful for  (besides just pointing to Zanzotto et al. 2020).
  - ll. 310-1: it is not clear why the dataset of Davidson et al. should be a good candidate for experiments related to the second research question, while it is not the case for that of Waseem and Hovy, which is the one on which SaettaKermit gave the best results ever. Wouldn't it be worth replicating the experiments of round 2 on this dataset as well?
  - ll. 314-15: the data shown in the figure seem to contradict what is stated in this sentence: from what I see in the other two datasets BERTbase also prefers, in non-"neutral" tweets, the classification of the tweet content as hateful, rather than as offensive (although in the Democrats dataset this difference is much less marked).

 - Appendix C: 1) it should be made clearer in the text that sub-figures b in the examples are those were one or more words were changed
	2) line 472 says "parse trees obtained from KERMITbert", but based on what described in sect. 4.3 the qualitative analysis was carried on the parse trees from SaettaKermit, so using examples from a different model sounds very confusing. Should I assume this is just a typo or is there any other reason?

Typos:
 - l. 75: dialectic -> dialect (?)
 - non-syntactic citations should be put in brackets, e.g. in line 114: " [...] for decisions in neural networks (Zanzotto et al. 2020). "
 - l.91: models can "right for the wrong reasons" -> can be right (?)
 - l.116: model for hate speech recognizer -> recognition
 - l. 162: it should helps -> help
 - l. 319: has matches-> matches (?)
 - Appedix C: the parse tree i Fig. 6a is exactly the same as the one in Figure 6b
Experimental design: - p.6+Appendix A: (on the creation of the US elections dataset) an hashtag is not necessarily an endorsement, so it is not clearbased on what the presence of a hashtag referring to Biden or Trump can establish in such a deterministic way the political orientation of the user. Further details are provided in Appendix A about this, but it is not clear what the percentages in Table 5 actually refer to; is it the proportion of tweets in the respective dataset? Furthermore, the authors try to show that the choice of using the hashtag as a proxy for the assignment to one of the two datasets is reflected in the data on exit polls. However, the data given in the dedicated table 1) are not systematic (for some states the proportions are reversed), 2) the numerical difference does not appear so striking, except in a few cases 3) the table is incomplete, in that the data is not shown for all states, but only for those in which - according to the authors - the gap is greater. If, as it seems to understand, the gap refers precisely to the percentage of votes given to one of the two parties, the use of the hashtag as a discriminating factor seems to be a fairly weak motivation

 - p.7: the sample used for the Blind test and follow-ups is quite small, especially considering the original size of the novel datasets created for the experiments. Is there a specific reason for such a choice? Also, what background did the annotators have? Were they under/graduate students, crowdworkers, experts on syntax/deep learning, other? This seems to be a relevant information to add, especially to help the reader understand the possible reasons behind the low IAA results

Appendix C: Fig. 5a: I don't see how this can be considered as an instance of offensive tweet. Is the example taken from the sample of tweets used for the second round of experiments? What I mean is whether human annotators actually agreed on the final label provided by Kermit
Validity of the findings: no comment
Additional comments: The paper introduces KERMITz (or SaettaKermit, as I will call it henceforth in the detailed comments), a visual tool that extends a syntax-based hate speech recognizer (Kermit), and describes the experiments carried out with such tool. More specifically, the two research questions the authors aimed to explore in this work are 1) whether the use of syntactic information is helpful and contributes to the improvement of the performance of a hate speech classifier, 2) if the use of such information can help remove possible biases, which other models, based mainly on lexical information alone, may tend instead to reproduce and amplify. With respect to the first point, the reported experiments show how the use of syntax - in the form of constituent trees - significantly increases performance. As for the second, however, the experiments carried out show that even the use of syntax is not able to reduce the presence of biases in the system's decision-making process.
 Precisely the negative results detected with respect to the latter point represent the main contribution of this work, together with the possibility of using a hate speech recognition system that also provides a visualization tool for post-hoc analyses of the system's decisions.
The paper, however, is not always easy to follow, and relevant information is sometimes reported in a confusing or incomplete way.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYNTAX AND PREJUDICE: ETHICALLY-CHARGED BIASES OF A SYNTAX-BASED HATE SPEECH RECOGNIZER UNVEILED
Review round: 1
Reviewer: 4

Output:
Basic reporting: The article is generally well written and clear, but there are some typos:
Line 141: the definition [...] that focusES (not "focus")
Line 156: language is THE real reason
Line 162: as it should help (not "helps")
Line 187: 3.4.2 Black Lives Matter corpus (with capital "L" and "M")
Note 3, Line 202: Available at (with capital "A")
Line 205: was the United States presidential election (without "about")
Line 315: and this predisposition (without the "-")
Appendix A, Line 458: two datasets; (2) states (put a semicolon between point 1 and 2!)
Appendix B, Line 463: if they agreed (using "he/she" is correct too, but the neutral "they" is also a good choice in standard formal English)

Also, pay attention to bibliographic references, because some of them are not between brackets:
Line 114: for decisions of neural networks Zanzotto et al. (2020) > for decisions of neural networks (Zanzotto et al. 2020).
Line 130: The transformer component consists of BERT Devlin et al. (2018) > The transformer component consists of BERT (Devlin et al. 2018).

Tables and figures are clear, but it is fundamental to correct:
Table 2, bottom line: Is "SaettaKermit" KERMIT(with the bolt in subscript)? You never call it SaettaKermit in the rest of the article, so, please, align "SaettaKermit" with your usual terminology, or at least explain in the notes what "SaettaKermit" is;
Appendix C, Figure 6 (FUNDAMENTAL!): Example (a) has the same image as example (b). Please, change it with the correct one (Mary is American), because Figure 6's examples are really powerful and useful for the paper's comprehension.

The Introduction should contain examples of the prejudice you are describing, especially at Line 33, and at Line 39-40-41. Otherwise, it might appear that you believe that hate speech producers are somehow unfairly censored. For this reason, it might be useful for you to specify that this kind of unjust censorship is applied to those who produce non-hateful messages, thus also involving marginalized people who are simply describing their experiences (such as "Mary is African", from Appendix C).
Also, it might be a good idea to specify that hate speech moderation is not censorship. This is motivated by the fact that hateful messages make their targets less prone to communicate and to express their point of view, thus de facto censoring them. This is the reason why preventing hate speech is a meaning to protect freedom of speech. Useful bibliography on this subject:
West, C. (2012), Words That Silence? Freedom of Expression and Racist Hate Speech, in I. Maitra & M. K. McGowan (eds.) Speech and Harm. Controversies Over Free Speech, Oxford, University Press Scholarship Online.

VERY IMPORTANT: The Introduction should contain a definition of hate speech, as hate speech currently does not have a universally accepted definition.

In 4.1 Experimental Set-up you should include the inter-annotator agreement in the description of the Blind/Inside-out/Prejudice Test's, in each passage.
Experimental design: No comment
Validity of the findings: All underlying data have been provided; they are robust, statistically sound, & controlled: I am uncertain about this.
The methodology used is generally solid, the underlying data have been provided and the experiments can be replicated.
However, in my opinion, I think some sections could be improved.
1) 3.4.2 Black lives matter corpus: I am not convinced by the choice of investigating "non-offensive utterances produced during the Black Lives Matter (BLM) protest and written by the black community" through gathering tweets and Instagram posts about a movie. "Black Is King" is indeed a significative movie for the Black community, but you cannot be sure that only Black people commented it on Twitter/Instagram. Although this corpus is not without potential for studying BLM movement, perhaps a corpus composed of tweets directly about the BLM movement could have been more adequate, albeit also with more noise and hate speech.
2) 4.1 Experimental Set-up (from Line 272): The Blind/Inside-out/Prejudice Test is excellent, but I am not convinced by the number of sentences analysed, especially in the case of the Blind test, from which the sentences analysed in subsequent tests derive. I think 34 sentences are too few, especially if selected randomly from both corpora and if you want to compare the human annotation of HS with the annotation of BERT and KERMIT. There is no need to annotate manually over 6,000 tweets, like in HaSpeeDe 2 task from EVALITA 2020 (Sanguinetti et al., 2020), for example, but at least 100. Also, who are the annotators? You need to include some information about them, such as:
- Are they English native speakers? Or at least they have a high/good comprehension of written English?
- What is their nationality? Are they from the United States, thus emotionally involved in the BLM/presidential election?
- Are there black people among them? There is a study on misogynistic hate speech's recognition that highlights how, in that case, annotators, who are part of the group targeted by the specific hate they are annotating, annotate hate speech differently from the annotators who are not targeted by the same hate speech (Look at Wojatzki et al. (2018) for the full study). Could we be facing a similar case with black people and white people annotating racist hate speech? Even without answering this question, this possibility should be taken into consideration in the paper.
- Do they have any training in annotating hate speech? Have you given them a definition of hate speech?

Sanguinetti M, Comandini G, Nuovo ED, Frenda S, Stranisci M, Bosco C, Caselli T, Patti V, Russo I (2020) Haspeede 2 @ EVALITA2020: overview of the EVALITA 2020 hate speech detection task. In: Basile V, Croce D, Maro MD, Passaro LC (eds) Proceedings of the seventh evaluation campaign of natural language processing and speech tools for Italian. Final Workshop (EVALITA 2020), Online event, December 17th,2020, CEUR Workshop Proceedings, vol 2765. CEUR-WS.org. http://ceur-ws.org/Vol-2765/paper162.pdf

Wojatzki et al. (2018), Do Women Perceive Hate Differently: Examining the Relationship Between Hate Speech, Gender, and Agreement Judgement, in A. Barbaresi, H. Biber, F. Neubarth & R. Osswald (eds.) Pro-ceedings of the 14th conference on Natural Language Processing (KONVENS 2018), pp. 110-120.
Additional comments: This paper discloses important findings for the study and the automatic recognition of hate speech.
It needs to revise its language in a few paragraphs and to improve its data, in order to make them more solid.
I think that the authors have done an excellent job and I believe they should be given the opportunity to revise their text for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: SYNTAX AND PREJUDICE: ETHICALLY-CHARGED BIASES OF A SYNTAX-BASED HATE SPEECH RECOGNIZER UNVEILED
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper is significantly improved with respect to the previous version but several sentences are hard to read and there are some typos.

For example:

Line 35 "HSRs aim to recognized posts" -> "HSRs aim to recognize posts". Reformulate the whole sentence
Line 51-54 Reformulate the sentence.
Line 373 "Hate speech recognizers (HSRs) are a dual-use technology as these" -> "Hate speech recognizers (HSRs) are a dual-use technology that"
Experimental design: I think that more details about the datasets created (a couple of concrete examples of the texts that contain) could be useful to better understand the complexity of the task
Validity of the findings: I like the fact that the focus of the paper is on negative results. In this version is more evident.
Additional comments: No additional comments