Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TOWARDS FAIR PROTOCOLS AND WORKFLOWS: THE OPENPREDICT USE CASE
Review round: 1
Reviewer: 2

Output:
Basic reporting: Disappointingly, the paper is confusing, muddled and sometimes impenetrable. This makes it difficult read.  Material is presented or referred to without context or explanation. The paper is in a confusing order and the background needs to be rewritten. It really about a semantic model, competency questions to build and validate that model.

-	The background is a mixture of literature, the model and then some competency questions that are complementary but without actually presenting the questions that they are complementary to, or an example of a workflow. The literature cited in the background is a strange mixture of reproducibility, ontologies and knowledge representation technologies. Why is SHACL relevant here?

-	For a paper on FAIR it would be expected to discuss what the exact interpretation of FAIR is by the authors and a clear explanation of what is meant by a workflow.

-	The FAIR ecosystem presented as background is the GO-FAIR ecosystem, not the FAIR ecosystem. This is not a problem so long as it’s clearly stated that this is particular ecosystem that the authors are operating in and is used in their methodology, and that other services and approaches are available (it is not THE FAIR ecosystem). It is one take on FAIR.

-	The FAIR ecosystem section is (i) a description of the GO-FAIR process in []; (ii) a superficial discussion on FAIR software (as computational workflows are software); and (iii) some discussion about FAIR incorporated into the workflows of scientists, which is different to the issue of  describing workflows so that they are themselves FAIR. This is all muddled together.

-	The FAIR Workflows Approach section competency questions are presented ok (though here would be the place to compare CQ in a table for example, rather than the background section. The model itself is presented in a dense and confusing way so that it’s difficult to relate to the Profile and you have yet to state a concrete example of the workflow. Muddling up references to other work just makes it even more confusing. This whole section needs to be grounded in a concrete example – like the PREDICT workflow. The whole section on SHACL (line 357) is obscure – what is the intended point of this?

-	The PREDICT workflow needs to be presented earlier and used throughout

-	The evaluation on “responding competency questions” is cluttered with references to CQ in other approaches.

-	The discussion confuses the representation of PREDICT with the completeness of the PREDICT model.  Some of the discussion doesn’t seem to relate to the modelling exercise but more generally (points 3 and 4)

-	The section “foundations of FAIR workflow semantic profile” should be moved to the presentation of the profile.

-	The role of CWL, CEDAR and FAIRsharing are thrown into the discussion (no citations) without context. These should be incorporated properly.

-	Many English issues need to be fixed (for example, there is no such word as “identificator”).
-        All Acronyms should be explained. All cited systems should be referenced.

It is not clear who is the intended readership. If it is computer scientists then there needs to be more explanation of the experimental steps undertaken by life scientists and the tools they use. If it is a life scientist then the semantic model for FAIR workflows needs much more careful and simpler presentation. If it is an ontologist or semantic modeller then the discussion about the model is interesting but the narrative mixes up the model with the sources of the model makes it more laboured than need be and obfuscates the main messages. One assumes that this is the intended audience as otherwise terms like “endurant” would not be used, and the separation of the profile from the model would not have been presented.  Figure 1 is not clear unless the reader is an expert ontologist

The literature is referenced but confusingly presented and terms are not always defined –for example, what is the difference between a parameter, a meta-parameter and a hyper-parameter? Given that the paper is about workflows it is necessary to explain what is meant by a workflow and give an example of one (this doesn't happen until the PREDICT case study, and this is too general). The paper mentions workflow languages and systems, but it appears that the actual example is a Jupyter Notebook – this is reveal incidentally. (By the way, Jupyter Notebooks do not enforce rigorous stepwise executions – one can bounce between cells, so the step order is only recorded by log trails and that requires a system like ProvBook, which is not a native JN subsystem).
Experimental design: The research question is partially defined. The abstract implies that the paper will be about FAIR principles applied to workflows. The paper is about defining a semantic model for steps of instructions in experimental practice in scientific investigations using an example for exposition and validation. This is a necessary step towards FAIR workflows but the claim that the paper presents general FAIR guidelines seems a little exaggerated.

Appendix 1 gives details of the interviews. The interview methodology for the competency questions should be sketched  in the paper. Although the number of researchers is a low number (6) this might be sufficient if the selection criteria and range of skills of the interviewees was clearer. None appear to have used computational workflow platforms. Many appear to be from the authors laboratory (a high incidence of SPARQL experience is unusual). It is unclear to the extent they are wet lab (using lab protocols/SOPs) or dry lab (using computational frameworks). This leads to some suspicions of a rather skewed interview cohort.

The interview results were interesting and although the competency questions make sense it was not entirely clear how the competency questions came out of the interviews. Were the standard practices of thematic analysis - coding semi structured interview transcripts - undertaken? How selective were the questions?  For example, interviewees highlighted the need to describe parameters “what are the parameters for workflow X” but this did not appear as a competency question. Other CQs are not adequately explained – for example, CQ2.4 (line 236) which is not presented in the subsection on “responding competency questions” (line 630) so this reviewer could not work what it means by examining an example.

The methodology of “FAIRified Data Collection” (line 435) starts well enough and in general can be followed but many major steps are glossed over and I am sure it is not reproducible from this paper. We have to take on word that the section on “addressing the FAIR principles” (line 613) is achieved. The FAIRifier approach is not well enough described and the RDF snippet using the profile ontology is trivial.

Line 436 – why is it necessary to achieve FAIR data by generating Linked Data? Is this “FAIRification” or RDF publishing? The step “define a semantic model for the data” (line 459) seems like a major activity akin to semantic harmonisation and integration and a recognised global challenge. How hard was this? Where did you “search” for “interactor and source concepts”?
Validity of the findings: The model is interesting and the CQs are a worthwhile contribution, though is they address “previously unaddressed reproducibility requirements” is unclear -which previously unaddressed requirements? The evaluation rests in the capability of the model to answer these CQ.

The abstract implies that the paper will be about FAIR principles applied to workflows – business processes, computational workflows / scripts, and manual protocols. In fact there is little about “FAIR” as such – how workflows and protocols may be found, accessed, interoperated and reused. Computational workflows are also software artefacts that challenge the FAIR data principles especially with regard to interoperability and reuse – see (Anna-Lena Lamprecht, et al Towards FAIR principles for research software, Data Science, vol. Pre-press, pp. 1-23, 2019 DOI: 10.3233/DS-190026) for discussions on FAIR principles applied to software and a number of papers in the recent special issue on Data Intelligence (http://www.data-intelligence.org/p/67/), for a wider discussion on the applicability of FAIR to workflows, protocols and e-infrastructure.

The paper is fundamentally about defining a semantic model for steps of instructions in experimental practice in scientific investigations using an example for exposition and validation. The contribution for “general guidelines to make scientific workflows open and FAIR” is not really substantiated.

-	Line 924 “in this work we examined how the FAIR Principles can be applied to scientific workflows”. This is not really substantiated.
-	Line 924 “we adapted the FAIR principles to make the PREDICT workflow [ ] open and reproducible”. This is not fully substantiated. Reproducibility of software and method is more that describing a process in RDF. Did you mean “adopt” rather than “adapt”?

CWL (and WDL) are recognised workflow description languages (adopted by GA4GH and others). The authors state that CWL is the “de facto standard for syntactic interoperability of workflow management systems” line 93), and that CWL has a Prov profile to define how to record provenance of a workflow run, (https://github.com/common-workflow-language/cwlprov, https://doi.org/10.1093/gigascience/giz095 it would be useful to compare the profile on page 7 with CWL (there are some references to CWLProv).

As the model covers Lab Protocols and SOPs, it is surprising that models such as Smart protocols (https://bioportal.bioontology.org/ontologies/SP), Autoprotocol (http://autoprotocol.org) and Procotols.io are not discussed.

It is not clear the range of expressivity of the workflow that can be expressed. How are forks and conditionals addressed? The implication is that these are simple pipelines.
The profile (Figure 1) seems entirely generic except for the Machine Learning part. Why did you make the decision to explicitly include ML and not, say, a simulation sweep or an ODE model construction step?
Additional comments: In this paper there is some very good and interesting work.
The work to apply FAIR to workflows is important. The separation of “instruction” from the “step” is particularly insightful and the approach of interviews and CQ is very good. the Interviews were insightful

However, the paper needs to be substantially rewritten to be publishable – everything is thrown together and the messages are lost or confused or both. The current presentation does not do your work the credit it deserves.

-	Who is the audience you intend to address? And is the background and terminology suitable for that audience?
-	Restrict your claims to the presentation and evaluation of a model for workflows
-	Be clear and clean about what you mean by workflows, by FAIR and the model itself.
-	Suggested reorganisation:
o	background presents literature on FAIR data and software and explain your interpretation of the FAIR principles;
o	present the workflow landscape: manual protocols, business processes, computational workflows (interactive and non-interactive) and their operating environments (notebooks, scripts, BPMs, WfMS etc).
o	then introduce the PREDICT workflow as a running example. Present the TWO OpenPREDICT versions diagrammatically so we can understand them when we work through the representation later.
o	present the competency questions, with a table that shows the relationship to other questions and highlights the novelties, and how these map to the FAIR principles
o	the semantic model is then presented without obscuring the model with the profile (or clean this up – the profile is one place and the model in another) with concrete examples using the PREDICT workflow. The section “foundations of FAIR workflow semantic profile” should be moved to here and repetition removed.
o	then have a section on FAIRifying the data collection which, as you are following just one methodology, you can introduce here and work through concretely. Currently we have to take on word that the section on “addressing the FAIR principles” (line 613) is achieved.
o	Then present the workflow in the model relating the description to the visual steps or annotating the description. Be clear that this is a Jupyter Notebook. Maybe even show the notebook
o	In the evaluation do not clutter the “responding competency questions” with references to CQ in other approaches. Include all the CQs (2.4 is missing)
o	Sort the discussion into (i) the ability of the model to represent the expressivity of the workflows faithfully (ii) the completeness of the representation and (iii) other topics regarding workflow FAIRness and reproducibility not addressed by the model or by a static semantic representation of processes.

Other comments
Line 144 – are the FAIR principles are not a minimal set of requirements? They seem to be more an aspirational set of principles and it is unlikely that all principles will be implemented for any dataset and therefore unlikely to be minimal. See Mons et al Information Services & Use. 37. 1-8. 10.3233/ISU-170824)

The FAIR ecosystem (Line 143) is the GO-FAIR ecosystem, rather than the FAIR ecosystem. GO-FAIR metrics is one of several initiatives developing indicators for measuring principles, notably the RDA FAIR Data Maturity WG. There are numerous initiatives developing FAIR services. This GO-FAIR ecosystem is oriented to publishing RDF, which is an expensive step – actually the FAIR principles do not pertain to any technology and argue for machine processability and standards – JSON and Schema.org would qualify as well as anything.  Other FAIR pipelines annotate with 'standard' vocabularies rather than turn everything into RDF

The Hodson interim report (line 975) has been superseded by the final report (DOI: 10.2777/1524).

Line 488 references Figure 4.1 - there are only figures 1 and 2.

Line 940, Acknowledgements. Perhaps you should acknowledge Douglas Adams?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TOWARDS FAIR PROTOCOLS AND WORKFLOWS: THE OPENPREDICT USE CASE
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper is much clearer than the first draft. Its audience is now clearly that of a semantic modeller. The presentation is tighter and has been reorganised to be clearer and logical.  the CQs are much clearer as is the purpose of the paper.

The model is much easier to understand.

A few niggles.

The definition of workflow in the background could still benefit from some attention. I am not convinced the DOLCE definition is the most useful. There is a big jump from this to the description of workflow systems. The main thing about a workflow is the separation of the workflow specification from its execution, be that manual (so called Human in the Loop) or automated.

The paper focuses on provenance of a workflow when reviewing past models but not so much the description of the workflow before execution (so called "prospective provenance"). The discussion of CWL still seems cursory, focussing on CWLProv. CWL and WDL (still not mentioned) are the most widely supported common semantic model for workflows in open science and yet rather dismissed (or ignored). Which is a pity.

In the meantime the BioCompute IEEE standard 2791-2020, has been published (https://opensource.ieee.org/2791-object/ieee-2791-schema/).

Section 2.3 (Applying FAIR to Workflows) still seems to focus more on the FAIRificaton of data to be consumed (or produced?) by the workflow than on FAIR workflows as objects in themselves (which is the main focus of the model presented). [1] highlighted these two sides of FAIR for workflows - the role they play in data fairification and their FAIRification as artefacts in their own right.

The paper still has some typos and odd spellings that should be addressed.

[1] C. Goble, S. Cohen-Boulakia, S. Soiland-Reyes, D. Garijo, Y. Gil, M.R. Crusoe, K. Peters & D. Schober. FAIR computational workflows. Data Intelligence 2(2020), 108–121. doi: 10.1162/dint_a_0003
Experimental design: The rewriting of the case study, making it clearer which datasets were used) makes it easier to judge the design.  The focus on the CQs and the use of the Case Study for evaluating the model is much more convincingly presented.
Validity of the findings: The paper is far more convincing about its results than previously as it now recognises that this is a step towards FAIR workflows and has toned down its claims.

If the paper is thought of as an exercise in a model and the ability of the case study to answer to competency questions then the findings are valid.

There is still the odd niggle - the abstract claims that "In this paper we describe an inclusive and overarching approach to apply the FAIR principles to workflows and protocols and demonstrate its benefits".  This is still overclaiming.  The section 5.1 is a little cherry picking in the choice of FAIR principles to comply with (e.g. F4. (Meta)data are registered or indexed in a searchable resource, or maybe that is the FAIR datapoint).

The CQ are a little bit cherry picked too. CQ2.4 still doesn't get a response in section 5 and was not answered in the rebuttal.
Additional comments: I thank the authors for the detailed rebuttal and for responding to the recommendations and comments. The paper is much better and the model and competency questions are a valuable contribution to the field.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE DISTRIBUTED LEDGER TECHNOLOGY AS A MEASURE TO MINIMIZE RISKS OF POOR-QUALITY PHARMACEUTICALS CIRCULATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: * This paper gives an analysis of the Distributed ledger technology (DLT) as an innovative means of poor-quality pharmaceuticals prevention.
* The introduction explains the subject and the logic is clear.
* The authors briefly explain why DLT can help to address the aforementioned issues.

However,
* The Material& Methods section says the paper would contain analysis of laws from different countries but I see very little of them in the rest of the paper.
* The results section and figures show the very basic blockchain technologies but didn't reveal many relations between blockchain technologies with the pharmaceuticals supply chain.
Experimental design: * The authors collect a number of pharmaceutical supply chain DLT solutions by conventional web search and academic papers/reports survey.
Validity of the findings: * The authors have found a number of DLT solutions in the pharmaceutical supply chain. The collected works have a good coverage.

* However, the paper lacks a discussion on how candidate solutions solve the challenges when using DLT. These challenges include but are not limited to:
a. how each candidate solution ensures the data (e.g., temperature, humidity) to the ledger is correct, e.g., a dishonest logistics company may insert false temperature to escape the responsibility.
b. how each candidate solution handles poor network connectivity when pharmaceuticals are shipped in some suburb areas.
c. how each candidate solution realize the authentication? the paper only mentions DLT provides authentication but didn't evaluate the authentication in each DLT solution. I have no clue how the authentication is achieved and don't know whether their claimed authentication is trustworthy or not.

* The concluding section says DLT can provide desired properties to reduce the risk of the defective product entering the market and increase the effectiveness of its detection, however, through the results and discussion sections, readers cannot clearly see how these properties are ensured in each solution analyzed by the authors.
Additional comments: * The desired properties of DLT mentioned in the paper are well-known.
* The paper didn't explain how DLT works closely with the pharmaceuticals supply chain and failed to explain how some key challenges in DLT of the supply chain are addressed by solutions surveyed by the authors.
* If the authors can add how surveyed solutions address the challenges in applying DLT to the pharmaceuticals supply chain, the paper will have enough value to be published.

Minor issues:
* Line 71 is not scientific, if you claim DLT is the best, it's better to have some citations or enough supporting proofs.
* Line 86: a better way of this is to say blockchain is a promising way of implementing a distributed ledger. Because distributed ledger != blockchain.
* Line 93-94: In BTC, transactions will not be checked by all the peers.
* Line 220-235: How to ensure the pharmaceuticals are still good when in hand of the first-hand users? Before medicines are returned, seems there is no way to trace.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE DISTRIBUTED LEDGER TECHNOLOGY AS A MEASURE TO MINIMIZE RISKS OF POOR-QUALITY PHARMACEUTICALS CIRCULATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: Please enhance/correct:
Line 38: "Analysis of relative scientific publications had been made."
L 54:: Give numbers for Russian market. This is your target market.

L 71: You state "The Distributed ledger technology is the best way to track goods at all stages..." Please explain why? Give evidence.

L 75-79: Please caplin in detail. The list is not self-explaining.

L 83-84: You need to explain this in detail.

L 101: Meanwhile itis -> it is

L 108: This is like a summary but you should evaluate also alternatives to this technology; then show pros and cons, an finally, draw a conclusion.

L 145: You mentioned the search in "The first source was online journal databases". Please check the sources mentioned below - taken from a journal database.
Experimental design: Please take to following articles into account for the state of the art and comparison to your model:

@article{ ISI:000506382200001,
Author = {Hastig, Gabriella M. and Sodhi, ManMohan S.},
Title = {{Blockchain for Supply Chain Traceability: Business Requirements and
   Critical Success Factors}},
Journal = {{PRODUCTION AND OPERATIONS MANAGEMENT}},
DOI = {{10.1111/poms.13147}},
Early Access Date = {{JAN 2020}},
ISSN = {{1059-1478}},
EISSN = {{1937-5956}},
ResearcherID-Numbers = {{Sodhi, ManMohan/K-7322-2018}},
ORCID-Numbers = {{Sodhi, ManMohan/0000-0002-2031-4387}},
Unique-ID = {{ISI:000506382200001}},
}

@article{ ISI:000470999900036,
Author = {Jamil, Faisal and Hang, Lei and Kim, KyuHyung and Kim, DoHyeun},
Title = {{A Novel Medical Blockchain Model for Drug Supply Chain Integrity
   Management in a Smart Hospital}},
Journal = {{ELECTRONICS}},
Year = {{2019}},
Volume = {{8}},
Number = {{5}},
Month = {{MAY}},
DOI = {{10.3390/electronics8050505}},
Article-Number = {{505}},
ISSN = {{2079-9292}},
ORCID-Numbers = {{Hang, Lei/0000-0002-9336-9274
   Jamil, Faisal/0000-0003-1994-6907}},
Unique-ID = {{ISI:000470999900036}},
}

@article{ ISI:000446107500005,
Author = {Sylim, Patrick and Liu, Fang and Marcelo, Alvin and Fontelo, Paul},
Title = {{Blockchain Technology for Detecting Falsified and Substandard Drugs in
   Distribution: Pharmaceutical Supply Chain Intervention}},
Journal = {{JMIR RESEARCH PROTOCOLS}},
Year = {{2018}},
Volume = {{7}},
Number = {{9}},
Month = {{SEP}},
DOI = {{10.2196/10163}},
Article-Number = {{e10163}},
ISSN = {{1929-0748}},
ResearcherID-Numbers = {{Sanchez-Gomez, Nicolas/K-3758-2014
   }},
ORCID-Numbers = {{Marcelo, Alvin/0000-0001-6250-9169}},
Unique-ID = {{ISI:000446107500005}},
}

@article{ ISI:000436087400001,
Author = {Fan, Kai and Wang, Shangyang and Ren, Yanhui and Li, Hui and Yang,
   Yintang},
Title = {{MedBlock: Efficient and Secure Medical Data Sharing Via Blockchain}},
Journal = {{JOURNAL OF MEDICAL SYSTEMS}},
Year = {{2018}},
Volume = {{42}},
Number = {{8}},
Month = {{AUG}},
DOI = {{10.1007/s10916-018-0993-7}},
Article-Number = {{136}},
ISSN = {{0148-5598}},
EISSN = {{1573-689X}},
ORCID-Numbers = {{Fan, Kai/0000-0001-6870-6657}},
Unique-ID = {{ISI:000436087400001}},
}

@article{ ISI:000428646500002,
Author = {Kim, Henry M. and Laskowski, Marek},
Title = {{Toward an ontology-driven blockchain design for supply-chain provenance}},
Journal = {{INTELLIGENT SYSTEMS IN ACCOUNTING FINANCE \& MANAGEMENT}},
Year = {{2018}},
Volume = {{25}},
Number = {{1}},
Pages = {{18-27}},
Month = {{JAN-MAR}},
DOI = {{10.1002/isaf.1424}},
ISSN = {{1055-615X}},
EISSN = {{1099-1174}},
ResearcherID-Numbers = {{Sanchez-Gomez, Nicolas/K-3758-2014}},
Unique-ID = {{ISI:000428646500002}},
}

@inproceedings{ ISI:000450065900012,
Author = {Hulea, Mihai and Rosu, Ovidiu and Miron, Radu and Astilean, Adina},
Editor = {{Miclea, L and Stoian, I}},
Title = {{Pharmaceutical Cold Chain Management Platform Based on a Distributed
   Ledger}},
Booktitle = {{2018 IEEE INTERNATIONAL CONFERENCE ON AUTOMATION, QUALITY AND TESTING,
   ROBOTICS (AQTR)}},
Series = {{IEEE International Conference on Automation Quality and Testing Robotics}},
Year = {{2018}},
Note = {{21st IEEE International Conference on Automation, Quality and Testing,
   Robotics (AQTR THETA), Cluj Napoca, ROMANIA, MAY 24-26, 2018}},
Organization = {{Inst Elect \& Elect Engineers; Inst Elect \& Elect Engineers Comp Soc
   Test Technol Tech Council; Tech Univ Cluj Napoca, Dept Automat; IPA R\&D
   Inst Automat Ctr Technol Transfer; Bosch; Arqes; Emerson; Baumann
   Automat; Primaria Consiliul Local; Centru Cultura Urbana Casino;
   Ministerul Cercetarii Inovarii; Casino Urban Culture Ctr Cluj; Cluj
   Napoca City Hall \& City Council; Romanian Gov, Minist Res \& Innovat;
   IPA R\&D Inst Automat, Cluj Napoca Subsidiary; Tech Univ Cluj Napoca}},
ISSN = {{1844-7872}},
ISBN = {{978-1-5386-2205-6}},
Unique-ID = {{ISI:000450065900012}},
}
Validity of the findings: I would like to see some more scientific sources included (see above) before drawing a conclusion.
Additional comments: Your work is very important and well motivated. The paper need some enhancements as mentioned above. Feel encouraged to spent some more time on the literature review and take the mentioned sources into account for the state of the art and the conclusion/comparison.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: THE DISTRIBUTED LEDGER TECHNOLOGY AS A MEASURE TO MINIMIZE RISKS OF POOR-QUALITY PHARMACEUTICALS CIRCULATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Thank you for taking the recommentations into account.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CREATION OF MUTANTS BY USING CENTRALITY CRITERIA IN SOCIAL NETWORK ANALYSIS
Review round: 1
Reviewer: 1

Output:
Basic reporting: The English in the paper has to be improved substantially, as it is very difficult to understand and follow the arguments included in the paper. Table 7, for example, is not in English but Turkish.

Sufficient background information and references have been provided but again due to the bad use of the language it is not very easy  to follow them.

The structure needs to be revised as well, just Table 7 shows the deficiency in that respect.

The paper is self contained and relevant results to the hypothesis have been obtained. But the presentation of the results (e.g. Figure 3) is not very clear.

Otherwise the paper is written in accordance with formal rules.
Experimental design: The paper proposes  a mutant selection method using different criteria derived from social network analysis. Mutants are artificially introduced faults used to test the efficiency of tests developed for Finite state machines. It is important to generate include these into systems efficiently in order to measure the performance of the test sets.

The question adequately designed and and has some importance and contribution, but in my opinion it is not a very significant one.

The author has performed rigorous analysis, and results reflect these although not very well presented.

These results would be useful for future researchers who may want to compare their results with others.
Validity of the findings: As stated before there is some novelty in the findings but the impact may be questioned.

The author presented the study in detail but again these are veiled due to language usageç

The conclusion may be improved, but contents are appropriate for the findings.
Additional comments: None


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CREATION OF MUTANTS BY USING CENTRALITY CRITERIA IN SOCIAL NETWORK ANALYSIS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The work presented in this study is qualified for publication, however, there are some errors related to the spelling and structure of the work, please modify them.
Experimental design: The experiments are designed properly.
Validity of the findings: The data provided by the user is appropiately stated.
Additional comments: 1.	Page 8, please have a check at Table 4.
2.	Could you please check the names of Table 7?
3.	Please check Table 9, why the name of the table is below the table while others are at the above？


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: CREATION OF MUTANTS BY USING CENTRALITY CRITERIA IN SOCIAL NETWORK ANALYSIS
Review round: 2
Reviewer: 1

Output:
Basic reporting: The paper has been significantly improved and became more readable hence understandable. The tables and figures were also corrected. There are still some minor English mistakes which may be useful to revise.
Experimental design: The aims and scope has been more clearly defined.
Validity of the findings: The findings are clearer after the revision.
Additional comments: None


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TEXT-MINING FORMA MENTIS NETWORKS RECONSTRUCT PUBLIC PERCEPTION OF THE STEM GENDER GAP IN SOCIAL MEDIA
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper treats a relevant and debated topic in a structured and methodologically sound manner. Overall I found it very easy to follow.

I have very few minor suggestions:
1) The caption of Figure 1 should explain what black words and grey links are. They are likely to be associated with neutral sentiment but it is better to be clear.
2) The caption of Figure 2 has inverted top/bottom labels.
3) At page 10 the subsections have unneeded numbering
Experimental design: No comment
Validity of the findings: No comment
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TEXT-MINING FORMA MENTIS NETWORKS RECONSTRUCT PUBLIC PERCEPTION OF THE STEM GENDER GAP IN SOCIAL MEDIA
Review round: 1
Reviewer: 2

Output:
Basic reporting: The manuscript “Text-mining forma mentis networks reconstruct public perception of the STEM gender gap in social media ” introduces the applications of Forma Mentis Network (FMN)-approach to explore how gender gap in STEM (science, technology and mathematics education) is perceived in social media. The research reported in the manuscript continues the research line the author has developed in previous publications, mentioned in refs. Stella et al 2019,Stella & Zaytseva 2019, Stella 2020. In the new contribution, the method of analysis to explore the mental mind sets through textual analysis is developed further, and it is called Textual Forma Mentis Network (TFMN). The new method makes use of edge-coloured graphs and the multiplex structure of the networks. The FMN methods as it was applied to lexical networks has gradually, step-by-step, been develop to current TFMN method which is methodologically quite advanced and powerful, and is developed in a way that it can adapted to big data sources. I find it exciting and welcome new technique that opens up entirely new types of approaches in science education research and subjects related to STEM. These kinds of methods provide valuable complementary information to more traditional studies based on phenomenographic or interpretative studies, which are common in case of gender gap and gender issues.

The topic of gender gap is issues related to gender bias are extensively researched topics in science education research and no common single reasons for the existence of gap and bias are found. However, it is clear that expectations, stereotypic images and role models play key roles in formation of these gaps. The present study approached the problem from viewpoint how public perceives the gender gap in discussing STEM related topics. This is well chosen viewpoint, which provides background information what kinds of related views, beliefs and conception the public discussion reveals. It is still far from being able to resolve the issues, but provides a big picture of how the topic is perceived. The author anchors the study and its viewpoint very carefully in the recent research, which demonstrates that biased and distorted mind-sets are closely connected to the gender gap. The TFMN method is then designed and adapted to reveal indicators of such distorted minds sets from the twitter messages.

The context of the study and the current understanding of how gender gap and mind-sets related to gender issues in STEM are carefully discussed in the manuscript and a very good and insightful summary is provided. As such, the summary is very valuable because it not only summarise the recent research but also provides a balanced synthesis of what is known of the topic. It also provides strong argument in favour to focus on analysing views as they are expressed in public discussion. Such contribution shows a good scholarship and good acquaintance with the subject and is valuable contribution on the current research.

The method (TFMN) is then very fluently and even seamlessly developed to answer the questions raised. The study and its approach is thus skilfully composed so that the questions it attempts to answer the theoretical basis and the applied methods form a coherently composed whole. The study is exemplary in that regard.
The exposition of the material, results and discussion are all very clearly done. The language and structure of the manuscript appear to me quite perfect (though I am not native in English language) throughout. In addition, the stucturisation of the text makes it easy to follow and the argumentation folds out easily and with clarity.
Experimental design: The experimental design is for analysis of datasets from social media. The dataset is here the online perception of the gender gap in STEM as retrieved from Twitter.  The collection and curation of the dataset is explained and documented carefully and in details. The method to construct the textual forma mentis network is explained in main text, and in more details in the supplementary material. In addition, the data includes the cognitive data used for detecting conceptual overlap in meaning between words and their valence. The data sets are rich and they support each other, providing basis for very thorough picture of the phenomena, which are studied.

The method of analysis is developed on basis of work already reported in Stella et al. 2019 and Stella 2020. The new TFMN method is also tested carefully used the previous forma mentis networks. In addition, in the construction of networks the synonymous expression are paid careful considerations and taken into account. These aspects greatly increases the reliability of the method. Method is also described in detailed enough way to allow replication of the study in similar contexts and by using similar data collection.

The only aspect, the author could consider is if the meaning of edge-coloured graphs and multiplex networks needs to be explained in a bit more accessible way after or before figure 1. Probably these concepts are well known for most readers of PeerJ, but in the lucky case that also science education researchers find this paper, it might be helpful to provide a sentence or two to explain those concepts in more detail. However, I leave this as an option to be considered by author.
Validity of the findings: The context of application of the method is to my knowledge new. The context is very important, currently extensively researched but still in need of better understanding of the factors affecting the formation of gender gap and biased attitudes. The present work bring a new viewpoint on this problem, and provides new powerful tools to analyse the problem. The results of the study are discussed from many perspectives. In addition, the limitations of the study are pointed out. What I find as a particular strength of the study is how it manages to join the emotional and conceptual aspects, as they are present in the twitter discussions. This shows the complexity of problems, where conceptual and rational aspects are mixed with emotional ones.

The results are nearly self-evidently valid within the scope of the study and how it is framed; the public view of gender gap as it appears on twitter. Of course, the twitter tweets and social media discussions reveal what topics are of interest, how public picks out topics they tweet etc., and as such provide window to the mind-sets. This is part of the problem, but then, in addition, there are factors related to institutional organisation, social dynamics and societal factors which affect how the mind-sets expressed in e.g. twitter are formed. Of course, it is not reasonable to assume that in the present study, such dimensions need to be discussed, but in the end of the study, authors note that it could be interesting to extend the study in direction of social dynamics and sociosemantics. I think this is very good idea and the methods the author presents, seems to be perfectly well adapted for such extensions.
Additional comments: The author have produced a very interesting study, which introduces powerful methods to explore mind-sets and provide information on basis of big-data. The topic of the study, the gender gap, is very important and currently one of the most researched topics in STEM, but still many questions remain open. The present study is a valuable new view on that discussion. Methodologically, the study demonstrates the power of network approach in complex issues related to STEM research and helps to see new, complementary ways to advance research and provide access to information not available easily or even at all by using methods that are more traditional. I think, that research in STEM urgently needs to adapt this kinds of new approaches and start making use of the network science and its methods to make progress, and even more importantly, to start to conceptualise the problems from complex system perspective. I hope that this study will find it ways to hands of scholars and researcher in science education, since I believe that it would have a significant impact in that area of research.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TEXT-MINING FORMA MENTIS NETWORKS RECONSTRUCT PUBLIC PERCEPTION OF THE STEM GENDER GAP IN SOCIAL MEDIA
Review round: 1
Reviewer: 3

Output:
Basic reporting: I thought this paper had an interesting combination of methods applied to an important topic. However, I also found the story hard to follow in several places, and I ended with several questions about the analysis and its implications. So there's a lot of questions/suggestions below, but I hope they will help to clarify and strengthen the paper.

Language: The article was written in technically correct English and was unambiguous at the sentence level. There were some ambiguities/unclear areas at the conceptual/paragraph level, which I comment on below.

Literature/background: Appropriate prior literature was referenced. I believe some additional background is needed to help link the ideas required to understand this paper. There are several complex elements in the "story"--text analysis theory, implicit bias and the gender gap in STEM, and network analysis--and very few readers will have expertise in all areas. Figure 1 does a good job of clarifying what makes up a TFMN. It’s less clear how that network object connects in the analysis to "valence auras." Similarly, "mental lexicon" is referred to frequently in the introduction, but I'm not sure what it actually means in the analysis. Is it possible to add a schematic (possibly expanding Figure 1), showing either a "toy example" of the rest of the analysis steps, or a conceptual diagram of how the stages of analysis connect to each other? I found the paper very interesting but also very hard to follow, because the theory is so elaborate, and I kept flipping back and forth to try to understand how all the pieces fit together.

Structure, figures, tables, raw data: The article format follows standard sections, and the figures are relevant. Much of the figure text is small and difficult to read without magnification, especially on the network diagrams. There were several other clarity issues for figures:

Questions about figures
1. Figure 3: I've read this caption several times and still don't follow, especially the "Top" part. "Man" is in the top row, not the bottom row as is stated. The bottom/top and left/right structure described in the caption doesn't seem to match the structure of what’s shown. For example, the bottom left position does not seem to show all neighbors of the word "women" (as the caption says), but an extracted subset. The top/bottom distinction for the "Networks on the right" is also confusing. Does the bottom right panel show all words that were connected to "person," which happens to be many fewer than the number connected to "man," of which only a subset are shown here?

Basically, I think the attempt to explain this figure using rows and columns doesn't help because there are too many qualifiers and exceptions to the categories even within the four-panel "Top" part of this figure. (In contrast, this structure worked much better with Figure 4.) Talking about the "top of the top" and "bottom of the top" just muddies things further.  I suggest breaking Figure 3 into at least two figures (the "Top" and "Bottom" parts), and at least consider further breaking apart the "Top" to more clearly step through what each panel of it is showing. Having two views of the “woman” network makes it very hard to follow which one is analogous to what’s happening in the right-hand column (confusingly, I think the top-left view is analogous to the bottom-right panel, and the bottom-left is showing the community view, analogous to the top-right?).

2. On Figures 3, 4, and 5, the emotional profile plots have different vertical scales, which makes it hard to compare them and is visually misleading (e.g., at a glance the profile for “women” has fewer joyful associations than “men,” but it actually has a higher percentage). If it’s meaningful to compare those plots between Figures 3, 4, and 5, they should be on the same scale--if it’s not meaningful, that should be explained.

Minor issues:
Figure 2 caption: The "Top" and "Bottom" labels are switched.
Figure 4 caption: The "Bottom" caption should say "gender" and "gap" rather than "woman" and "man," I believe.

The full raw data is not available, but Tweet IDs to retrieve it are provided.

Self-contained, relevant results: I believe the paper is self-contained, after some clarifications.
Experimental design: Aims and scope: The paper is within the stated aims and scope of the journal.

Research questions and identified knowledge gap: The paper does not state a specific research question, but explores one application of TFMNs. The knowledge gap around this application is well articulated.

Rigorous investigation/Methods described in sufficient detail: Several parts of the data collection and analysis need to be clarified. My data collection questions also bear on the authors' claims in the Results and Discussion (further comments on that in the Validity and Findings section of this review).

Data collection questions:
1. Were all tweets in the thread gathered, or just the ones containing the four search terms? This is important because many substantive conversations on Twitter happen around multi-tweet threads, and some tweets may lack the keywords even though they are on the same topic. I can see arguments for or against including all threaded tweets, but the paper should clarify which approach was used.
2. Similarly, were comments (replies) from other users collected and analyzed, or only the original tweet (plus any comments that also had keywords)? Excluding comments from the analysis could seriously skew the findings, because it omits the insults, harassment, and general "trolling" that often come in response to online discussions of gender gaps in STEM. If comments were not included, this should be stated clearly, along with the rationale.
3. Less a question than a comment, but the set of keywords used to find tweets seems like it might filter for a pretty academic set of conversations. ("Gender gap," in particular, is the kind of phrase I hear mostly in diversity talks, so seems likely to find people already "in the system.") This is relevant to the gender gap in STEM, of course. But there are other conversations that might be less likely to show up--for example, people talking about workplace sexism they experience, or the gendered slurs and other pushback that come up when people talk about this online (especially women or gender non-conforming folks).

Analysis questions:
1. "Concepts" and "words" seem to be used interchangeably. If that's true, it should be stated explicitly. Otherwise, places like page 9, line 389 or page 10, line 451-452, are confusing because saying "concepts" implies an additional layer of analysis (beyond words -> nodes, find most central nodes).
2. A single TFMN was made by pooling all 10K Tweets together, right? Are syntactic network links only drawn within sentences, or can they bridge sentences in a Tweet? If they can cross sentences, how are borders between Tweets in a thread handled?
3. These networks are multiplex (with syntactic and semantic edges), but the closeness measure defined is for single-relation networks. How was it adapted for multiplex networks (alternately, how were the two edge types combined for use in equation (1))?
4. Are the networks weighted by frequency of occurrence, or unweighted? If they’re weighted, how is that used in closeness centrality calculations? If they’re unweighted, isn’t that throwing a way a lot of information?
5. Page 10, line 419, refers to “the networked mindset of 59 complexity science researchers.” I'm not sure what this refers to--were the "Complexity explained" paragraphs by 59 different people, or what is this?
6. The benchmark text from the Complexity Explained booklet doesn't seem to be very similar to the target text (thousands of tweets). For that reason, it's not clear to me that it's a good benchmark. Does performance on a few well-ordered paragraphs mean that we can assume that the method works for tweets, which are filled with hashtags, abbreviations, informal phrasing, and emoji? It would be good to at least comment on the differences, and why this benchmark is believed to be appropriate anyway.
Validity of the findings: Underlying data: The raw data is not provided, but it should be possible to reconstruct the data set given the list of tweet IDs. Source code is provided for analyzing a cleaned data set.

Conclusions: As stated in part 2, I'm concerned that the results may have been biased toward positive words/content if only tweets and not their replies were used. Or it's possible that positive voices in the replies would "prevail"--but if they weren't analyzed, that isn't known, and should be clearly stated as a limitation. The abstract says that "This analysis thus identified Twitter discourse as promoting a mostly stereotype-free, positive/trustful perception of gender disparity, of relevance for closing the gap." This is a fairly sharp contrast to the many complaints about Twitter's failure to impose meaningful anti-harassment policies. It's possible that the subset of words used to gather tweets filtered out most of that phenomenon. Alternately, maybe there really is more positive-aspected talk, and people just remember the harassment and threats more because of the damage they cause. Regardless, I think this study will come off as disconnected from reality if it makes that claim in the abstract without also discussing Twitter's famous problems in this regard, and how the author interprets the results in light of those issues.
Additional comments: To reiterate, I think this is a neat combination of methods + problem, but in its current form I think the paper has several serious ambiguities. I hope my comments are useful.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TEXT-MINING FORMA MENTIS NETWORKS RECONSTRUCT PUBLIC PERCEPTION OF THE STEM GENDER GAP IN SOCIAL MEDIA
Review round: 2
Reviewer: 1

Output:
Basic reporting: The author resolved all my concerns about the manuscript.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: no comment


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MANAGING SLOW-MOVING ITEM: A ZERO-INFLATED TRUNCATED NORMAL APPROACH FOR MODELING DEMAND
Review round: 1
Reviewer: 1

Output:
Basic reporting: (a) Some examples where the language could be improved include lines 95,99,281 and so on.
(b) Almost all pictures are blurry, and some of them even affect the judgment on the experimental results.
(c) The layout of the table is not good and worthy of improvement.
(d) It's better to add figures to illustrate the execution of the proposed method.
Experimental design: (a) The  comparison experiment on simulation data is clear, however, there is no compare on real data in section 3.3. The simulated data conforms to the article's data distribution assumption, and the results are certainly not bad. It is persuasive to beat other methods on real data sets.
(b) In practical applications, as merchants will not only sell one product, the inventory management of multiple products instead of single product should be analyzed.
Validity of the findings: (a) Please correct the mistakes in Table 3  RankTOPSISorder(%) .
(b) The raw data in section 3.3 is needed.
Additional comments: This paper adapted existing work for slow-moving management, using ZINO and ZINOsum distributions to model DPUT and LTD.  The authors  did a lot of work, but there are still some shortcomings. The aspects that need improvement are listed above.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MANAGING SLOW-MOVING ITEM: A ZERO-INFLATED TRUNCATED NORMAL APPROACH FOR MODELING DEMAND
Review round: 1
Reviewer: 2

Output:
Basic reporting: - It is important to clarify that the mixture distribution defined here is a zero-inflated Truncated Normal - the Normal distribution component's domain is defined in positive Real only (at least from the code).
- The extensive use of acronyms is a bit distracting: I recommand keeping only DPUT, LTD, TOPSIS and ZINO, and spell out random variable (RV), exponential smoothing (SES), and Croston's (CROST) / Croston's sum (CROSTsum).
- Line 90-91 since K is a fixed value, there is no need to describe the mean and variance.
Experimental design: In general, the author should improve the supplementary code:
- in ZINO.R it is better to use the standard Normal density function dnorm in dZINO instead of your own implementation.
- A bit of formating would help the readability of the R codes (using tools like https://styler.r-lib.org/index.html).
- If I understand correctly, Algorithm 1 is implemented in jitter.R - It would be great that this is made more clear either in the name of the file or the comment in the code.
- In the TOPSIS simulation code, the optimization with the iterative process basically copies and paste of the same lines multiple times - these should be rewritten into a for loop or a while loop
Validity of the findings: - Table 3 is badly formatted and quite challenging to read. It would be ideal if it could be presented as a figure.
- Figure 1 and 2 needed to be better formatted so every subplot are consistent in size, also you should increase the resolution and font size.
- I am not sure I understand the B-spline analysis, and the phrasing that nonlinear correlation between the r and v could be "explained" by B-spline is problematic and not theoretical motivated. To me it is more an exploratory analysis on the underlying nonlinear correlation.
Additional comments: The current MS introduced a slow-moving item management model (intermittent DPUT) that takes advantage of the flexibility of mixture distribution (i.e., zero-inflated normal) to better distinguish between non-zero and zero DPUTs. The author shows the advantage of the proposed method compare to models with other distribution for DPUT and LTD using TOPSIS.

The MS is well written, and I only have some minor comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MANAGING SLOW-MOVING ITEM: A ZERO-INFLATED TRUNCATED NORMAL APPROACH FOR MODELING DEMAND
Review round: 1
Reviewer: 3

Output:
Basic reporting: This paper presents Managing slow-moving item using of intermittent demand per unit time and lead time demand.

I would suggest  to accept this submission after getting the following major modifications incorporated:
- it is suggested to get the draft proof-read by a language expert to improve the writing of the draft
- the reference list should be updated with more recent references from strong publication venues
- in section 2, the subsections should be further divided into the following subsections: 2.1, 2.2, 2.3, and 2.4
- in subsection 3.2 and 3.3, Figure 1, 2, and 3 should be redrawn with better visualization
- in subsection 3.1 and 3.3, Table 3 and 7 are mixed up, it should be redrawn
Experimental design: - in subsection 3.3, Figure 3 shows a very straight data line through the plot, experiments should be done with more data to show
- subsections 3.1 and 3.2 should be discussed in details with more experiment setting
- in subsection 3.4, more experimental results should be added to the proposed statistical distribution inventory model
Validity of the findings: Novelty, clarity of expression, and background and related work are good but it lacks the experimental results.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MANAGING SLOW-MOVING ITEM: A ZERO-INFLATED TRUNCATED NORMAL APPROACH FOR MODELING DEMAND
Review round: 2
Reviewer: 1

Output:
Basic reporting: A.	All pictures are still blurry. Even the original eps version is not clear enough to view.  Please enhance the resolution to prove originality.
B.	The same items in Table 3 can be merged.
Experimental design: no comment
Validity of the findings: no comment
Additional comments: If the resolution of figures was improved, the paper can be accepted.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MANAGING SLOW-MOVING ITEM: A ZERO-INFLATED TRUNCATED NORMAL APPROACH FOR MODELING DEMAND
Review round: 2
Reviewer: 2

Output:
Basic reporting: no comment
Experimental design: The author should went through the code again to make sure naming is consistent since ZINO is now renamed to ZITNO.
Validity of the findings: no comment
Additional comments: The author have addressed all my comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MANAGING SLOW-MOVING ITEM: A ZERO-INFLATED TRUNCATED NORMAL APPROACH FOR MODELING DEMAND
Review round: 2
Reviewer: 3

Output:
Basic reporting: The revised version is in improved form but I would suggest the following modification:
- update the literature section from the past 3 years papers
- Figures 1 and 2 in sections 3.1 and 3.2, respectively, should have clear labels
Experimental design: In the revised manuscript, the response from the authors is satisfactory.
Validity of the findings: Section 4 should address possible future research problem domains.
Additional comments: Overall, the authors have responded satisfactorily and the manuscript is now in an improved form. I would suggest enriching section 4 by adding some comments on low performing traditional methods as well. Moreover, in future work, some concrete directions should be added to help the reader what kind of research problems in this domain are still unaddressed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DATA IN THE TIME OF COVID-19: A GENERAL METHODOLOGY TO SELECT AND SECURE A NOSQL DBMS FOR MEDICAL DATA
Review round: 1
Reviewer: 1

Output:
Basic reporting: The paper is poorly written and organized
Experimental design: The design of experiments is not upto mark.
Validity of the findings: Unable to obtain the validity analyses, author should consider some reputed journal's paper to revise the paper.
Additional comments: The paper is poorly organized.
There is no relationship between the COVID-19 and the written paper, except authors mentions somewhere in the entire manuscript.
I am unable to obtain the novel contributions of the paper.
Authors should discusses the motivation and contribution more efficiently.
This paper needs to be improved carefully prior to the publication in the reputed journals.

In current form, I am unable to recommend this paper for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DATA IN THE TIME OF COVID-19: A GENERAL METHODOLOGY TO SELECT AND SECURE A NOSQL DBMS FOR MEDICAL DATA
Review round: 1
Reviewer: 2

Output:
Basic reporting: In this paper, authors discuss various NoSQL DBMS types, compare their difference, and propose a method to choose the most appropriate type for a certain application.
Experimental design: In the algorithm, the authors describe an algorithm to select a NoSQL DBMS based on various criteria.
Validity of the findings: This is a paper that does a survey of various NoSQL DBMS systems, and propose an approach to select the most appropriate one for a certain case.
Additional comments: In this paper, authors discuss various NoSQL DBMS types, compare their difference, and propose a method to choose the most appropriate type for a certain application.

The title indicated that the research work is closely related to COVID-19, however, the content of the paper did not have much of a discussion. The quality of the paper could be further improved if the authors could provide more COVID-19 related work.

Also, there are some syntax and grammar errors. For example, in line 836, "This paper offers a set of web services that four encryption cryptosystems.", etc.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DATA IN THE TIME OF COVID-19: A GENERAL METHODOLOGY TO SELECT AND SECURE A NOSQL DBMS FOR MEDICAL DATA
Review round: 2
Reviewer: 1

Output:
Basic reporting: Paper is still not in proper format. No heading and sub headings are defined.
Experimental design: Although authors have added suitable diagram but the explanation of diagram is ambiguous.
Validity of the findings: Authors have added too many snapshots. Why so??
Additional comments: 1. Many potential references are missing such as Classification of COVID-19 patients from chest CT images using multi-objective differential evolution–based convolutional neural networks, Classification of the COVID-19 infected patients using DenseNet201 based deep transfer learning,  Automated deep transfer learning-based approach for detection of COVID-19 infection in chest X-rays.
2. Too many grammatical and typo mistakes are there.
3. Paper needs a good formatting.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DATA IN THE TIME OF COVID-19: A GENERAL METHODOLOGY TO SELECT AND SECURE A NOSQL DBMS FOR MEDICAL DATA
Review round: 3
Reviewer: 1

Output:
Basic reporting: Authors have improved it quite well.
Experimental design: Aim of design is clear
Validity of the findings: Findings seem staisfactory
Additional comments: Authors have improved the manuscript quite well. It can be accepted for publication.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEMOCRATIZING AI: NON-EXPERT DESIGN OF PREDICTION TASKS
Review round: 1
Reviewer: 1

Output:
Basic reporting: Overall, the article is written in professional language with minimal typos. Supplementary materials are well-appreciated as they were critical for understanding the experimental design. There are several areas that require additional clarification, restructuring, and/or modifications.

There are several terminologies and concepts in the article such as “meaningful task”, “interesting task”, “learnability”, “ML application” (as opposed to ML model), “creative task”, “interesting task” and “task importance” which are sprinkled throughout the article with clear definition and subsequently lack clear operationalization of those concepts. Overall, I would encourage the author to pay attention to defining all concepts (especially those mentioned in research questions and those that were used in analysis) and provide sufficient background in their relevance before using them in methods and analysis. For example, the research question 1 asks if non-ML experts can design “meaningful” prediction tasks. What does it mean for a task to be meaningful (and for whom or from whose perspective) and how do you measure meaningfulness of a task? The author hints at importance and learnability as two ways to potentially measure meaningfulness of tasks, and it would be great if the authors made well-articulated arguments as to why those two concepts would be associated with meaningfulness. Learnability may be a well-known concept in machine learning community, but its definition is debatable. Since task learnability is a significant portion of the analysis, I would encourage authors to provide some background into its definition and why it is relevant for measuring meaningfulness. “Prediction tasks lack learnability when there is insufficient diversity in the dataset” is also very confusing. I think the author means class balance rather than actual diversity.

I would also encourage the author to make direct connections between all research questions proposed and their corresponding methods, analysis, and results. Without these anchors, the results are difficult to follow. This could be achieved by labeling research questions and pointing to the appropriate research questions in the subsequent sections or organizing the sections to directly align with the research questions. There’s also an abrupt change in topic between sections 3-4 and section 5. It seems like the article consists of two studies, but one of them has dedicated methods and results sections while the other has results as a subsection. I would encourage reorganizing the structure to clearly delineate separate sections and to have a common pattern of presenting the study such that the readers can easily link different results to different research questions and study designs.

Abstract could also be restructured into: (1) Problem or need statement, (2) Current state of the art, (3) Summarization of research goal, (4) Research design/methods/analyses, (5) Summary of results, (6) Summary of implications if available. Currently, the last sentence in the abstract paints a broad picture of the problem or need and is not directly connected to how this study has addressed the issue at hand.

There are a few suggestions for related works:

1. This article tries to examine non-experts’ ability to design prediction tasks. What’s critical to grounding this study is in discussing why it’s important for non-experts to design prediction tasks. The related works in this article is a bit lacking on this aspect while it has a lot of materials on crowd-based tasks related to ML. Why is it insufficient for the crowd/people to provide features? In motivating why non-ML experts (i.e., those with domain knowledge but no ML knowledge) need to be able to design ML tasks, I would encourage referring to publications on machine teaching by https://www.microsoft.com/en-us/research/group/machine-teaching-group/#!publications

2. Another missing background relevant to this article is a discussion around the process of designing a task by ML experts or non-ML experts and the challenges surrounding them. A well-designed process for non-experts should also work for ML experts. Does the MTurk task that the author has designed work equally well for ML experts? Which component or step of designing an ML task is most challenging for non-experts? The challenges that non-experts face must also be the same challenges that novice ML practitioners have faced before becoming ML experts. The article tries to answer this question by looking at experimenting with providing examples (i.e., priming), restricting inputs to booleans and numbers (i.e., representation), providing balanced set of labels, etc. I would encourage the authors to look at prior literature on the practice of ML (e.g., https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf) to draw connections to the current research questions and experimental designs.

3. The author talks about the similarity of the current study to existing ranking algorithms and tasks. I would encourage first discussing why ranking is relevant here and then discussing similar examples.
Experimental design: The authors propose importance and learnability ranking to determine if a prediction task is meaningful. Although the mathematical formulation and the subjective rating of determining importance and learnability seems valid, it is unclear why such a formulation is useful. This goes back to my prior comment about clarity of concept definitions, but perhaps more importantly the readers need to understand why learnability leads to meaningfulness and why that matters. Because learnability in this article is rather mathematical (can you separate the two classes) than practical (can you design a meaningful task), its relevance to non-experts is a bit lost in the analysis. Is a learnable task necessarily a meaningful task and an easy task for non-experts? It is difficult to evaluate experimental design because the narratives that support the design are weak, but perhaps there is a way to pivot these. Currently, there are bits and pieces but they are not threaded to tell a cohesive and compelling story.

The design of asking crowdworkers to propose a prediction task was clever in that the structure of target vs input questions and the types of data that they are working with helps workers think about prediction tasks more systematically by providing a framework. In my opinion, this is an important contribution for helping non-experts design prediction tasks that needs further exploration (i.e., without the structure, can non-experts design prediction tasks at all?). From that, learnability, importance, task characteristics analysis becomes the evaluation and description of the tasks generated with or without the structure. If the goal of the article was to help non-experts design tasks, then I would encourage the authors to consider organizing experiments into 3 contributions: (1) what tools (i.e., structure + examples) help non-experts (this also needs a baseline), (2) how to measure/characterize the quality of prediction tasks, (3) potential priming effect of examples. Methods and results were thoroughly explained. So threading the experiments in a clear narrative is encouraged.
Validity of the findings: The author did a great job detailing the findings and making sure the results are based on solid analysis and reproducible. I would have liked to see deeper dive in the discussions. For example, non-experts favored Boolean questions, but there needs to be some discussion on why. With ranking algorithm, there is an opportunity to provide feedback that helps with making prediction tasks “better” interactively. The second to last paragraph in discussion (442-448) needs some rewriting because it is unclear why researchers need to take their workload off of feature engineering or model fitting or what is exactly the issue of medical researchers requesting only prediction tasks focused on health care.
Additional comments: I believe the experiments and results are valid, but without a clear articulation of why, the article fails to deliver a convincing narrative and actionable takeaways. I would encourage taking a step back and asking the "why" question in putting the pieces together. I would also make sure to clearly state your research questions with all variables and concepts clarified and defined and making sure methods and results well align with the research questions.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEMOCRATIZING AI: NON-EXPERT DESIGN OF PREDICTION TASKS
Review round: 1
Reviewer: 2

Output:
Basic reporting: The paper is very well written and uses a clear and comprehensible style of writing. The ductus is scientific and technically correct.
The cited literature is sufficient and provides a good overview of the the topic.
Figures are suited to convey the intended message and are easy to read. Tables follow the standards in the field and raw data is shared in fighsare.
The article is understandable on its own, although some figures from the supplemental material could also be part of the original article (the preview of the survey form).
The document has some minor orthographic errors that need fixing.

E.g.
l. 206: Yet, (missing comma)
l. 118: Here, (missing comma)

The author uses the term "three-armed" experiment to indicate three different branches in a randomized controlled trial. This was unusual to see, but seems reasonable when reading the whole document. Maybe a figure showing the procedure in a flow chart would help understanding this more easily.
Experimental design: The set of experiments address a well-defined research question using crowdsourced participants from amazon turk. The experiments are well designed to address the research question, which is both relevant and meaningful.
The experiment addresses a research gap derived from literature and proposes a procedure to identify new ML-tasks from non-experts.
The methods used are suitable to answer the analytical questions and conducted with ethical review. Since data and methods are available and clearly defined, replication should be realizable by other investigators.
Validity of the findings: The findings are well documented and reflect both results and literature. Experimental effects are controlled for by a three-way randomized controlled trial. Conclusions address the problem statements from the introduction. Limitations are clearly defined and address a challenge with the experiment. Improvements for future research are clearly outlined.

I found only one aspect lacking from the whole research design was domain expertise and user characteristics. It may be interesting to see level of education and domain expertise from the sample. However, the findings are relevant without this information as well.
Additional comments: I was positively surprised with this manuscript, as it addresses a research area that is insufficiently studied. The proposed process seems reasonable, however it may need additional tailoring for more domain-specific and relevant tasks. Good work!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEMOCRATIZING AI: NON-EXPERT DESIGN OF PREDICTION TASKS
Review round: 2
Reviewer: 1

Output:
Basic reporting: First of all, I thank the authors for such thoughtful and detailed responses to the questions I raised. I am very pleased with the responses and the revision. I have minor suggestions for clarity and strengthening of the arguments, which might be subjective so bare with me.

The authors’ responses were direct, thoughtful, and clear, which I loved, but I’m not the only reader, so I would love to see those strengths shine in the paper. I would take the approach you took in responding to these questions in the actual paper content. For example, the statement of “The crowd/people will not be able to apply ML to a new area of their own interest if they can only contribute features to pre-existing problems in pre-existing areas.” is a much stronger statement than “Perhaps…?” (line 39-40) style of writing. Or “research gap” is a great phrase to use to position this paper. This point (of non-experts needing to define their own tasks in new areas to take part in ML) is so critical to this paper. I would strongly encourage stating this as a problem directly, rather than as a question.

Nit: The term “ML problem” and “ML prediction task” seem to be used interchangeably, and I would encourage the authors to provide clarity on these two or choose one. Are they identical or is there a nuanced distinction here that the readers need to understand? “ML prediction task” sounds to be closer to a function that maps an input (feature vector) to an output (label) (e.g., given TF-IDF counts from an email, what’s the probability that it’s related to travel or not?) whereas “ML problem” might be closer to describing a domain area that the non-experts might be familiar with (e.g., favorite travel destinations for hikers). Or perhaps, there is no distinction such that one term should be used throughout the paper.

IRB information seems to be duplicated. (line 282)

Line 503: Should it be “to power “pre-defined” machine learning tasks”? Powering ML methods seem more complex (computational/mathematical) rather than providing labels and features.
Experimental design: Again, thank you for providing clarity in the organization of the research questions and sections as well as thoughtful responses to questions/comments raised. I have a few high-level comments that hopefully can help improve the paper.

I really appreciate the extra description on the definition of learnability of a task. I have to be very honest that this is perhaps the hardest part of the paper to understand still. Again, the experiment design and results are valid. The issue is in understanding the transition from learnability to class balance to perceived/estimated class balance by crowdworkers. Based on the stated definition of learnability ("the ability for a predictive model trained on data collected for the prediction task to accurately generalize to unseen data") and one potential measure of learnability (“balance of class labels”), learnability is a property of the task AND the dataset (potential or train) rather than a property of the task by itself. Does your definition of a prediction task also include (potential) training data? The authors discuss this distinction in lines 391 (“this does not mean the questions are not meaningful, only that the data collected does not lead to useful predictive models”). So, what’s learnable? The task? The task + a train dataset? The task + a true distribution?

Another source of confusion is that class balance is one of many factors that influence learnability. It would be great to provide examples of other factors that influence learnability of a task (maybe...choice of models, features, label quality/consistency, VC dimensions from PAC theory?). The choice of “class balance” as one measure of learnability is understandable because it needs to be something that the crowdworkers can evaluate. But class balance seems to be an oversimplification of learnability. Class balance in the dataset is an important aspect, but moderate imbalance in the dataset is certainly learnable, especially if the prediction task is highly niche (the motivation for this paper is that we want to enable non-experts with their potentially niche problems to be able to ideate prediction tasks). The term “learnability” might be the red herring here. If it was just labeled as “estimated label balance” (such that the model can come up with a set of possible hyperplanes), it would be so much more straightforward. OR, it might be more helpful to position this in an AutoML scenario, where class balance is possibly the only thing that the non-expert should control (selection of model, feature engineering, ML theory is all out of scope for them?). So, instead, you can position "estimated/perceived class balance" as the only "controllable" factor for learnability by non-experts.
Validity of the findings: I was very happy with the findings in the first round, and the additional restructuring and clarification made the paper even stronger.

Method, analysis, and results for RQ1 seem to be sprinkled throughout the paper. What percentage of the solicited prediction tasks were meaningful (learnable AND important)? The analysis seems to decouple learnability (model performance, class balance) separately from importance (subjective rating), as also mentioned in discussion. Should RQ1 be rephrased to “Can individuals … design important or learnable prediction tasks”? Is there a value in defining “meaningfulness” as learnability plus importance in methods if meaningfulness (together) is only going to be mentioned in discussion?
Additional comments: Thank you for your detailed responses and revisions!


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEMOCRATIZING AI: NON-EXPERT DESIGN OF PREDICTION TASKS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The article presents a study with five sub-studies on the question if laypeople are able to design machine learning tasks, what task they design, if they are learnable, and how the quality of the tasks gathered can be increased.
In general, the article is written outstandingly well, the overall research methodology is sound, and the question addressed is quite relevant and well framed and argued in the article. Actually, this is one of the best articles I reviewed in the past months and in my opinion, the article can be published with only minor corrections (technically this would be a minor revision, but I do not need to look at changes). I hope the comments below help the author to make this article a bit sounder and more legible.

On the highest level, my two key suggestions are: First, a clearer presentation of the applied methodology (and the findings) in writing and also in form of illustrations (one bigger for an overview of the sub-studies and one for each study). Second, if possible, to make the article a bit more concise. I found the general approach and the breadth of this work really outstanding. However, the current form of the manuscript is in some parts a bit repetitive (due to the structure of presenting all methods first and then all results instead of an interleaved approach) and has a few —from my perspective—unnecessary “side stages”, such as the discussion about the exact procedure to rank the tasks (section 3.2.1, a reference would be enough, I would assume).
Experimental design: The experimental design of all sub-studies and the overall story of all studies combined is sound. My key reservations stem from the sometimes limited sample size(s) and—more importantly—the fact that it builds on MTurk (see for example https://link.springer.com/article/10.3758/s13428-019-01273-7 and other studies for an explanation of the limitations) for recruiting participants.

My suggestion would be to make the sample sizes and the number of targets generated/evaluated much clearer within the first sentences of the respective sections. Currently, all information is provided but not necessarily at the same spot or in the same structure (this would help readers to get this information much quicker).

Section 4.3 is interesting and relevant, but I had troubles following your approach. I would assume that a model is sufficient/learnable/… if it makes a significant better suggestion than a random choice. Your explanations sound more complicated (and with my background in social science, I had troubles following them completely ;) maybe you check if this section is clear enough.
Validity of the findings: Based on the sound approach, the findings appear to be quite sound. The only bigger reservation I have is the use of MTurk instead of other samples, but this is clearly mentioned in the article (and the implications might be discussed a bit more).
There are some smaller issues than can easily be fixed (see below).
Additional comments: The title is a bit too broad and should more clearly reflect the nature of the tasks collected. I also have some troubles with the breadths of the prefix “DemocratizingAI”, which is excellent “clickbait”, but does not fully reflect the work that was done in this article. Hence, I would advise the author to change the title, which would in turn also increase the “googleability” of this article. Maybe something like crowdsourcing prediction tasks or so…
Also, the study builds on MTurk cloud workers and thus the term “Democratizing” is especially troubling as these people are the (hidden) backbone of numerous research papers, training sets, and implementations, yet they do not participate sufficiently in the design of the tasks, usually do not get any credits etc., and do not participate from the created added value. Hence, is it really “democratizing” if we, from richer countries, exploit people in poorer countries for generating prediction tasks for fractions of a coffee?

The abstract is a bit vague and does not properly summarize the whole article. I would suggest to a) add the multi-step approach, the respective sample/trial sizes and b) shorten it a bit, as- for example- the last sentence does not really help researchers to quickly screen thousands of articles for relevance (it is currently rather an interesting teaser than an abstract).

The type of prediction tasks collected should be introduced much earlier in the manuscript. I read the article from top-to-bottom and, as the title was not clear about it, had quite some troubles to imagine what prediction tasks were to be crowdsourced. It is presented right in the article, but–at least for my preferences—a bit too late which makes it unnecessary difficult to read the article.

Method:

Most of the method(s) is argued and presented extremely well. Yet, I personally do not like the current structure of the method / result section that presents the five method sections sequentially and then the five results section. Instead, I would suggest presenting the five studies sequentially, as it would be easier to follow. However, I leave this open to the author or the journal’s requirements.

All information needed to understand the method and the results are provided, yet I sometimes had troubles finding them quickly. Thus, I suggest that for each of the individual method sections, the number of participants and the number of collected questions/items/data points should be clearly stated.
The overall research methodology and the individual studies should be illustrated using a diagram. This might, for example also include the sample size, parameters measured, number of items gathered etc. I firmly believe that this would increase the overall legibility of this article.

I find the term “workers” I bit troubling and would suggest using the term participants instead.

Minor tidbits

ll. 78. I don’t think that the term “wisdom of the crowd” is appropriate here, as you are rather doing crowdsourcing (the nuances are a bit different).

Figure 1: In the caption the part for the first figure is missing.
Figure 3: I would suggest adding SE or Ci-95% error bars to the diagram. Also, are factual and health/wellness questions distinct categories?

ll. 348. Suggestion: “To rank the /expected/ learnability…”. Also, I think that this whole paragraph would appreciate some shortening.

ll. 398: Make the results section brief but to not be as explicit about it (the shortness is good, but it reads as if you were unmotivated to write more… just remove the “in brief…” :).

Table 4: Please define a level of significance and stick to it. Usually one would use <.001, <.05, and (if really necessary) <.1. The reporting here is a bit chaotic, as some p’s are reported as p=.12 or p>.1.

ll. 410: Why chi^2 and not a t-test/anova? A factor (experimental condition) with two/levels and one ordinal or even metric variable would speak for an anova, which would also the interpretation and presentation of the results easier.

ll. 414: I see this interpretation of apparent but not significant differences frequently, but it makes no sense. Either there is a difference or there isn’t. Here, you argue that the difference is not significant, yet you discuss the difference in detail. Some reviewers would expect that non-significant differences are not reported at all. I would rather suggest to phrase this more neutral. Group A has x points, group B y points. Yet, the difference is not sig. (p=xxx).

Section 3.2, ll 195/196. A rather philosophical point: I don’t agree with the general definition of a “meaningful” task. I do see, however, that your definition is very useful within the scope of your study. I would suggest clarifying that this is just a working definition for the porpose of this article and that meaningfulness, in general, might be more than ML-compatibility.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DEMOCRATIZING AI: NON-EXPERT DESIGN OF PREDICTION TASKS
Review round: 3
Reviewer: 1

Output:
Basic reporting: Thanks for the new changes that address the comments. Please make sure to read through to ensure smooth transitions between paragraphs, especially around the newly added parts.

Just a small suggestion. Sometimes it helps to style these constructs that you measure (e.g., perceived learnability, perceived importance) with italics to make them stand out as something that you had defined as a construct earlier.
Experimental design: No additional comments
Validity of the findings: Just two small comments for Discussion section.

Line 460-466: I think it's a bit of overclaiming to state that non-experts can estimate learnability for classification (all of them?). They can estimate learnability for the classification tasks in this paper, but that's still a small set of all classification tasks. Generalizing beyond what's done in the paper should be called out as future works, with potential avenues or dimensions that other researchers should be looking at. Reiterating the study findings is great. But generalizing beyond it is the problem here. In that regard, please be careful in separating out what you've found in your work and what can be generalized beyond that.

Line 467: With the insertion of the learnability paragraph, this doesn't flow anymore. "these results" you refer to appear in 2 paragraphs ago. Could you make sure your stories are consistent here?
Additional comments: I'm happy with the latest changes around learnability and clarifications that were addressed based on Reviewer 3's comments.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: MODULAR AUTOMATIC DESIGN OF COLLECTIVE BEHAVIORS FOR ROBOTS ENDOWED WITH LOCAL COMMUNICATION CAPABILITIES
Review round: 1
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: This paper describes a comparative study of an automatic design method for swarm robotics systems. In particular, it investigates the effect of message exchange on generating effective collective behaviours for robot swarms using a developed automatic designed method named Gianduja. The study describes different variants of Gianduja based on the increasing complexity of exchanged messages (i.e., by increasing number of bits in a message) then compares Gianduja and its variants with another automatic design method (e.g., EvoCom
 and Chocolate).

Generally, this paper is properly organised and follows good and extensive experiments to provide sufficient conclusive evidence on the outcomes of this research. In my opinion, the results comprise useful message to the research community in the field of automatic design methods to generate collective behaviour of robot swarm.

The paper is well written with clear English language. To the best of my English, I could not spot any language problems in the paper (except for few missing spaces e.g., “the (pseudo-)reality gap. L659, P22”, “generated by Gianduja2E’performs”, L673, P22 ).

The paper provides elaborative supplementary materials which complement the material described in the paper.  Furthermore, the Appendix section provides further experiments to explain unclear findings in the paper and to provide empirical evidence on the conjecture of the authors about the results.

However, I have few comments to be addressed in the followings:

- Why the white area in the Stop experiment (in study A) is smaller than the white and black areas in other experiments. Although it seems that in the presence of massage exchange, only a few robots need to find the white spot to signal the rest of robots to stop. However, the larger size of the white area:
i) increase the probability of finding the spot within the evaluation time (i.e., 120 s).
ii) can accommodate the majority of robots inside it.
This could ultimately guide the optimization process to relying on finding the spot rather than on receiving a signal from other robots. It seems the smaller size of the spot adds bias toward relying upon message exchange to increase the fitness of the group. Please, add a justification text in the paper about why the white area in the Stop experiment is smaller than the areas in other experiments in study A.

- Line 343, p9 “For Chocolate, Gianduja, and EvoCom, the control software produced is run once also on real robots”. This statement is unclear to me. Do you mean only one evaluation trial? If yes, why only one? If not, please, mention the number of evaluations conducted using real-robots. I could not find this information in the paper. This information is important to provide better evidence that validates your findings.

- The experiment runs on real robots need to be explained. How the parameters of the objective functions are measured (what tools are used to do that). For example, In the Decision experiment, how do you measure that the robots are in which half of the arena at the end of the evaluation (after 120 s elapses). Given that some robots in continuous movements. Similarly, in Stop experiment, how to measure the robot is moving or not at the end of every evaluation. If you did this measure by visual inspection of the recorded videos, please mentioned that in the paper.

- In Figure2, p11,  the graphs do not show boxes for evaluations with real-robot for GiandujaE. Is that because you did these evaluations for Gianduja and since GiandujaE is derived from Gianduja and differs from it only in the way received messages are handled. If this is true, please, mentioned it in the paper.

- In probabilistic finite state machine graphs in figure3 and figure8, p12 and p18 respectively. Please indicates what are the symbols p, w, t, brd, rwm and att refer to. Please, do this in the figure caption or as a legend in the graph.

- In figure6, p16. It would be more clear if you used a different colour for the BECON robot to distinguished it from the rest of the group.