Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A TECHNICAL REVIEW AND GUIDE TO RNA FLUORESCENCE IN SITU HYBRIDIZATION
Review round: 1
Reviewer: 1

Output:
Basic reporting: Young and colleagues present a timely review that focuses on an important and largely unaddressed theme, a rational explanation of the diverse protocols for RNA FISH. The author’s description of diverse protocols with steps that have been inherited from earlier versions of uncertain utility to their current application, intermixed and often indistinct from elements critical to the success of the experiment, certainly resonated with my experience in this field in the past 15 years. Thus, I think the article will find a significant and interested audience.  The author’s execution of this aim is largely successful in the manuscript.
The subject matter of RNA FISH taken alone is much to broad to easily cover comprehensively, even if the focus is just on protocols.  Still, I find the treatment a little dated, as it misses a significant amount of innovation in RNA FISH protocols in the last 5 years which I think have made a dramatic impact on this field.  I would recommend the authors to look into some of these works.  I believe the push to higher throughput multiplexed detection and sensitive deep tissue detection has put greater pressure to optimize the protocols, and that these improvements to some degree benefit ALL forms of RNA FISH. Many of the modern protocols are variations of the earlier protocol varieties the authors already discuss, but I think their more stringent applications (and more recent use) can be used as a proxy to distinguish which protocols aspects are better.  In particular, recent work from the Singer, Raj, Cai, Zhuang, Pelkmans, and Gregor labs.
some examples:  Eng, …, Cai,. Nat. Methods 2017; Eng, Cai Nature 2019; Moffitt...Zhuang PNAS 2016a + 2016b; Shah…Cai Neuron 2016+2017; Sha…Cai Cell 2018; Battachi…Pelkmans Cell 2015; Chen…Zhuang 2015 Science, Battachi…Pelkmans Nat. Methods 2013; Eliscovich…Singer PNAS 2017;  Moffitt, .... Zhuang, PNAS  (2016a) + 2016b; Moffitt…, X. Zhuang Elife (2016), Shah…Cai, Neuron (2016) , (2017).  Shah, … Cai, Cell. (2018); Chen…Zhuang Science 2015; Levesque Raj Nat. Methods 2015. Little…Gergor 2013-2015.

Minor comments by line:
Line 86 “…to label DNA the same cell types…” it seems a word is missing here
Line 111, 127: I would have liked to see Raj 2006 cited in the introduction on the smFISH. Moreso than the later Nature Methods paper (Raj 2008, cited line 111 and 127), this 2006 first introduced the significant tile and high contrast smFISH which expanded dramatically in the years following.
Line 156-8: A reference here could help, even if the author does not have space to elaborate more “…but we will not focus on this aspect here. However, characteristics such as the GC content, the propensity to form secondary structures, the overall length and specificity and probe quantity and quality must be considered”.  The recent proliferation of purely synthetic probes in place of cloning derived probes has provided much greater control of characteristics such as GC content and the propensity to form secondary structures. E.g. Beliveau et al 2012 PNAS.
185 – I recommend boadening the time range, many modern protocols use shorter fixation for cultured cells, e.g. 10 minutes. (Chen…Zhuang 2015 ‘MERFISH’ on human cultured cells) or Skinner…Golding 2013 for e Coli.
~214 on alcohol fixation. Something could be said about “Turbo-FISH” protocol – a fast methanol based fixation.
229 – in discussion of ‘protection of native protein structure’ – it could be mentioned that alcohol solutions are generally denaturing and disrupt native protein structure, which can lead to loss of antigenicity for immuno FISH protocols (protein + RNA detection). The same is true of heat fix. GA fix can significantly reduce the permeability by providing to dense a crosslinking.
Line 244 – I would not recommend proteinase K treatment for Drosophila embryos. I think a more thorough search of the recent smFISH literature in Drosophila (not cited) will bear this out.  See Pare…McGinnis 2009 Curr Bio, Boettiger…Levine 2009 Science, Boettiger…Levine 2013 Cell Reports, Little…Gregor Cell 2013, 2011 PLoS Bio, I could go on.
Some discussion of the Pepsin treatment (in place of proteinase K) as developed more recently by the Singer lab could go well alongside the proteinase K discussion. (Bauxbaum…Singer 2014 Science).
Line 286: 50% formamide – 10 -50% might be a more accurate way to put it.  For example see the high resolution smFISH protocol described by Raj et al (2006 or 2008), and built on in recent smFISH work (see work from Long Cai and Xiaowei Zhuang’s groups on highly multiplexed smFISH protocols)
331 – have you validated the effect of EDTA on blocking RNases?  RNase are not dependent on divalent cations the way DNases are.
364 – While I agree with the author that generally hybridizations can not go ‘too long’, it might be instructive to discuss the ‘5 minute’ ‘TURBO-FISH’ protocol as a nice example of rapid hybridization.
444 – recommendations for positive controls:  Probes that are expected to give signal in every cell are not so easy to distinguish from background in every cell.  I think it would be good to recommend in addition or in place of these controls, to use a patterned control in a heterogenous tissue (something say that should label epithielial but not mesenchyme, or neurons but not others) or a cell cycle probe in replicative tissue or replicative cultured cells KI67, or the histones loci make good probe targets as the mRNA is long or abundant but cell cycle specific in expression).
Some conclusion paragraph to pull the review together at the end would help the flow.
Experimental design: N/A (this is a review)
Validity of the findings: N/A this is a review -- see comments above for minor suggestions of accuracy.
Additional comments: Young and colleagues present a timely review that focuses on an important and largely unaddressed theme, a rational explanation of the diverse protocols for RNA FISH. The author’s description of diverse protocols with steps that have been inherited from earlier versions of uncertain utility to their current application, intermixed and often indistinct from elements critical to the success of the experiment, certainly resonated with my experience in this field in the past 15 years. Thus, I think the article will find a significant and interested audience.  The author’s execution of this aim is largely successful in the manuscript.
The subject matter of RNA FISH taken alone is much to broad to easily cover comprehensively, even if the focus is just on protocols.  Still, I find the treatment a little dated, as it misses a significant amount of innovation in RNA FISH protocols in the last 5 years which I think have made a dramatic impact on this field.  I would recommend the authors to look into some of these works.  I believe the push to higher throughput multiplexed detection and sensitive deep tissue detection has put greater pressure to optimize the protocols, and that these improvements to some degree benefit ALL forms of RNA FISH. Many of the modern protocols are variations of the earlier protocol varieties the authors already discuss, but I think their more stringent applications (and more recent use) can be used as a proxy to distinguish which protocols aspects are better.  In particular, recent work from the Singer, Raj, Cai, Zhuang, Pelkmans, and Gregor labs.
some examples:  Eng, …, Cai,. Nat. Methods 2017; Eng, Cai Nature 2019; Moffitt...Zhuang PNAS 2016a + 2016b; Shah…Cai Neuron 2016+2017; Sha…Cai Cell 2018; Battachi…Pelkmans Cell 2015; Chen…Zhuang 2015 Science, Battachi…Pelkmans Nat. Methods 2013; Eliscovich…Singer PNAS 2017;  Moffitt, .... Zhuang, PNAS  (2016a) + 2016b; Moffitt…, X. Zhuang Elife (2016), Shah…Cai, Neuron (2016) , (2017).  Shah, … Cai, Cell. (2018); Chen…Zhuang Science 2015; Levesque Raj Nat. Methods 2015. Little…Gergor 2013-2015.

Minor comments by line:
Line 86 “…to label DNA the same cell types…” it seems a word is missing here
Line 111, 127: I would have liked to see Raj 2006 cited in the introduction on the smFISH. Moreso than the later Nature Methods paper (Raj 2008, cited line 111 and 127), this 2006 first introduced the significant tile and high contrast smFISH which expanded dramatically in the years following.
Line 156-8: A reference here could help, even if the author does not have space to elaborate more “…but we will not focus on this aspect here. However, characteristics such as the GC content, the propensity to form secondary structures, the overall length and specificity and probe quantity and quality must be considered”.  The recent proliferation of purely synthetic probes in place of cloning derived probes has provided much greater control of characteristics such as GC content and the propensity to form secondary structures. E.g. Beliveau et al 2012 PNAS.
185 – I recommend boadening the time range, many modern protocols use shorter fixation for cultured cells, e.g. 10 minutes. (Chen…Zhuang 2015 ‘MERFISH’ on human cultured cells) or Skinner…Golding 2013 for e Coli.
~214 on alcohol fixation. Something could be said about “Turbo-FISH” protocol – a fast methanol based fixation.
229 – in discussion of ‘protection of native protein structure’ – it could be mentioned that alcohol solutions are generally denaturing and disrupt native protein structure, which can lead to loss of antigenicity for immuno FISH protocols (protein + RNA detection). The same is true of heat fix. GA fix can significantly reduce the permeability by providing to dense a crosslinking.
Line 244 – I would not recommend proteinase K treatment for Drosophila embryos. I think a more thorough search of the recent smFISH literature in Drosophila (not cited) will bear this out.  See Pare…McGinnis 2009 Curr Bio, Boettiger…Levine 2009 Science, Boettiger…Levine 2013 Cell Reports, Little…Gregor Cell 2013, 2011 PLoS Bio, I could go on.
Some discussion of the Pepsin treatment (in place of proteinase K) as developed more recently by the Singer lab could go well alongside the proteinase K discussion. (Bauxbaum…Singer 2014 Science).
Line 286: 50% formamide – 10 -50% might be a more accurate way to put it.  For example see the high resolution smFISH protocol described by Raj et al (2006 or 2008), and built on in recent smFISH work (see work from Long Cai and Xiaowei Zhuang’s groups on highly multiplexed smFISH protocols)
331 – have you validated the effect of EDTA on blocking RNases?  RNase are not dependent on divalent cations the way DNases are.
364 – While I agree with the author that generally hybridizations can not go ‘too long’, it might be instructive to discuss the ‘5 minute’ ‘TURBO-FISH’ protocol as a nice example of rapid hybridization.
444 – recommendations for positive controls:  Probes that are expected to give signal in every cell are not so easy to distinguish from background in every cell.  I think it would be good to recommend in addition or in place of these controls, to use a patterned control in a heterogenous tissue (something say that should label epithielial but not mesenchyme, or neurons but not others) or a cell cycle probe in replicative tissue or replicative cultured cells KI67, or the histones loci make good probe targets as the mRNA is long or abundant but cell cycle specific in expression).
Some conclusion paragraph to pull the review together at the end would help the flow.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A TECHNICAL REVIEW AND GUIDE TO RNA FLUORESCENCE IN SITU HYBRIDIZATION
Review round: 1
Reviewer: 2

Output:
Basic reporting: (1)	The manuscript fails to include the recent development of the RNA-FISH technique. As a promising technique for RNA labeling and imaging, RNA-FISH has been improved in different ways to increase the throughput (e.g., Eng et al., Nature, 2019; Shah et al., Cell, 2018) and signal level (e.g., Marras et al., PNAS, 2019; Choi et al., Development, 2018). These new versions of RNA-FISH may point to the future direction of the field. Without mentioning these recent advances and discussing their technical aspects, the manuscript looks a bit outdated. Hence, the authors should include these new versions of RNA-FISH technique in their paper.

(2)	Figure caption is missing for Figure 1. Although Figure 1 is a simple figure, it has only been cited once in the main text (line 83) without a detailed description. A caption is therefore required to help explain the whole figure. For example, the term “Chromogenic ISH” only appears in Figure 1, but not in the main text. With a figure caption, it is hard to understand what the term means.

(3)	In lines 83 and 415, “(Figure 1)” and “(Figure 2)” should be “(Fig. 1)” and “(Fig. 2)”, respectively, according to the Instructions for Authors (https://peerj.com/about/author-instructions/#figures).

(4)	In Tables 1 and 3, the temperature of each step is not listed.
Experimental design: no comment
Validity of the findings: The manuscript lacks a summary or conclusion part. A review article should not merely be a collection of previously published papers. As required by PeerJ, a Literature Review should have a conclusion part identifying unresolved questions/gaps/future directions about the field (https://peerj.com/about/author-instructions/#literature-review-sections). In this part, the authors may summarize their review of the technical aspects of RNA FISH and provide their view on the future development of the FISH technique. This will help the readers quickly understand the main points of the paper.
Additional comments: This paper reviews the technical aspects of RNA-fluorescence in situ hybridization (FISH), which serves as a powerful tool for visualizing the spatial distribution of specific RNA species in cultured cells, tissue sections or whole mount preparations. Focusing on the core steps of RNA-FISH procedure, i.e., tissue preparation, hybridization, and post-hybridization washing, the authors summarize reagents and experimental conditions commonly used in these steps and explain their functional roles. Also, the authors discuss probe selection and necessary controls for a robust FISH experiment.

In recent years, RNA imaging techniques, including RNA FISH, have been reviewed by multiple papers (Tutucci et al., Annu. Rev. Biophys., 2018; Pichon et al., Mol. Cell, 2018), most of which focused on comparing different RNA labeling and imaging techniques. However, a detailed review of the technical aspects of RNA FISH (or other RNA imaging techniques), which are critical for the success of an experiment, is still lacking. Hence, the information provided by this paper may help a wide range of investigators using RNA-FISH in their research. I think the manuscript is suitable for PeerJ. There are, though, some points that I would like the authors to address before publication:

(1)	The manuscript lacks a summary or conclusion part. A review article should not merely be a collection of previously published papers. As required by PeerJ, a Literature Review should have a conclusion part identifying unresolved questions/gaps/future directions about the field (https://peerj.com/about/author-instructions/#literature-review-sections). In this part, the authors may summarize their review of the technical aspects of RNA FISH and provide their view on the future development of the FISH technique. This will help the readers quickly understand the main points of the paper.

(2)	The manuscript fails to include the recent development of the RNA-FISH technique. As a promising technique for RNA labeling and imaging, RNA-FISH has been improved in different ways to increase the throughput (e.g., Eng et al., Nature, 2019; Shah et al., Cell, 2018) and signal level (e.g., Marras et al., PNAS, 2019; Choi et al., Development, 2018). These new versions of RNA-FISH may point to the future direction of the field. Without mentioning these recent advances and discussing their technical aspects, the manuscript looks a bit outdated. Hence, the authors should include these new versions of RNA-FISH technique in their paper.

(3)	In some places of the paper, the same content has been reviewed twice, which is not necessary. For example, in “The development of RNA-FISH and smFISH” part, the first paragraph reviews the development of different RNA-FISH methods, while the second paragraph more reviews explicitly the development of different FISH probes. However, since the main difference between various RNA-FISH methods is their probes, the contents of the two paragraphs are highly overlapping. The authors should combine the two paragraphs and review the history of different RNA-FISH methods based on the development of different types of probes.

(4)	In the last paragraph of the “Hybridization” part, where the authors discuss the hybridization conditions, the statement “three parameters regarding hybridization: salt concentration; hybridization temperature; and duration of hybridization” is not complete, as the pH value of the hybridization buffer is also an important parameter to affect the stringency of the hybridization (Choi et al., Development, 2016; Mannheim, Biochemicals Catalog, 1992). The authors should discuss how pH value affects hybridization in their paper.

Minor points:

(5)	Figure caption is missing for Figure 1. Although Figure 1 is a simple figure, it has only been cited once in the main text (line 83) without a detailed description. A caption is therefore required to help explain the whole figure. For example, the term “Chromogenic ISH” only appears in Figure 1, but not in the main text. With a figure caption, it is hard to understand what the term means.

(6)	In lines 83 and 415, “(Figure 1)” and “(Figure 2)” should be “(Fig. 1)” and “(Fig. 2)”, respectively, according to the Instructions for Authors (https://peerj.com/about/author-instructions/#figures).

(7)	In Tables 1 and 3, the temperature of each step is not listed.

(8)	In line 81, the subtitle “the development of RNA-FISH and smFISH” is not correct, as smFISH is a specific type of RNA-FISH. The authors may modify the subtitle to “the development of RNA-FISH”.

(9)	In the last row of table 3, except for the first column, the duration of the final wash is missing.

(10)	 There are some language issues in the manuscript. Here are a few examples. The authors will have to make sure all such errors are corrected.

•	In line 293, “…is used protect RNA-based probes” should be “…is used to protect RNA-based probes”.
•	On the last page, “Table 3. Table 2. Representative panel of fluorescence…” should be “Table 3. Representative panel of fluorescence …”.

References:
Tutucci, Evelina, et al. "Imaging mRNA in vivo, from birth to death." Annual review of biophysics 47 (2018): 85-106.Salvatore A. E. Marrasa,b, Yuri Bushkina,c, and Sanjay Tyagi (2019).

Pichon, Xavier, et al. "A growing toolbox to image gene expression in single cells: sensitive approaches for demanding challenges." Molecular cell 71.3 (2018): 468-480.

Eng, Chee-Huat Linus, et al. "Transcriptome-scale super-resolved imaging in tissues by RNA seqFISH+." Nature 568.7751 (2019): 235.

Shah, Sheel, et al. "Dynamics and spatial genomics of the nascent transcriptome by intron seqFISH." Cell 174.2 (2018): 363-376.

Marras, Salvatore AE, Yuri Bushkin, and Sanjay Tyagi. "High-fidelity amplified FISH for the detection and allelic discrimination of single mRNA molecules." Proceedings of the National Academy of Sciences (2019): 201814463.

Choi, Harry MT, et al. "Mapping a multiplexed zoo of mRNA expression." Development 143.19 (2016): 3632-3637.

Mannheim, Boehringer. "Nucleic Acid Hybridization—General Aspects." Biochemicals Catalog (1992): 14-17.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A TECHNICAL REVIEW AND GUIDE TO RNA FLUORESCENCE IN SITU HYBRIDIZATION
Review round: 1
Reviewer: 3

Output:
Basic reporting: See below.
Experimental design: See below.
Validity of the findings: See below.
Additional comments: In this Review, Young et al. present a brief historical account on the development of the most used RNA-FISH techniques, followed by a description of the most typical protocol steps, with emphasis on the rationale behind the use of the most commonly used reagents and procedures.

A review like this could be of use as an introduction for researchers that are interested in using RNA FISH but do not have prior experience in method development. In that sense, I think this article is worthwhile publishing. Below I list a series of comments for the authors to consider before publishing this piece in PeerJ:

1.	Lines 110-111: “Since these seminal studies, the applications of smFISH have expanded to visualize differential gene expression within heterogeneous cells”. I think that using the term “heterogeneous cells” makes this sentence ambiguous. I think that the authors could change it to “(…) visualize gene expression heterogeneity at the single-cell level”.

2.	Lines 112-113: The authors cite Shalek et al. 2014 when they talk about RNA-FISH multiplexing. But that reference uses mainly single-cell sequencing, using RNA-FISH only as a secondary confirmatory technique, I, therefore, think that this reference can be omitted.

3.	Following comment #2, the authors mention smFISH multiplexing but they do not cite the most up-to-date RNA-FISH multiplexed techniques, such as MERFISH (Chen et al., Science, 2015, Moffitt et al., Science, 2018) or d seqFISH (Lubeck et al. Nat. meth., 2014; Eng et al, Nature, 2019). I think it is fitting to cite such techniques as the final evolution of RNA-FISH.

4.	Lines 122-123: “Both methods had issues producing signal to noise ratios that could allow reliable quantification of transcript expression”. This phrase is vague. What “issues” is the author referring to? This could be solved by stating explicitly the problem of using a single fluorophore per mRNA (or a few, in case of antibodies): that probe binding to mRNA cannot be distinguished from non-specific binding to other cellular structures. smFISH allows obtaining the absolute number of molecules, while previous methods only could measure relative expression levels.

5.	Lines 125-126: The authors mention self-quenching. Where do they get that reference? I know that Raj et al., Nat. meth., 2008 mention self-quenching as a drawback of the method used by Femino et al. and they cite Randolph et al., Nucleic Acids Res., 1997. The authors should cite either of those sources (or any source they used to back up their claim). Also, the authors may want to mention that probably the most important reason this method is not widely adopted is the difficulty to create such probes (As mentioned in Raj et al., Nat. meth., 2008). Self-quenching by itself does not avoid the method from reaching single-molecule resolution (as evidenced by the many works from the Singer lab)

6.	Lines 130-132: The authors found that this approach was more effective in labeling individual mRNA targets compared to traditional probes that spanned the full length of the transcript”. Here again, the language is vague. What do the authors mean by “more effective”? Also, which comparison the authors mean? I do not remember Raj et al. 2008 making such a comparison. Also, please provide a reference when making claims.

7.	Lines 135-137: The authors mention Stellaris as a product that was created based on smFISH technology. I find that only mentioning a single product seems to be unnecessary favoritism. I imagine there are other commercial products that are used for other types of RNA-FISH. If this is the case, It would be better the authors add a table for a series of technologies or remove the reference to commercial products altogether.

8.	Line 270: ”(…) is time well spent”. It seems too colloquial and vague. It would be better to say explicitly what can be improved, for example, “it is necessary to obtain a high signal to noise ratio” or something like that, depending on what the authors want to emphasize.

9.	Line 284: “(…) reduces the annealing temperatures of nucleic acid strains”. That’s confusing. The annealing temperature is an experimental parameter that can be varied. A more appropriate parameter to mention would be the free energy of binding (which depends on the temperature).

10.	Lines 343-345: It is true that the salt concentration is generally kept constant. Temperature is changed often, but also formamide concentration is often changed (see multiple reviews from Raj lab.)

11.	Line 426: I would suggest the authors to add references to works where the authors use the comparison of the foci intensity in samples with or without the transcript of interest (for example, Skinner et al. Nat. Protoc., 2013) or the presence of a single foci intensity peak (Trcek et al., Nat. Protoc., 2012)

12.	Finally, the review would benefit from a “Conclusion” or “Discussion” section. Also, it would also benefit from a section describing, even briefly, the type of analysis researchers perform with the data (particularly for smFISH).


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: A TECHNICAL REVIEW AND GUIDE TO RNA FLUORESCENCE IN SITU HYBRIDIZATION
Review round: 2
Reviewer: 1

Output:
Basic reporting: The authors have addressed my concerns.
Experimental design: The authors have addressed my concerns.
Validity of the findings: The authors have addressed my concerns.
Additional comments: The authors have addressed my concerns.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERY OF POTENTIAL TARGETS OF TRIPTOLIDE THROUGH INVERSE DOCKING IN OVARIAN CANCER CELLS
Review round: 1
Reviewer: 1

Output:
Basic reporting: 1) The English used is very poor, which often hinders the understanding of the manuscript. The manuscript should be reread and corrected by a person skillful in scientific English writing. Past perfect should be overall replaced by present perfect.

2) Reference to manuscript J. Chem. Inf. Model. 2019, 59, 2467−2478 should  be given in the inverse docking section.

3) Figure 1 is of extremely low quality. It should be replaced by one having a better resolution.
Experimental design: The molecular docking protocol is described in an irreprodicible fashion. Use Chem. Res. Toxicol. 2014, 27, 2136−2147 as a template and quote it.
Validity of the findings: No comment.
Additional comments: Chemical structure of triptolide should be provided in a separate figure.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERY OF POTENTIAL TARGETS OF TRIPTOLIDE THROUGH INVERSE DOCKING IN OVARIAN CANCER CELLS
Review round: 1
Reviewer: 2

Output:
Basic reporting: -- The manuscript is full of typos and syntax(even in the affiliation of the authors which should be Pharmacy not Parmacy) , It should be checked by an English native speaker to improve the quality of the used language.
-- Good background and suitable references
-- Article structure should be revised, It will be better to start with the reverse docking and then based on the results from the reverse docking , it could flow to the in vitro testing but not the reverse
-- The results are relevant
Experimental design: -- Original research within the scope of the journal.
The research question is well defined and relevant but the flow of the work is not logic concerning the in silico study, it should be the first step before paying effort and money and it should be used as guide for the subsequent work.
-- Methods are described in details except in the docking study more unwanted facts and details about the principle rather than the method itself were given. more details and references should be given for all the methods used
Validity of the findings: -- Most of the biological results have no statistical analysis and the level of significance and differences from the control are missed.
Additional comments: The manuscript: Discovery of potential targets of triptolide through inverse docking in ovarian cancer cells is interesting however there are many points that required more attention by the authors
among these points:
1- Since the abstract might present the only source of knowledge for many researchers, more rigid and real results should be given instead of vague sentences as it cause upregulation in ..... a measurable value should be given
2- From the title, it seems that inverse docking was used to discover potential targets in ovarian cancer cells then it was used to direct the in vitro study but the design was reversed, it seems that the authors start the in vitro testing then apply the reverse docking (the link between the in silico and in vitro testing was totally missed)
3- The purity of the triptolide, the specs of all the devices used and the version of the used software are not found.
4- It seems very weird to measure the MTT assay at 490 nm instead of 570 nm in that advanced device.
5- the method used for determination of the IC50 should be clearly stated because graphically the values seems away from the reported values in table 1
6- The raw data for determination of the platinum cytotoxicity are missed
7- The structure of triptolide should be drawn


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERY OF POTENTIAL TARGETS OF TRIPTOLIDE THROUGH INVERSE DOCKING IN OVARIAN CANCER CELLS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The manuscript contains some spelling and grammatical a so it is better to be wholly revised by an English language expert.
Experimental design: no comments
Validity of the findings: no comments
Additional comments: Title: Discovery of potential targets of triptolide through inverse docking in ovarian cancer cells

The manuscript is somewhat interesting but certain points should be considered and are written below.
1.	Regarding the introduction section, it should be more comprehensive with more details about the disease, its causes, methods of propylaxis and what previously used to alleviate the disease and what will this drug offer better than what previously reported.
2.	The manuscript contains some spelling and grammatical a so it is better to be wholly revised by an English language expert.
3.	The structure of the compound should be drawn and structural activity relationship should be comprehensively discussed
4.	The references should be carefully checked to be all in the same style comprising the author’s name as some lack end pages.
5.	Most of the figures are unclear


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERY OF POTENTIAL TARGETS OF TRIPTOLIDE THROUGH INVERSE DOCKING IN OVARIAN CANCER CELLS
Review round: 2
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: All the issues raised by this reviewer were successfully addressed. Consequently, the manuscript has been significantly improved and can be in its current version recommended for publication in PeerJ.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DISCOVERY OF POTENTIAL TARGETS OF TRIPTOLIDE THROUGH INVERSE DOCKING IN OVARIAN CANCER CELLS
Review round: 2
Reviewer: 2

Output:
Basic reporting: The English used is still not acceptable
Experimental design: The experimental design has much improved and corrected
Validity of the findings: Interesting findings
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPACT OF BREAST SURGERY ON SURVIVAL OF PATIENTS WITH STAGE IV BREAST CANCER: A SEER POPULATION-BASED PROPENSITY SCORE MATCHING ANALYSIS
Review round: 1
Reviewer: 1

Output:
Basic reporting: This retrospective study analysed impact of breast surgery on outcomes in de novo metastatic breast cancer patients using SEER derived data. The research question is relevant, however, the appropriateness of this research approach with using propensity score matching method and SEER-derived data in this setting is questionable. Nevertheless, numerous similar studies have been performed and published beforehand. The authors are to be congratulated for providing complete raw data. However, the methods could be more precisely described.

There are several major issues that need to be considered before potential publication. Please also refer to the comments made in the peerj-reviewing-41899-v0.pdf.

1.	Language: It is written in professional English that is in parts unclear mostly due to poor syntax, and redundancy, less because of ambiguousness. Please refer to in-line comments which are far from thorough but are meant to emphasize the problem which needs to be tackled.

2.	Title “Impact of surgical treatment on survival for patients with stage IV breast cancer”: Authors should consider using "breast surgery" instead of "surgical treatment", as, for example, metastasectomy is also a type of surgical treatment but is not a subject of this research. This should be corrected throughout the manuscript.

3.	In the Introduction the authors grasp the complexity of the research question well and they do reference the pertinent studies, however, besides minor language slips some other corrections are needed as well:

     a.	Line 50: "…has been transferred…" is an expression used to describe an active transfer, whereas in this case consider something along the line "has disseminated to distant sites" or similar. The authors could as well consider the whole first sentence redundant.

     b.	Line 50: “…to the site away from the breast…” Imprecise. For example, regional nodes are a site away from the breast but N+ disease is not considered stage IV on its own. The authors should recheck and clarify this.

     c.	Line 51: “…5-10% of female…” Considering the references 1-3 provided by the authors, the percentages are 4.4-6% and are valid for USA only, whereas proportion of stage IV breast cancer at diagnosis worldwide ranges from 3 to 25% according to Unger-Saldaña et al (1). The authors should recheck and clarify this.

     d.	Line 54-56: The authors should support this statement with pertinent reference. Or if this is their general observation from their practice, they should make a more general or/and clear statement.

     e.	Line 60: References [5-10]: Two of these studies (10. Lim et al, 5. Rashaan et al) failed to demonstrate a statistically significant positive impact of breast surgery on overall survival in the studied cohort. The positive effect was evident only in subgroups of patients. The authors should either correct the statement or discuss these two studies separately.

4.	Materials and methods:

     a.	The authors used propensity score matching method (PSM). According to Biondi-Zoccai et al (2) the decision between using PSM or standard multivariable analysis is not as straightforward as it seems when it comes to retrospective analysis. One of the key factors in this decision-making process is the number of events per variable, yet nowhere in this article can the number of events be found. As this is crucial for declaring this article statistically sound, I would like to hear from the authors why they chose PSM over, for example, Cox proportional hazard analysis. Furthermore, the number of events should be provided by the authors in their paper.

     b.	The authors should provide the reader with the SEER definition of "de novo metastatic breast cancer". According to SEER the initial stage is defined as metastatic if metastases are diagnosed in the first four months after the diagnosis (3). This definition is crucial for understanding the limitations of retrospective studies assessing the effect of breast surgery on survival in this setting.

     c.	The authors should provide the reader with the patient selection flow-chart describing the patient selection process with number of initially selected patients (all C50 in SEER in 2010-2015) and so on. Furthermore, the authors should provide the reader with the basic information about SEER which also partially describes the studied cohort, e.g. "SEER collects and publishes cancer incidence and survival data from population-based cancer registries covering approximately (28%) of the population of the United States."

     d.	Why did the authors feel the need to dichotomize the continuous variable age? According to Cohen et al (4) this leads to a reduction in statistical power and potentially also to misleading results (5). Could the authors at least categorize the age into 10-year-intervals and redo the analyses? Alternatively, could the authors provide the reader with the reasoning behind dichotomization and the cut-off value used?

     e.	The authors should provide the reader with more information on what data regarding radiation was available to them from the SEER database, but not only in Discussion, but also here in Material and Methods. Line 75: Is this radiation to the primary site? To the regional nodes? To the metastatic site? Before surgery? After surgery? Before systemic therapy? After systemic therapy? This definition is crucial for understanding the limitations of the retrospective studies assessing the effect of breast surgery on survival in this setting.

      f.	Line 75, “chemotherapy”: As is evident from the supplementary material, "no" and "unknown" were considered together in analyses. I am very interested in why did the authors not include the chemotherapy status as "yes", "no", and "unknown" separately in the analysis as they did with other covariates?

     g.	Was the information on targeted treatment available to authors? If so, the authors should include this data in their analyses as non-receipt of targeted treatment in HER-2 positive patients could substantially influence the results. If not, the authors should state this crucial information already in Materials and Methods section, not only in Discussion. This was also one of the limitations in the prospective trial of Badwe et al (6) where only 9 out of 107 HER2 positive patients received anti-HER2 therapy.

     h.	Solitary bone metastasis. The SEER (7, see p.124) does not collect data on the number of metastases, therefore the authors could not have "divided those patients into solitary bone metastasis group and non-solitary bone metastasis". Did the authors mean to say "bone-only metastases" instead? This should be explained, rechecked and corrected accordingly throughout the text. This distinction is vital.

      i.	Line 92-93: “…patients who did not receive any formal resection…” Is it possible that some of the patients received "informal resection"? If so, the authors should elaborate on this. If not, the authors should reword by leaving out "formal".

      j.	Lines 101-103: Was delivery of radiotherapy not included in the analyses?

     k.	Line 101, “…survival risk…”: Further below the authors provide the readers with hazard ratios of death. Could it be more appropriate to use "risk of death" instead.

5.	Results:

     a.	Line 114-115, “…worse histology grade…”: "higher grade" would suffice.

     b.	Line 127: Why is radiotherapy not included in the analysis?

     c.	Line 146-147: Why is radiotherapy not included in the analysis? What about chemotherapy?

     d.	In the titles of the tables the PSM should be worded in full.

     e.	Kaplan-Meier curves: Several authors including Rich et al (8) stressed the importance of showing censored patients as tick marks in survival curves. Authors should consider adding these or adding the “number at risk” below the Y-axis. This should be corrected throughout the figures.

6.	In the Discussion the authors reference the pertinent studies, however, besides several language slips some other corrections are needed as well:

     a.	Line 166: The authors should rephrase this sentence, because metastatic breast cancer is still considered a systemic disease. The authors should rethink about the reference supporting this statement, because in reference 17 the authors performed a similar analysis with similar results in 2007 on 1988–2003 SEER data.

     b.	Line 176: Authors should recheck and correct their incorrect statement. Rashid et al observed in their murine model: "Surgical stress increased tumour burden only transiently without affecting survival. When primary tumour resection decreased overall tumour burden substantially, further growth of metastatic lesions did not increase overall tumour burden compared to observation and survival was improved, which was not the case when resection did not significantly reduce overall tumour burden."

     c.	Line 176-177: The authors should restate this, as this statement is incorrect. First, the primary tumour in situ is inducing immune suppression and not the other way around. Second, neither of these three references shows causal connection between restoring immunocompetency and improved survival. Third, the review by Demicheli et al provides us with a short review of possible mechanisms responsible for suppression of metastatic growth by the presence of primary tumour by circulating angiogenesis inhibitors as well and is as such not the most appropriate reference in this position.

     d.	Line 187, “evidence”: The authors should add "...evidence from registry-based retrospective studies that...", as omission of this fact could mislead the reader.

     e.	Line 233-234: The authors need to add a reference.

      f.	Limitations: For the sake of transparency, the authors should provide the reader with additional limitations:
              i.	Guarantee time bias: those who do not live long enough to undergo surgery are classified to the no-surgery group, leading to erroneously improved prognosis in the surgery group.
             ii.	A possibility of late diagnosis of metastatic disease (see above about the SEER’s 4 months to define definite initial stage) could mean that some of the patients with breast surgery were initially metastases-free or at least had metastases for a shorter period and were therefore asymptomatic at diagnosis. This could lead to a biased attribution of improved prognosis to the surgery group.
             iii.	Problem of residual confounding is not presented in detail as it should be. Perhaps most importantly, data on performance status and comorbidities are vital as these could be the factors that prevailed in the surgery decision-making process and are important determinants of survival.
             iv.	The problem of disease burden: see above - SEER does not provide us with the number of metastases, but only with the information if they are present and if they are present in any of the major sites (bones, lung, liver, brains, distant lymph nodes) at most. This is vital for the interpretation of the results, as this difference could not be discerned from the data provided by SEER, but this is a major factor in breast-surgery decision making and has an impact on survival.

     g.	Supplement material
              i.	Column heading: “ICD-O-3 Hist/behav, malignant （1=IDC，2=ILC，3=其它)”: this should be corrected.

References
1. 	Unger-Saldaña K. Challenges to the early diagnosis and treatment of breast cancer in developing countries. World J Clin Oncol. 2014;5(3):465.
2. 	Biondi-Zoccai G, Romagnoli E, Agostoni P, Capodanno D, Castagno D, D’Ascenzo F, et al. Are propensity scores really superior to standard multivariable analysis? Contemp Clin Trials. 2011 Sep;32(5):731–40.
3. 	Ruhl J, Adamo M, Dickie L. Section V. In: SEER Program Coding and Staging Manual 2016. National Cancer Institute, Bethesda, MD 20850-9765; 2016.
4. 	Jacob Cohen. The Cost of Dichotomization. Appl Psychol Meas. 1983;7(3):249–53.
5. 	Royston P, Altman DG, Sauerbrei W. Dichotomizing continuous predictors in multiple regression: a bad idea. Stat Med. 2006 Jan 15;25(1):127–41.
6. 	Badwe R, Hawaldar R, Nair N, Kaushik R, Parmar V, Siddique S, et al. Locoregional treatment versus no treatment of the primary tumour in metastatic breast cancer: an open-label randomised controlled trial. Lancet Oncol. 2015 Oct;16(13):1380–8.
7. 	NCISeer. SEER Program Coding and Staging Manual 2018 SEER Program Coding and Staging Manual 2018 Acknowledgements ii SEER Program Coding and Staging Manual 2018. 2018;(January).
8. 	Rich JT, Neely JG, Paniello RC, Voelker CCJ, Nussenbaum B, Wang EW. A practical guide to understanding Kaplan-Meier curves. Otolaryngol Neck Surg. 2010 Sep;143(3):331–6.
Experimental design: Please see 1. Basic reporting and attached files.
Validity of the findings: Please see 1. Basic reporting and attached files.
Additional comments: Please see 1. Basic reporting and attached files.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPACT OF BREAST SURGERY ON SURVIVAL OF PATIENTS WITH STAGE IV BREAST CANCER: A SEER POPULATION-BASED PROPENSITY SCORE MATCHING ANALYSIS
Review round: 1
Reviewer: 2

Output:
Basic reporting: a.	The study is written in a clear, unambiguous and professional English.
b.	The authors described the background behind this topic clearly.
c.	The figures and tables are relevant and of high quality.
d.	The study is self-contained with relevant results to the hypothesis.
Experimental design: a.	This study is not original when it comes to Ideas. However, it presented a large number of patients, which strengthen its evidential power.
b.	The research questions were solid, rational and can be answered without a great deal of bias.
c.	The investigatory process and statistical analysis are nearly perfect. Also, the methodology is unambiguously described.
Validity of the findings: a.	Despite limitations implied by retrospective data, the authors presented good evidence about the use of surgery in selected cases of stage IV breast cancer. It can help in directing future studies.
b.	Unfortunately, the evidence coming from this study is not powerful enough to change the standard of care. However, it can provide another solid option to physicians and patients regarding the management.
Additional comments: The well-written manuscript, raw data, statistical analysis done and the presented results indicate how much effort and time the authors have spent in this study.

Although the data is retrospective with several limitations as the authors themselves described, the study added new bricks to the growing evidence in the matter of resecting oligometastatic disease.

Still, the study failed to characterize and examine the effect of different treatment approaches, such as new anti-her2 or endocrine therapies. Moreover, the study failed to identify the timing effect of local treatment and the best modality to deal with it. Unfortunately, there is no head to head comparison between radiotherapy and surgery in this issue, especially for a single metastasis. When it comes to defining patients' groups; Is there a cut-off for the number of metastases, it is three, four, or just one. This study failed to answer all of these questions. We are looking forward to seeing more solid RCTs to answer these questions.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPACT OF BREAST SURGERY ON SURVIVAL OF PATIENTS WITH STAGE IV BREAST CANCER: A SEER POPULATION-BASED PROPENSITY SCORE MATCHING ANALYSIS
Review round: 2
Reviewer: 1

Output:
Basic reporting: I congratulate the authors for implementing the suggestions in their manuscript. I believe the manuscript is now concise and prepared for publishing.

A)	This should take the authors only a few minutes to correct:

1.	At the end of the second paragraph in discussion: »However, other experimental studies indicated that local surgery could substantially reduce overall tumor burden and improved immunologic response to cancer by inducing immune suppression and restoring immunocompetence [22-23].«
a.	Please see the comment in the first review again. This statement is still incorrect. I would suggest writing something like: “However, other experimental studies in the murine model indicated that substantially reducing the overall tumor burden with local surgery leads to improved survival. This could be explained by the primary tumor’s suppression of the immune response and its removal leading to restored immunocompetence.”

2.	Figure 4c: Replace "Solitary bone metastasis" with "bone-only metastases".

3.	Table 4: Replace "Solitary bone metastasis" with "bone-only metastases", in two places.

B)	This might as well be my misunderstanding of the presented data. I only ask the authors to recheck the numbers before publishing to be sure everything is in order.
According to figure 1 there were 13034 cases before propensity score matching and 4538 after (2269 in each of the breast-surgery and no-surgery group). However, the numbers in the presented tables don’t add up.

i.	Table 2. All the numbers in the table should be rechecked.
1.	The n numbers should be the same for BCSS and OS as these were analysed in the same group of patients.
2.	The numbers do not add up. For example, the sum of the two highlighted numbers (see the pdf attachment) is 7433 below BCSS and 7949 below OS, but there should be 13034 cases according to figure 1.

ii.	Table 3. All the numbers in the table should be rechecked.
1.	The n numbers should be the same for BCSS and OS as these were analysed in the same group of patients.
2.	The numbers do not add up. For example, the sum of the two highlighted numbers (see the pdf attachment) is 2438 below BCSS and 2603 below OS, but there should be 4538 cases according to figure 1.

iii.	Table 5. All the numbers in the table should be rechecked. According to the manuscript there should be 2269 patients in the breast-surgery group after propensity score matching.
1.	The n numbers should be the same for BCSS and OS as these were analysed in the same group of patients.
2.	These numbers do not add up. For example, the sum of the two highlighted numbers (see the pdf attachment) is 1086 below BCSS and 1171 below OS, but there should be 2269 cases.
Experimental design: Please see 1. Basic reporting.
Validity of the findings: Please see 1. Basic reporting.
Additional comments: Please see 1. Basic reporting.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: IMPACT OF BREAST SURGERY ON SURVIVAL OF PATIENTS WITH STAGE IV BREAST CANCER: A SEER POPULATION-BASED PROPENSITY SCORE MATCHING ANALYSIS
Review round: 3
Reviewer: 1

Output:
Basic reporting: no comment
Experimental design: no comment
Validity of the findings: no comment
Additional comments: Manuscript title: Impact of surgical treatment on survival for patients with stage IV breast cancer: a population-based propensity score matching analysis (#41899)

Third review

This retrospective study analysed impact of breast surgery on outcomes in de novo metastatic breast cancer patients using SEER derived data. The authors implemented the suggested corrections of the previous reviews well. The research question is relevant, the methodology is sound, and the results are well interpreted. I congratulate the authors for their manuscript, and I recommend this paper for publishing.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTIVE VALUE OF CONUT SCORE COMBINED WITH SERUM CA199 LEVELS IN POSTOPERATIVE SURVIVAL OF PATIENTS WITH PANCREATIC DUCTAL ADENOCARCINOMA: A RETROSPECTIVE STUDY
Review round: 1
Reviewer: 1

Output:
Basic reporting: The conceptual framework is incomplete and with some references that can be improved (15, 16, 17, 18)

Some bibliographical references are incomplete, for example, 24,35, 39.
In addition, many are old should improve this aspect.
Experimental design: The authors must describe in the methodology that they consider CONUT high and Low, as they established.
Similarly, the grouping of CA199 must be in the methodology and not in the results.
Validity of the findings: In the database in survivals there are several cases of 1 month (OS and RFS), but excluded who died within 30 days of operation. Explain this.
Additional comments: I encourage you to continue contributing to scientific knowledge,  thank you for giving me the opportunity to review this manuscript.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTIVE VALUE OF CONUT SCORE COMBINED WITH SERUM CA199 LEVELS IN POSTOPERATIVE SURVIVAL OF PATIENTS WITH PANCREATIC DUCTAL ADENOCARCINOMA: A RETROSPECTIVE STUDY
Review round: 1
Reviewer: 2

Output:
Basic reporting: In this article the authors describe the development of a novel scoring metric for pancreatic ductal adenocarcinoma that concatenates the predictive scoring power of controlling nutritional status (CONUT) score and serum carbohydrate antigen 199 (CA199) levels. In general the use of English is acceptable. Data has been shared and references are, on the whole, appropriate.But there are a couple of non-conventional uses that should be addressed:
Line 67 – Use of the word occult is not appropriate. Perhaps change to ‘unpredictable’.
Line 84 – State the CONUT score is a new scoring method. The initial paper (Ref 22) was published in 2005, almost 15 years ago. This is hardly a most recent and up to date method and the text should be addressed as such.
Table 2 - The legend needs to include full details on all statistical data displayed
Experimental design: The experimental design is well defined and outlines the research question clearly. All human particiapnt ethics are in place and there seems to be enough detail to allow replication within the text.
The statitstical methodology also seem to be approporiate. Use of the Chi-Squared test on categorical variables and ROC curve to test the predictability of the study are also suitable.
My only concern, and this needs to be justified in any response - is why was a p value of 0.05 chosen? A level of statistical significance should be chosen by the study design and not by arbitrary convention, Also, there is no mention of power within the findings. How do we know a sample size of 294 is large enough?
Validity of the findings: Leading on from the last question, levels of CA-199 have been shown to more associated with females (in an experiment with double the number of subjects):

https://bmccancer.biomedcentral.com/articles/10.1186/s12885-017-3738-y

Yet the model documented in this article does not agree. Can the authors please comment on this. Is it a facet of the difference in statistical design or a power issue?

Also, the authors need to discuss fuller the significance of the ROC curve analysis. Yes, the combined score is larger than the individual scores but at only 0.7 Swets and others have offered benchmarks for gauging AUCs, suggesting that values ≥0.9 are “excellent,” ≥0.80 “good,” ≥0.70 “fair,” and <0.70 “poor.”

Swets J A, Dawes R M, Monahan J. Psychological science can improve diagnostic decisions. Psychological Science in the Public Interest. 2000;1:1–26.

All ROC scores are pretty poor and need to be discussed further
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: PREDICTIVE VALUE OF CONUT SCORE COMBINED WITH SERUM CA199 LEVELS IN POSTOPERATIVE SURVIVAL OF PATIENTS WITH PANCREATIC DUCTAL ADENOCARCINOMA: A RETROSPECTIVE STUDY
Review round: 2
Reviewer: 1

Output:
Basic reporting: Correct
Experimental design: Correct
Validity of the findings: Correct
Additional comments: The authors have made the requested changes.

Congratulations


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TRANSCRIPTION LEVELS AND PROGNOSTIC SIGNIFICANCE OF THE NFI FAMILY MEMBERS IN HUMAN CANCERS
Review round: 1
Reviewer: 1

Output:
Basic reporting: There were no novel findings in the manuscript compared to the previous reports.
Experimental design: All results of this article is from Oncomine, TCGA, KM plotter and SurvExpress databases. There is no any further experimental evidence for the transcription level change in any cancer type. That could not help to verify the correlation of NFI family in human cancer.
Validity of the findings: The finding is without any novelty and lack of enough evidences to  support this opinion.
Additional comments: Although database is a kind of strategy to find out the correlation of target gene and human cancer, the experimental data is more important to link the finding of database. Nuclear factor I transcription factors was already found the relationship of human cancer. The article did not support any more novel findings.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TRANSCRIPTION LEVELS AND PROGNOSTIC SIGNIFICANCE OF THE NFI FAMILY MEMBERS IN HUMAN CANCERS
Review round: 1
Reviewer: 2

Output:
Basic reporting: Clear English is used throughout.  The article is very well written.
Good literature references and sufficient field background/context are provided.
The authors used a professional article structure, figures and tables and the raw data is shared through supplemental tables and figures.
The article is self-contained with relevant results to the hypothesis that NFI transcription factors may be important in a variety if cancers.
Experimental design: The authors clearly defined the research question, which is relevant and meaningful. The knowledge gap being investigated has been well identified, and statements have been made as to how the study contributes to filling that gap.  While only a review and mining of the current literature and databases, the authors make a good effort to point out the significance of their findings.  The techniques used by the authors appear valid and consistently applied.  They are described in good detail in the methods section.
Validity of the findings: The conclusions of the authors appear to be clearly consistent with the data presented.  All of the data are from pre-existing databases and is freely available.  The statistical methods used appear sound.  The conclusions are clear though somewhat vague.  No obvious speculation is present.
Additional comments: No additional comments


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TRANSCRIPTION LEVELS AND PROGNOSTIC SIGNIFICANCE OF THE NFI FAMILY MEMBERS IN HUMAN CANCERS
Review round: 1
Reviewer: 3

Output:
Basic reporting: The manuscript is clear and english was easy to follow.
Literature references are scarce, at the introduccion, and can be improved but not a problem for a bioinformatic analysis.
The structure of the articles can be largely improved by proving deeper analyses.
Experimental design: The experimental design is poor since does not consider why the expression changes across human cancer or to which other variables are the expression correlated. This shows that the experimental design is quite superfluous.
Research question is well defined but not so relevant if not appropriately analyzed.
The investigation was not rigorous since statistical thinking is lacking. Suggestion are recommended in general comments.
Validity of the findings: Not novel (checking expression across cancers) and low impact if not complemented by deeper analyses.
Statistically, analyses does not sound.
Conclusion is meaningless if not deeper analysis is provided.
Additional comments: Review for the manuscript #38719 “Transcription levels and prognostic significance of the NFI family members in human cancers” received from PeerJ. In this work, the authors carry out a bioinformatic gene expression analysis of the family of 4 transcription factors in human cancers. For this, they used Oncomine, UCSC Xena, and SurvExpress. First, the authors check the differential expression of the genes across cancers to show that in about 1/5 of the analyses, these genes are altered. Then, they describe more details in some cancer types including
survival analyses related to the expression of these genes. The authors conclude that the NFI worth to be further studied. The manuscript is well written and easy to follow.

Although the analysis may have some value, I believe it lacks statistical rigor, deeper analysis, and predictions or plausible hypotheses. The following comments show details.

-	The observations in figure 1 could be by random chance. If this is rando, why readers should care about NFI? So, what is the probability of observing differences as those shown in NFI gene family? The authors need to take several families of proteins and do the same analysis in order to show that the expected number of differences is higher in NFI than random chance.
-	Survival analysis also needs to be done for other protein families in order to show whether survival is more affected by NFI expression than random proteins.
-	The authors describe in text sections the difference in expression per cancer type. However, deeper analysis of why the expression seems to be altered is needed (otherwise is just a mere description of what is observed in Figure 1). Are the changes in expression related to copy number alterations? To mutations in specific genes? To subtypes within the same study? Correlated to other proteins? Associated to changes in methylation?
-	Do the analysis encourage plausible biomedical hypotheses? This needs to be discussed.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TRYPOPHOBIA AS AN URBANIZED EMOTION: COMPARATIVE RESEARCH IN ETHNIC MINORITY REGIONS OF CHINA
Review round: 1
Reviewer: 1

Output:
Basic reporting: Trypophobia is not simply due to the internet because many of the individuals with trypophobia I have talked to experienced trypophobia long before the internet was born. The internet has, however, been responsible for bringing isolated sufferers together.
Experimental design: Needs further work.
The report addresses a sensible question and the results are interesting. The statistical power in the studies is low, however. I would like to see a larger sample of individuals, and more than one sample of locations in order to be sure of the estimates of the prevalence of trypophobia in city versus country dwellers. One needs to be able to estimate the within-category variability.
Validity of the findings: See below
Additional comments: Provocative visual patterns are prevalent in the modern urban environment, see Wilkins, A.J. Penacchio, O. and Leonards, U. (2018). The built environment and its patterns: a view from the vision sciences. SDAR Journal of Sustainable Design and Applied Research, 6, (1) 41-48. These patterns tend not to occur in nature. They can induce visual distortions, headaches and seizures, a phenomenon known as visual stress. Visual stress is a component of trypophobia, at least in so far as a correlation between visual discomfort and trypophobia has been found, see Imaizumi, S., Furuno, M., Hibino, H., & Koyama, S. (2016). Trypophobia is predicted by disgust sensitivity, empathic traits, and visual discomfort. SpringerPlus, 5(1), 1449. doi: 10.1186/s40064-016-3149-6 So one possible argument the authors might wish to make is that the modern urban environment has increased the levels of visual stress generally, and that trypophobia is one result of this increase.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TRYPOPHOBIA AS AN URBANIZED EMOTION: COMPARATIVE RESEARCH IN ETHNIC MINORITY REGIONS OF CHINA
Review round: 1
Reviewer: 2

Output:
Basic reporting: Not professional English.

Good background.

Results fine.
Experimental design: All fine.
Validity of the findings: All fine.
Additional comments: In this paper the authors report results from a single study in which sensitivity to trypophobia stimuli was examined in people who live in an urban environment and those living in a less-urban environment. Participants were shown a series of trypophobic images (and controls) and asked to rate each for discomfort. The central results showed that both populations showed trypophobia but the size of this effect was greater in the urbanised population. No such difference occurred for the control images. The authors thus conclude that urbanisation plays a contributing role in trypophobia.

Evaluation.

I like the work and do think it will make a nice contribution to the journal. I only have a few points.

1)	I don’t think there is a confound in their experiment but the authors constantly suggest that such a confound exists. They state that they are comparing an urbanised population with  ethnic minorities  (then conclude that urbanisation is important), but this makes it sound like we can’t know whether the effect they observe is due to urbanisation or how prevalent the ethnic group is. In other words, it should be either ethnic minority versus ethnic majority or urban versus non-urban, not ethnic minority versus urban.

2)	The authors give a very weak explanation, concerning clothes, as to why older people showed less trypophobia. The reason that older people are less trypophobic is simply that many phobias are known to reduce as age increases (e.g., Fredrikson, et al. 1996). I would definitely drop the clothes theory, which is also too anecdotal.

3)	The authors also give a very weak explanation as to why interest and knowledge in trypophobia only occurred in the past few years, including ‘bad luck’, ‘researchers negligence’ and an urbanisation-based account. These are all a bit bizarre and need to be dropped (definitely the ‘negligence’ one; perhaps lost in translation but ‘negligence’ is when someone does something very seriously bad). The truth is that the internet has helped to publicise it; it’s a good meme. Also, when I gave the first presentation on the phenomenon, in 2012, a journalist from the Washington Post was present and wanted to run a story on it. This then helped to publicise the paper we published in Psych Science. Its as simple as that.

4)	The English needs tidying up. E.g., Line 42. “This phenomenon is called as trypophobia”

5)	Line 74. The authors state that “ there are numerous cross-cultural studies on emotions showing cultural difference in emotional processing”. I think a couple of sentences are needed here stating which emotions are particularly different.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TRYPOPHOBIA AS AN URBANIZED EMOTION: COMPARATIVE RESEARCH IN ETHNIC MINORITY REGIONS OF CHINA
Review round: 2
Reviewer: 1

Output:
Basic reporting: This is a second review
Experimental design: The authors appear to have misunderstood my objection to their sample size and distribution, which has nothing to do with power analysis, p-hacking or other considerations.  In order to make a claim that the differences between samples are due to their being "urban" or "rural" one needs more than one sample of individuals from each category, particularly when the individuals are from single locations.  These locations may simply not represent the category assigned to them.
Validity of the findings: This is a second review
Additional comments: Personally, I do not feel that the paper is ready for publication. The sample size is small and the samples are not necessarily representative.  More work needs to be done.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TRYPOPHOBIA AS AN URBANIZED EMOTION: COMPARATIVE RESEARCH IN ETHNIC MINORITY REGIONS OF CHINA
Review round: 2
Reviewer: 2

Output:
Basic reporting: See below
Experimental design: See below
Validity of the findings: See below
Additional comments: This is a revision to a ms I reviewed previously which I was largely happy with. The authors have made a good job of the revision. I recommend publication as is.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TRYPOPHOBIA AS AN URBANIZED EMOTION: COMPARATIVE RESEARCH IN ETHNIC MINORITY REGIONS OF CHINA
Review round: 3
Reviewer: 1

Output:
Basic reporting: see below
Experimental design: see below
Validity of the findings: see below
Additional comments: I am delighted that the authors have collected additional data so that they can separate the categories of urban and rural from the particular locations they used. This has greatly improved the paper, and it is now nearly publishable.  I would however, like to suggest one additional analysis. I think there are now several groups, depending on location.  It would be important to analyse the data using an analysis of variance that grouped the individuals by the several locations, so that the reader can appreciate that the categories of urban and rural transcend the locations from which they are derived. This will involve small changes to the analyses they have undertaken and the measures of error involved. Is the variance due to category (urban/rural) greater than the variance due to location?


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: TRYPOPHOBIA AS AN URBANIZED EMOTION: COMPARATIVE RESEARCH IN ETHNIC MINORITY REGIONS OF CHINA
Review round: 4
Reviewer: 1

Output:
Basic reporting: OK
Experimental design: OK
Validity of the findings: Improved
Additional comments: The authors have greatly improved their paper and it will now make a valuable contribution to the literature.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OVEREXPRESSION OF MYO1E PROMOTES ALBUMIN ENDOCYTOSIS BY MOUSE GLOMERULAR PODOCYTES MEDIATED BY DYNAMIN
Review round: 1
Reviewer: 1

Output:
Basic reporting: 1. 	The English language should be improved to ensure that an international audience can clearly understand your text. Examples include the title and research question as posed in the abstract (.. .whether Myo1e albumin endocytosis in mouse glomerular podocyte enocytic albumin and it possible mechanism). Other examples include lines 78, 82, 85, and 93.
Also, I would advise to refrain from using sentences such as ‘and so on’ (line 55)‘and other functions’(line 86)
I would advise using a professional editing service to ensure a thorough improvement, which is needed.

2. The text in the introduction (especially lines 54 to 68) is very difficult to comprehend and contains many references that don’t seem immediately relevant to fill a knowledge gap for your particular research question. It isn’t necessary to provide a comprehensive review of all the functions podocytes have in this regard, as this would become quite extensive. However, some key elements seem to be missing now. Stating that the role of podocytes in proteinuria can be attributed to ‘nephrin and podocin and so on’ does not provide enough information. Also, there is extensive literature on nephrin and podocin and proteinuria (NEJM 2006; 354 Tryggvason et al for example). Placing only self references here does not seem thorough enough.
Please consider rewriting this section of the introduction. Consider including reviews on the role of podocytes in the development of proteinuria, endocytosis in podocytes in proteinuria/maintenance of the GFB and podocyte apoptosis in proteinuric diseases. For example, see Nagata’s Kidney International paper (PMID 27165817)

3. In the introduction (line 49), the author states that damage to podocytes inevitably leads to albuminuria. This is not necessarily true, as for example increased reabsorption by the PTC can take place. Also, not all damage to the GFB is irreversible. An increase in glomerular permeability can occur without leading to proteinuria.

4.	In line 240 the author claims that little research has been done on podocyte molecular changes. The paper’s own reference list contradicts this statement. Please revise.
Experimental design: 1.	Please clearly state the research question in the introduction (and abstract). The current research question formulated in the abstract reads: ‘we aimed to analyze whether Myo1e albumin endocytosis in mouse glomerular podocyte endocytic albumin and its possible mechanism.’
For example: we aimed to analyze whether albumin endocytosis by mouse glomerular podocytes is dependent on Myo1e expression. Also, we aimed to elucidate whether the underlying mechanism is mediated by Dynamin.
2. To allow full comprehension of your data, please include mean values, SD and the p value (or CI) when reporting your results.
3.	Validation and quantification of your Myo1e overexpression and shMyo1e experimental models is necessary for proper interpretation of results. Please add these experiments (Myo1e expression levels) to the current paper.
4.	Are relative expression levels as shown in Figure 5 corrected for GAPDH expression? Is yes, please state this in your methods.
5.	Figure 7B shows that the effect of Dynasore on endocytosis is much greater than the effect of Myo1e overexpression. Please provide statistical comparison between the empty vector+dynasore and Myo1e overexpression+dynasore groups.
6.	Validation of the effect of Dynosore on both Myo1e and Dynamin expression should be added to allow for a proper interpretation of results (dynamin expression levels). Please add this to the current paper.
7.	Figure 3: please provide images in a higher magnification. The current image is hard to appreciate. Please highlight the morphological changes you want the reader to pay attention to (larger somas, foot process effacement, dendritic bifurcation)
Validity of the findings: 1.	The relative increase in BSA-FITC fluorescence in Myo1e overexpression and decrease in BSA-FITC fluorescence in Myo1e knockdown is significant, but small. Please elaborate on whether you expect this increase to be clinically relevant. Your results seem to show that Myo1e is involved in albumin endocytosis, but that it does not necessarily play a large or essential role.
2.	Additional experiments are required to substantiate the hypothesis that the effect of Myo1e is mediated by dynamin.
Why did you choose to use dynasore and not a direct dynamin knockdown or knockout? As stated by Preta et al. in Cell Commun Signal 2015; 13: 24: Dynasore provides rapid and reversible inhibition of dynamin-dependent endocytosis, which is effective in cells from several species. However, in addition to inhibition of the GTPase of dynamin, dynasore has wider effects on cellular cholesterol, lipid rafts, and actin.
3.	Please elaborate as to why the observed increase in apoptosis in podocytes with Myo1e overexpression occurs and how this would lead to amelioration of podocyte injury (line 293). This statement seems contradictory.
Additional comments: In 'overexpressoin of Myo1e promotes albumin endocytosis in mouse glomerular podocyte endocytic albumin mediated by Dynamin’, Shen et al. Show that Myo1e plays a role in albumin endocytosis by podocytes, possibly mediated by Dynamin in a series of in vitro experiments.
The content of this paper shows merit and seems scientifically valid. However, there are some areas where the article could be improved. The major points of concern are stated above. The most serious points of concern are those mentioned under basic reporting point 1, and all points stated under validity of the findings.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OVEREXPRESSION OF MYO1E PROMOTES ALBUMIN ENDOCYTOSIS BY MOUSE GLOMERULAR PODOCYTES MEDIATED BY DYNAMIN
Review round: 1
Reviewer: 2

Output:
Basic reporting: .
Experimental design: .
Validity of the findings: .
Additional comments: The authors investigated the role of Myo1e in albumin endocytosis in the mouse glomerular podocyte culture cells.

Comments
1.	FITC-albumin is easy to dissociate and free FITC will affect the results (Medical Molecular Morphology 2008; 41 (2) 92-98). Have you check how many % of free FITC exist in each concentration of FITC-albumin solution? In Figure 1, if scales of Y-axis were same in each FITC-albumin concentration, total amount of FITC-albumin endocytosis may be not so increased and may become a plateau in 1mg/ml. To confirm the real endocytosis of FITC-albumin, the fluorescent microscopic pictures of FITC-albumin in MPC5 cell culture should be demonstrated in each concentration.
2.	In Figure 2, as the scales of Y-axis are different, it is difficult to see the absolute increase of endocytosis. Thus, increased endocytosis of FITC-albumin should be confirmed by immunofluorescence microscopy of FITC-albumin in Myo1e overexpressed culture cells. Figure 3 is difficult to see the morphological changes without some staining. It is much better to show the immunofluorescence microscopy to show the change in endocytosis vesicles.
3.	In Figure 5B, C there are no SD bars, how many experiments were repeated?
4.	Dynamin-dependent podocyte endocytosis is one of the pathways of albumin endocytosis. The other possibilities of podocyte albumin transport should be discussed; FcRn-mediated albumin transcytosis (Kidney Int 2011; 80: 1328–1338), clathrin-mediated endocytosis (JCI 2012), caveolin-mediated endocytosis (Am J Physiol Renal Physiol 2014;306:F941-951), fluid phase-endocytosis (2015) and dynein-microtubule related vesicle transport (Med Mol Morphol. 2017; 50(2):86-93), and so on.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: OVEREXPRESSION OF MYO1E PROMOTES ALBUMIN ENDOCYTOSIS BY MOUSE GLOMERULAR PODOCYTES MEDIATED BY DYNAMIN
Review round: 2
Reviewer: 1

Output:
Basic reporting: see  below
Experimental design: see  below
Validity of the findings: see  below
Additional comments: The authors responded well to all my comments, so I agree to accept the manuscript.

There is a typo: Line 19, 26 and 30, Myole should be Myo1e.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DETECTION AND VISUALIZATION OF ABNORMALITY IN CHEST RADIOGRAPHS USING MODALITY-SPECIFIC CONVOLUTIONAL NEURAL NETWORK ENSEMBLES
Review round: 1
Reviewer: 1

Output:
Basic reporting: Although the complexity of the content to solve a hard problem, the manuscript is clearly written in a professional, unambiguous language. It is well structured and easy to read.

The literature employed is pertinent and actual. But is suggested to authors include and consider some current works in this research:
     -	Efficient Deep Network Architectures for Fast Chest X-Ray Tuberculosis Screening and Visualization (https://doi.org/10.1038/s41598-019-42557-4);
     -	Deep learning in chest radiography: Detection of findings and presence of change (https://doi.org/10.1371/journal.pone.0204155);
     -	Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists (https://doi.org/10.1371/journal.pmed.1002686);
     -	Computer-aided detection in chest radiography based on artificial intelligence: a survey (https://doi.org/10.1186/s12938-018-0544-y).

The figures are relevant and present good quality. It is suggested a better explanation of figure 4, to show the source from base models with ImageNet; figure 6, for meta-learner should be provided more details; figure 9, what means the Thr: 0.1? And in figures 10 and 11, should be explained the impact of a satisfactory precision penalizing the recall metric. Figure 5 could be removed.
Experimental design: It is perceived that the core of work is the modality-specific learning. Like observed, that is a refined training under the classical CNN based on ImageNet (lines 268-9 and 449), with a wide CXR dataset (contains 18 different abnormalities) and refined again with a specific dataset (RNSA – specific for pneumonia). This process was namely of coarse-to-fine learning.

I consider that the authors use another nomenclature, modality-specific learning, to refer to the classical and commonly used techniques of transfer learning. If that is different, I suggest the authors differentiate it from transfer learning. If the same, the authors should adopt the term transfer learning to refer to this approach.

Attention for lines 174-6, for improving the explanation, it is understood that modality-specific is applied under the CheXpert dataset, but the successive repurpose for detecting abnormality in the RSNA dataset is a kind of modality-specific (or transfer learning) under a specific abnormality (pneumonia). The question about localizing regions is a function of visualizing techniques and doesn’t be the responsibility of methods for detecting abnormalities.

Contrarily, like exposed in line 180, the work doesn’t propose the visualization techniques. The visualization technique was proposed in another study (https://doi.org/10.3390/diagnostics9020038) from the same authors.

Opposed with the description in lines 182-5, it is not the first time that this kind of approach was realized (please see https://doi.org/10.1038/s41598-019-42557-4). This contribution should be reconsidered.

In relation of the main objective, in identify abnormalities or not in CXR, is perceived the use of widely imbalanced dataset (~1:10 for CheXpert and 1:3 for RSNA), but the authors don’t mention this question and its impact in the evaluation.

Sometimes is confused and hard to understand the work, that tries to aggregate two complex tasks, detecting and visualizing (like a way of interpreting). Considering the ambitious task to solve both problems conducted the work to lose quality. It is a little confusing, the proposal for detecting under several abnormalities from a dataset and just one for another dataset, and the visualization focused under the pneumonia dataset. The authors should reconsider what is the real contribution, delimit the scope and focus the work in your main question.

The authors should specify the difference of the dropout U-Net and the "all dropout" (like specified in Fully Convolutional Architectures for Multi-Class Segmentation in Chest Radiographs - arXiv:1701.08816). Should be explained in more detail why this dropout ratio in 0.2 impacts more than the original research that specifies this ratio in 0.1?

What means the Custom model (table 4)? Is it the same of the Baseline (tables 2 and 3)? And Pretrained CNNs are based too on the weights from ImageNet? This confused question should be better explained in the manuscript.

Why use CRM as a unique way of model visualization?
Why not consider an evaluation with other techniques, like Saliency Maps ((arXiv:1312.6034v2) widely mentioned in the current literature). For example, the recent work from Scientific Reports (https://doi.org/10.1038/s41598-019-42557-4) reports the success of this method for visualizing and localizing Tuberculosis in CXR.

In ensemble CRM is not clear how the CNN models complement each other if there is some disagreement with the different models? How the aggregation of this supposed discordance of the individual architectures to compose a final suggestion works? For example, if there is a more precise CNN model, why I have to believe in a CNN model that disagrees with this decision? It is suggested to authors provide more elements and better explanations to justify this question.
Validity of the findings: It is perceived the effort of authors in provided an extensive and challenging task, but some questions should be clarified and better detailed to highlight the real contribution of this work in function of its focus.

In a general view, is unclear the real contribution and innovation of the work. The contribution of the weighted averaging ensemble comes from (https://doi.org/10.1109/embc.2019.8856715) not cited; the visualization and localizing come from (https://doi.org/10.3390/diagnostics9020038). Then, the core of the work is the modality-specific learning, that I consider the classical use of transfer learning (with the use of baseline and pre-trained CNNs models under ImageNet) with another name.

As pointed by authors there are at least 18 abnormalities in CheXpert. RSNA treats only pneumonia, 1/18 abnormalities of CheXpert. The authors don’t mention anything about that, e.g., the diagnosis of pneumonia is easily confused with tuberculosis. And not is clear, what was the real influence of RSNA dataset, with a specific abnormality, when aggregated to fine learning. Like exposed in lines 317-20, the best ensemble was the weighted averaging, based on RSNA. It is suspect that the work could be a bias to pneumonia against a general CXR identification.

The description of CheXpert should specify how many instances there are for each abnormality.

In table 3, how exactly are determined the best layers? The authors should answer this question in more details

Like exposed, lines 446-50, with the explanation of table 4. Not is clear the evaluation process. Were all the evaluations, in terms of detection of abnormalities or not, done and reported considering the reserved test data from both datasets?

The problem that difficult a better understanding is the use of 2 different datasets for predicting abnormalities. It is recommended that the authors supply the results for each individual dataset. And for CheXpert dataset shows the results for each abnormality.

It is not clear the methodological process applied for evaluating the ROI of abnormal images (RSNA pneumonia). The figures 10 and 12 demonstrated didactics aspects, but should be specified if this evaluation was done under the 17833 abnormal images from the dataset. And what was the success rate obtained from this set of images?

The use and efficiency of metrics IoU and mAP don't are clear. There aren't academic works that reported the use of them. I consider it very dangerous to rely on these metrics. The authors should justify its adoption in better detail. Besides that, the authors don’t explain the expected values for these metrics (what is the range?) and what is the impact of the difference presented between tables 6 and 7.

The IoU rate, presented in figures 9 and 12, doesn't correspond with the rates presented in tables 6 and 7. What justifies these differences? Why just this metric is presented, and mAP?

I believe the authors could improve the conclusion; the work seems a lot of questions to evolve. However, the authors don’t mention any future work.
Additional comments: General typos:
- line 154: Huang G in the citation in place of just Huang;
- line 255: CheXpert in place of Chexpert;
- line 445: grid in place of gird;
- Define a pattern for IoU (or IOU) in text and in figures 10 and 11.


Instruction: You are one of the reviewers, your task is to write a review for the article. You will be given the title of the article, the number of the round in which the article is located, and your order among the reviewers.

Input:
Title: DETECTION AND VISUALIZATION OF ABNORMALITY IN CHEST RADIOGRAPHS USING MODALITY-SPECIFIC CONVOLUTIONAL NEURAL NETWORK ENSEMBLES
Review round: 1
Reviewer: 2

Output:
Basic reporting: The authors describe a multistep method to train an ensemble of CNNs to classify chest x-rays as normal/abnormal and identify the region of abnormality.

They train a U-Net to generate lung masks (what role did these play?).

They train a variety of different CNN architectures to a training partition of the CheXpert dataset and report 'coarse model' (i.e. single CNN architecture) performance on this dataset.

These base learners are then used as the pre-trained models in a second step to fine-tune them again on the RSNA pneumonia dataset. After this process, certain near-final convolutional layers  from these base learners are then chosen (through a process that is not clear to me - Figure 4 / text do not provide sufficient detail - line 294 - “We instantiated the convolutional part of the coarse models and extracted the features from different intermediate layers with an aim to identify the optimal layer for feature extraction toward the current task.”).

These extracted layers were then concatenated and another stacked model was trained for classification on RSNA CXR data. Additional hand-constructed ensembles of the model predictions were also created.

Ensemble Class-selected Relevance Mapping is used to demonstrate how ensembling can be used to improve various accuracy figures of abnormality detection as measured by IoU/mAP.

The authors report their process improves various normal/abnormal classification metrics (AUC, etc) and the accuracy per IoU/mAP of regions of abnormality. They conclude that their multiple-step training process and ensembling led to improved performance.

The literature review is good and much relevant work is discussed. The authors performed extensive experiments and report their results completely, although I believe that important experimental details are not fully explained and there are key parts of the analysis that are unclear to me (see below).
Experimental design: I summarized my understanding of the approach above.

There is room to explore how ensembling can improve predictions in medical imaging, and there has been fairly limited exploration of this so far, so the topic is important. My questions about the experimental design follow:

1. In the first step to train coarse learners - are you training to all CheXpert classes or to normal/abnormal target and then inferring abnormality? This should be described.

2. In the second step to train fine-tuned learners - normal/abnormal here just means pneumonia/no pneumonia, since this is how the data is labeled? This should be described.

3. It is not obvious to me that training on the second dataset improved performance. In Table 4, AUCs on coarse models are presumably evaluated on CheXpert, and AUCs on fine models are evaluated on RSNA data? In which case they're not comparable. If they're both evaluated on RSNA data, this should be described. Even then, it wouldn't be surprising that this improved performance - more data, and specific to the dataset. The more interesting question would be if performance on the original CheXpert dataset was better or worse after seeing data from a different modality.

4. In Table 5, ensembling performance is reported, presumably on RSNA test data (needs to be stated). A statistical test should be included to demonstrate that the AUC improvement to 0.97 is actually significant over the fine-tuned models - it's not obvious to me that it is.

5. In the stacking part of the ensembling experiment - were these combinations also trained on the RSNA train data? It is important to make clear they were not trained on test data. It is somewhat surprising that stacking classification performance was inferior to simple averaging - why do the authors believe this was?

6. "Table 3 lists the best-performing empirically determined model layers. The performance  metrics achieved through extracting the features from the optimal layers of the different coarse CNN models toward classifying normal and abnormal CXRs are shown in Table 4." --> how were these layers specifically chosen? The text is not clear on this: "We instantiated the convolutional part of the coarse models and extracted the features from different intermediate layers with an aim to identify the optimal layer for feature extraction toward the current task." This needs to be clearly and fully described - Figure 4 is far too superficial given the complexity of the experiments. Why was it necessary to extract the optimal layer for classification if a classification model was trained?

7. Table 7- how are these region of interest ensembles created? The text says they are generated from the top performing "CNN models that are selected based on the IoU and mAP scores as shown in Table 5" - but these scores / models are not identified in Table 5, which discusses ensemble classification performance. Were they constructed only using RSNA training data, or were you allowed to optimize to test data in constructing these models? Obviously there will be a big bias towards improved results if you were able to use test data in constructing them. Improved explanation of how these ensembles were created is needed, and provided test data was not used in their construction, statistical testing should be included to demonstrate that the performance improvement was significant (or to conclude that it wasn't, if not).

8. “We removed the mapping score value below 10% of the maximum score in each CRM to minimize a possible noisy influence of a very low mapping score during the ensemble process.” - I do not understand this, can the authors explain?

9. What role did the lung masks serve in this work? Were all the images cropped out to exclude what wasn't in the lung mask in training?

10. Mention should be made of what dataset is being used for evaluation in all tables (CheXpert test, RSNA test, etc.)
Validity of the findings: 11. Overall - I appreciate the extensive experiments the authors performed and the thoughtful content that is included in this work. However, the number and variety of different experiments run ends up being a hindrance to the final work, as the paper ends up disjointed and very difficult to follow. Multiple steps are performed sequentially and evaluated on different datasets - lung masking, then training on one dataset, then fine tuning on another, then extracting certain layers, then ensembling, and both classification and segmentation accuracy are assessed - and it is hard to draw any firm conclusions about how ensembling contributes to the overall stack given how many moving pieces there are. These clarifications are needed because despite the work being based on open-source datasets, insufficient details are given to allow authors to replicate the work.

12. Many of the technical details could be moved to a supplement, and more attention could be given to clearly outlining the various steps of the experiment and how they fit together, both in text and in figures as possible. The datasets used (and the partitions) need to be much more clearly stated for each step, and the reader needs to be assured that test was not used in the ensembling process except for final evaluation. Figures need to be used more effectively to display the complexity of the experiments. As much as possible, the contribution of ensembling needs to be isolated by providing apples to apples statistical testing against non-ensembles models on the same dataset.

13. Finally - the authors could alternatively simplify the experiment. If they wish to isolate the effect of ensembling on predictive accuracy, they could use their original results from CheXpert to develop an ensemble of models, evaluate it on CheXpert test, and (if they desired) show how this ensemble generalizes to RSNA as an additional validation. If they wish to focus on how ensembling improves the region of abnormality detection, they could limit their experiments to this on RSNA data. If the authors feels strongly that their full pipeline is needed, the various components need to be individually defended and justified.

I appreciate having had the chance to review this research and thank the authors for their contribution.
Additional comments: No additional comments