{'fraction_non_alphanumeric': 0.02371243014650355, 'fraction_numerical': 0.007292812911299545, 'mean_word_length': 4.95044293234048, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Shortening patient-reported outcome measures through optimal test assembly: application to the Social Appearance Anxiety Scale in the Scleroderma Patient-centered Intervention Network Cohort

Objectives The Social Appearance Anxiety Scale (SAAS) is a 16-item measure that assesses social anxiety in situations where appearance is evaluated. The objective was to use optimal test assembly (OTA) methods to develop and validate a short-form SAAS based on objective and reproducible criteria. Design This study was a cross-sectional analysis of baseline data from adults enrolled in the Scleroderma Patient-centered Intervention Network (SPIN) Cohort. Setting Adults in the SPIN Cohort in the present study were enrolled at 28 centres in Canada, the USA and the UK. Participants The SAAS was administered to 926 adults with scleroderma. Primary and secondary measures The SAAS, Brief Fear of Negative Evaluation II (BFNE II), Brief Satisfaction with Appearance Scale (Brief-SWAP), Patient Health Questionnaire-8 (PHQ8) and Social Interaction Anxiety Scale-6 (SIAS-6) were collected, as well as demographic characteristics. Results OTA methods identified a maximally informative shortened version for each possible form length between 1 and 15 items. The final shortened version was selected based on prespecified criteria for reliability, concurrent validity and statistically equivalent convergent validity with the BFNE II scale. A five-item short version was selected (SAAS-5). The SAAS-5 had a Cronbach’s α of 0.95 and had high concurrent validity with the full-length form (r=0.97). The correlation of the SAAS-5 with the BFNE II was 0.66, which was statistically equivalent to that of the full-length form. Furthermore, the correlation of the SAAS-5 with the two subscales of the Brief-SWAP, and the SIAS-6, were statistically equivalent to that of the full-length form. Conclusions OTA was an efficient method for shortening the full-length SAAS to create the SAAS-5.

1. Consider explaining the difference between a summed score versus factor score, especially since the factor score was what drove the length of the shortened SAAS to be 5 since any shorter would fail to meet the criteria on concurrent validity for the factor score. Although the authors note that there are "well known limitations of summed scores..." (p15 line38), it is not well known to a non-psychometrician rheumatologist reading this.
2. As a non-statistician reader, I did not understand why (p22 para1) those three items were included in all short forms and the other three were dropped, as compared to the items highlighted 1. While shortening scales for reducing burden is certainly an admirably goal, my major concern is that the methodology used for creating the SAAS short form in this study implies not only that the Optimal Test Assembly Method can be used to shorten an existing instrument, but that the shortened instrument can be used in this specific scleroderma population as an evaluative measure over time. Firstly, the method assumed that the items in the original 16 item-scale are representative of all the items in the original developmental sample and are appropriate for this disease-specific scleroderma population. Furthermore, the validation method used to select the reduced 5-item set were limited as compared to the original development of the 16-item scale. To conclude that the reduced 5-item scale has "comparable" validity and reliability is overreaching.

2.
Taking an existing fixed 16-item, scale and "reducing" it to 5 items based primarily on the Cronbach's alpha, convergent validity with one of the original scales used by the original developer of the SAAS (BFNE II), and a 6-item version of the SAAS-6, does not provide enough evidence that this newly developed 5-item scale maintains high reliability and validity, especially in the population with scleroderma. It is also of concern that the development sample population was quite different from the scleroderma population. The development sample population "included three different samples of participants from two large public universities, who volunteered for the study in return for credit toward their research requirement for an introductory psychology course", and "Samples 1 and 3 were recruited at Temple University, whereas Sample 2 was recruited at the University of Buffalo (SUNY)".
3. Furthermore, the SAAS was also a rationally derived measure, in which items were chosen based on examination of current generic measures of social anxiety, body image dissatisfaction, and body dysmorphic disorder. As such, even the original SAAS scale did not conduct focus group qualitative analysis to derive the content for the development item pool. In addition, if the purpose of the current 5item scale is to evaluate changes in a population of scleroderma patients over time, then it would be more appropriate to assemble expert committee panels and focus groups of health care providers and scleroderma patients for generating the content for the developmental item pool. Then the content would represent the coverage and operative ranges important to scleroderma patients and their physicians, so that clinical meaningful changes of worsening or improvement in social appearance anxiety could be detected.

Even, if the original content is appropriate for this
population, it is not certain that the 5item scale can be used in place of the 16-item scale, especially for assessing change since the current study only examined cross-sectional data. The 5-item scale was not tested using the entire item bank and validation methodology used in the development of the 16-item scale upon repeat administrations. The authors' analysis was based upon one baseline assessment in one disease condition.
5. The analysis does not qualify this 5-item scale to be a condition-specific instrument for scleroderma, nor does it provide evidence that it is an evaluative measure capable of detecting changes due to an intervention. In the manuscript, there was no discussion of coverage, test-retest reliability, discriminant validity or the ability to retain enough precision to be able to detect longitudinal changes due to treatments over time. This latter performance metric would seem to be the one of most interest to this group.
6. It also appears that the authors are claiming that optimal test assembly is a methodology for taking validated scales and shortening them using only the items in that scale, which was not the original purpose of the methodology. The authors should first consider the developmental methodology of the original SAAS scale (Hart and Flora, Development and Examination of the Social Appearance Anxiety Scale, Assessment, 2008) to ensure that the same 5-items would have been selected if the purpose was to use it in this scleroderma population, or to at the outset only select 5 items. For this analysis, they have assumed that the "item bank" is the 16 items, but the rationally derived method used to develop the SAAS, also did not use the original item content of the predicate scales from which the SAAS items were taken. Even if they started with just the 16-item data pool, at the very least, they need to conduct more performance testing than was the case here.

7.
Typically, optimal test-design methods are applied to rule-based item generation for test assembly from a pool of pre-generated, calibrated items; test generation on the fly from a pool of calibrated item families; and 3test generation on the fly directly from calibrated features defining the item families. It is possible that while the "optimal test" is in fact the selected 5 items based upon 16-item data pool ,there was still limited test performance analysis conducted in this study. The same five items might not be optimal, for the full item bank based upon the item content in the original scales, or one derived from a scleroderma expert panel and focus group qualitative analysis. In summary, the authors need to state the intent of the purpose of shortening the scale more clearly. Is it to simply show that the optimal test methods can be used to shorten a pre-existing instrument, or to use to describe the states of social appearance anxiety in this population? Is the intentto examine differences between groups (discriminant measure), or predict outcomes (predictive measure), or detect changes over time due to treatment and programs (evaluative measure)? The analysis of the performance metrics (validity, reliability, responsiveness and sensitivity) need to be appropriate for the purpose of the measure. Limitations of the shortened scale, and the disease-specific versus generic uses also need to be discussed.

Reviewer 1:
This study describes how optimal test assembly was applied to the Social Appearance Anxiety Scale (SAAS) to shorten it from a 16-item to a 5-item PRO, while maintaining most of the full-length version's internal consistency reliability and convergent validity with other PROs with relatable constructs, as pertaining to SAAS and other PRO data collected from the SPIN cohort. This methodology was previously described by the same group on shortening the Cochin hand function scale. The method has the potential to be very important and useful to reduce participant burden and produce succinct information. Therefore, this is an important topic.
The manuscript is well written, and methods and results are sound. I have minor suggestions: We thank Reviewer #1 for her recognition of the importance of our study and the potential of the methods we describe for reducing patient burden and still providing useful outcome information.
1. Consider explaining the difference between a summed score versus factor score, especially since the factor score was what drove the length of the shortened SAAS to be 5 since any shorter would fail to meet the criteria on concurrent validity for the factor score. Although the authors note that there are "well known limitations of summed scores..." (p15 line38), it is not well known to a nonpsychometrician rheumatologist reading this.
We now include an explanation. Page 15 now reads: "First, the summed scores across all items included in the form were calculated by adding item scores for each item included in the form. Second, factor scores, which estimate a level of a latent construct, were estimated from the generalized partial credit model for each participant for each form through an application of Bayes' theorem. Although summed scores are typically relied upon for clinical use, the factor scores were considered to provide a better estimate of the underlying construct. This is because of limitations of summed scores under the generalized partial credit model. Summed scores may result in an incorrect ordering of patients along the spectrum of the underlying construct. That is, patients with lower levels of fear may have higher summed scores than patients with higher levels of fear 30 31 .
2. As a non-statistician reader, I did not understand why (p22 para1) those three items were included in all short forms and the other three were dropped, as compared to the items highlighted previously (p19 para 1) to have the highest and lowest discriminative ability. Perhaps some explanation would make this more clear.

This occurs because the Optimal Test
Assembly procedure relies on more information than just the magnitude of the discrimination parameter. In fact, this illustrates the primary difference between selecting items into a shortened form based on discrimination parameters (or factor loadings) and selecting items through the optimal test assembly procedure. Harel and Baron (2018) [http://journals.sagepub.com/doi/full/10.1177/0962280218795187] discusses the difference between these two procedures thoroughly. We have now included a citation to this paper as well as a short explanation in the manuscript. Page 22 now reads: "Although question 13 had a higher discrimination parameter estimate than question 14, it was not included in shortened forms of lengths shorter than 4. This is because the OTA procedure accounts for a more complete assessment of an item than just its discrimination parameter 19 . That is, if two items have the same level of discrimination, but provide information at the same point on the latent spectrum, then the OTA procedure may not select both items into the shortened form." 3. Some discussion on the drawback of using a method such as this which does not take into account content validity would be helpful.
We have now added a discussion of this topic. The limitations section now includes: "The method used in this study does not include content validity or expert assessment of the items selected into the shortened form. Had an expert panel been convened, they may have selected a different subset of items into the shortened form. An expert panel may have been able to use their knowledge to select items that were appropriate for the detection of clinically meaningful changes of worsening or improvement. However, such a procedure would not rely directly on patient data, may not be replicable, and may result in reduced measure validity based on imperfect clinical intuition 19 ." 4. What might be the reasons why the convergent validity was not statistically equivalent for the PHQ-8? Is this important?
We have now added this to the discussion, which now reads: "Although the SAAS-5 did not maintain a statistically equivalent convergent validity correlation with the PHQ-8, this does not suggest poor convergent validity. This may have occurred because items that captured symptoms most relevant to depression in the SAAS are no longer included in the SAAS-5. Scores on the short-form remained moderately correlated in the expected direction with the PHQ-8." 5. Adding in the discussion that this cohort has a relatively low skin score and its implications would be relevant.
We have now added this to the discussion, which now reads: "In addition, this sample had a relatively low skin score, which may limit the generalizability of study findings to patients with low disease severity. "

Reviewer 2:
For the manuscript entitled "Shortening Patient Reported Outcome Measures through Optimal Test Assembly: Application to the Social Appearance Anxiety Scale in the Scleroderma Patient centered Intervention Network Cohort" I would like to offer these comments.

1.
While shortening scales for reducing burden is certainly an admirably goal, my major concern is that the methodology used for creating the SAAS short form in this study implies not only that the Optimal Test Assembly Method can be used to shorten an existing instrument, but that the shortened instrument can be used in this specific scleroderma population as an evaluative measure over time.
We note your point regarding sensitivity to change. We do not mean to imply that we have assessed whether the SAAS-5 is sensitive to change or intervention status. We have now clarified this in the discussion section of the manuscript, which now reads: "This study utilized cross-sectional data, and therefore the sensitivity to change or intervention status, discriminant validity, and test-retest reliability of the SAAS-5 were not investigated. The purpose in the present study was to illustrate the use of OTA in creating a shortened version of a full-length form, and to propose a new shortened version of the SAAS. Future studies should investigate these properties in order to assess the discriminant, predictive, and evaluative characteristics of the SAAS-5.

The method used in this study does not include content validity or expert assessment of the items selected into the shortened form. Had an expert panel or focus group of patients been convened, they may have selected a different subset of items into the shortened form. An expert panel may have been able to use their knowledge to select items that were appropriate for the detection of clinically meaningful changes of worsening or improvement. However, such a procedure would not rely directly on patient data, may not be replicable, and may result in reduced measure validity based on imperfect clinical intuition 19 …
The OTA procedure is data-driven and results of this study should be replicated in this patient population. An analysis based on one sample of SSc patients may not be sufficient for the derivation of a disease-specific measure." Furthermore, the conclusion section now includes the following: "This analysis should be replicated in this patient population, as well as other patient populations, to increase the generalizability of these findings. As well, expert opinions or focus groups should be solicited to assess whether the items selected into the shortened form match clinical intuition." Firstly, the method assumed that the items in the original 16 item-scale are representative of all the items in the original developmental sample and are appropriate for this disease-specific scleroderma population. Furthermore, the validation method used to select the reduced 5item set were limited as compared to the original development of the 16-item scale. To conclude that the reduced 5-item scale has "comparable" validity and reliability is overreaching.
We now note that the methodology used in this study treats the full SAAS as the total possible item bank, and that if a larger item bank would be used (such as the list of all items considered in the original paper), that a different shortened version may have been selected. However, since the SAAS is commonly used for patients with scleroderma, our goal was to derive a shortened version that performed comparably (as defined by our decision rules) to the full-length form. We further note that our decision rules are investigator-defined and could be changed in future studies.
The discussion section now reads: "The OTA procedure is sensitive to the choice of decision criteria in the selection of the final shortened version. These decision criteria, when applied in future studies, must be carefully considered by researchers. Furthermore, the OTA method treats the 16 items of the SAAS as if they represented a full item bank of possible items. It is possible that if other items were considered, that a different set of items would have been selected into the final shortened version." 2. Taking an existing fixed 16-item, scale and "reducing" it to 5 items based primarily on the Cronbach's alpha, convergent validity with one of the original scales used by the original developer of the SAAS (BFNE II), and a 6-item version of the SIAS-6, does not provide enough evidence that this newly developed 5-item scale maintains high reliability and validity, especially in the population with scleroderma.
We have changed the wording of the final sentence to read "comparable reliability and validity" rather than "high." We define comparable based on the decision rules used, which we note are investigator defined and could be changed in future studies.
It is also of concern that the development sample population was quite different from the scleroderma population. The development sample population "included three different samples of participants from two large public universities, who volunteered for the study in return for credit toward their research requirement for an introductory psychology course", and "Samples 1 and 3 were recruited at Temple University, whereas Sample 2 was recruited at the University of Buffalo (SUNY)".
We agree with the reviewer that the sample used in this study is different from the development sample. However, the SAAS has been shown to have good measurement properties in patients with scleroderma, and is used in such samples. Therefore, our goal was to develop a shortened form that could also be used in the scleroderma population.
The discussion section now reads: "The results of this study are only applicable for patients with SSc." 3. Furthermore, the SAAS was also a rationally derived measure, in which items were chosen based on examination of current generic measures of social anxiety, body image dissatisfaction, and body dysmorphic disorder. As such, even the original SAAS scale did not conduct focus group qualitative analysis to derive the content for the development item pool. In addition, if the purpose of the current 5-item scale is to evaluate changes in a population of scleroderma patients over time, then it would be more appropriate to assemble expert committee panels and focus groups of health care providers and scleroderma patients for generating the content for the developmental item pool. Then the content would represent the coverage and operative ranges important to scleroderma patients and their physicians, so that clinical meaningful changes of worsening or improvement in social appearance anxiety could be detected.
We now note this point in the discussion section, which reads: "The method used in this study does not include content validity or expert assessment of the items selected into the shortened form. Had an expert panel or focus group of patients been convened, they may have selected a different subset of items into the shortened form. An expert panel may have been able to use their knowledge to select items that were appropriate for the detection of clinically meaningful changes of worsening or improvement. However, such a procedure would not rely directly on patient data, may not be replicable, and may result in reduced measure validity based on imperfect clinical intuition 19 . More resource-intensive methods for developing short forms, such focus groups and content experts, along with replicable statistical criteria, would be ideal. However, the resources necessary to complete these procedures may represent a substantial barrier to the development of shortened forms. The OTA method provides a replicable method that maintains performance standards based on objective criteria and provides a more feasible method." 4. Even, if the original content is appropriate for this population, it is not certain that the 5-item scale can be used in place of the 16-item scale, especially for assessing change since the current study only examined cross-sectional data. The 5-item scale was not tested using the entire item bank and validation methodology used in the development of the 16-item scale upon repeat administrations. The authors' analysis was based upon one baseline assessment in one disease condition.
Please see our response to point 1.
5. The analysis does not qualify this 5-item scale to be a condition-specific instrument for scleroderma, nor does it provide evidence that it is an evaluative measure capable of detecting changes due to an intervention. In the manuscript, there was no discussion of coverage, test-retest reliability, discriminant validity or the ability to retain enough precision to be able to detect longitudinal changes due to treatments over time. This latter performance metric would seem to be the one of most interest to this group.
We now note that this data-driven analysis should be repeated in another sample of scleroderma patients. The discussion section now reads: "The OTA procedure is data-driven and results of this study should be replicated in this patient population. An analysis based on one sample of SSc patients may not be sufficient for the derivation of a disease-specific measure." Furthermore, as discussed above, we do not mean to imply that we have assessed sensitivity to change, and have clarified this point in the discussion.
6. It also appears that the authors are claiming that optimal test assembly is a methodology for taking validated scales and shortening them using only the items in that scale, which was not the original purpose of the methodology.
We agree that this was not the original point of this methodology. Please see Harel and Baron (2018) [http://journals.sagepub.com/doi/full/10.1177/0962280218795187] shows further how optimal test assembly can be used to shorten scales. We now cite this paper in the text.
The authors should first consider the developmental methodology of the original SAAS scale (Hart and Flora, Development and Examination of the Social Appearance Anxiety Scale, Assessment, 2008) to ensure that the same 5-items would have been selected if the purpose was to use it in this scleroderma population, or to at the outset only select 5 items. For this analysis, they have assumed that the "item bank" is the 16 items, but the rationally derived method used to develop the SAAS, also did not use the original item content of the predicate scales from which the SAAS items were taken. Even if they started with just the 16-item data pool, at the very least, they need to conduct more performance testing than was the case here.
Please see our response to point 1.

7.
Typically, optimal test-design methods are applied to rule-based item generation for test assembly from a pool of pre-generated, calibrated items; test generation on the fly from a pool of calibrated item families; and test generation on the fly directly from calibrated features defining the item families. It is possible that while the "optimal test" is in fact the selected 5 items based upon 16-item data pool, there was still limited test performance analysis conducted in this study. The same five items might not be optimal, for the full item bank based upon the item content in the original scales, or one derived from a scleroderma expert panel and focus group qualitative analysis.
Please see our response to this point above.
In summary, the authors need to state the intent of the purpose of shortening the scale more clearly. Is it to simply show that the optimal test methods can be used to shorten a pre-existing instrument, or to use to describe the states of social appearance anxiety in this population? Is the intent to examine differences between groups (discriminant measure), or predict outcomes (predictive measure), or detect changes over time due to treatment and programs (evaluative measure)? The analysis of the performance metrics (validity, reliability, responsiveness and sensitivity) need to be appropriate for the purpose of the measure. Limitations of the shortened scale, and the disease-specific versus generic uses also need to be discussed.
The intent of our study was to both illustrate how optimal test assembly may be used to shorten a pre-existing instrument, as well as propose a shortened version of the SAAS that can be used in the patients with scleroderma. We have added several limitations of our study to the discussion section (please see our responses above for the exact text added). One of these limitations is that we recognize that the results of our study should be replicated. We agree with the reviewer that future studies should assess sensitivity to change, discriminant validity, and reliability, however, we believe this is beyond the scope of the present study. Furthermore, we now discuss that our shortened form, at present, should only be considered for patients with scleroderma, not the general population, or other patient populations.

GENERAL COMMENTS
The authors have clarified my points of concern appropriately. However, Reviewer 2 who appears to be more knowledgeable in the methodology had some valid concerns, especially about the appropriateness of the way OTA was used. I would suggest for the authors to remove the third point in the "Strength and Limitations of this Study" that refers to "guaranteed to be similarly reliable and valid to the full length version." This statement is misleading, especially to the less methodology-informed reader.

GENERAL COMMENTS
For the manuscript entitled "Shortening Patient Reported Outcome Measures through Optimal Test Assembly: Application to the Social Appearance Anxiety Scale in the Scleroderma Patient centered Intervention Network Cohort" I have reviewed the revisions in the current submission, and I believe that they have addressed many of the following concerns in the Discussion section. I would like to offer these comments in light of the revised manuscript. Please note that I believe that Comment 2 and related Comment 5 still need to be addressed, because I feel that the intent of the original comment was probably not fully interpreted by the authors. 1. Original Comment: While shortening scales for reducing burden is certainly an admirably goal, my major concern is that the methodology used for creating the SAAS short form in this study implies not only that the Optimal Test Assembly Method can be used to shorten an existing instrument, but that the shortened instrument can be used in this specific scleroderma population as an evaluative measure over time. Firstly, the method assumed that the items in the original 16 item-scale are representative of all the items in the original developmental sample and are appropriate for this disease-specific scleroderma population. Furthermore, the validation method used to select the reduced 5-item set were limited as compared to the original development of the 16-item scale. To conclude that the reduced 5-item scale has "comparable" validity and reliability is overreaching. a. REVISION ACCEPTABLE: The authors addressed this concern in the Paragraph 4 in the Discussion section by clarifying under the list of limitations that: " This study utilized cross-sectional data, and therefore the sensitivity to change or intervention status, discriminant validity, and or test-retest reliability of the SAAS-5 were as not investigated. The purpose in the present study was to illustrate the use of OTA in creating a shortened version of a full-length form, and to propose a new shortened version of the SAAS. Future studies should investigate these properties in order to assess the discriminant, predictive, and evaluative characteristics of the SAAS-5.

Original Comment:
Taking an existing fixed 16-item, scale and "reducing" it to 5 items based primarily on the Cronbach's alpha, convergent validity with one of the original scales used by the original developer of the SAAS (BFNE II), and a 6-item version of the SAAS-6, does not provide enough evidence that this newly developed 5-item scale maintains high reliability and validity, especially in the population with scleroderma. It is also of concern that the development sample population was quite different from the scleroderma population. The development sample population "included three different samples of participants from two large public universities, who volunteered for the study in return for credit toward their research requirement for an introductory psychology course", and "Samples 1 and 3 were recruited at Temple University, whereas Sample 2 was recruited at the University of Buffalo ( population, it is not certain that the 5-item scale can be used in place of the 16-item scale, especially for assessing change since the current study only examined cross-sectional data. The 5-item scale was not tested using the entire item bank and validation methodology used in the development of the 16-item scale upon repeat administrations. The authors' analysis was based upon one baseline assessment in one disease condition. a. REVISION ACCEPTABLE: The authors now address this concern in Paragraph 6 of the Discussion section by adding "Furthermore, the OTA method treats the 16 items of the SAAS as if they represented a full item bank of possible items. It is possible that if other items were considered, that a different set of items would have been selected into the final shortened version. " 5. Original Comment: The analysis does not qualify this 5-item scale to be a condition-specific instrument for scleroderma, nor does it provide evidence that it is an evaluative measure capable of detecting changes due to an intervention. In the manuscript, there was no discussion of coverage, test-retest reliability, discriminant validity or the ability to retain enough precision to be able to detect longitudinal changes due to treatments over time. This latter performance metric would seem to be the one of most interest to this group. a. NEEDS FURTHER REVISION: If Comment 2 above is addressed along with mentioning that clinically meaningful "longitudinal changes" due to treatments in a clinical trial of SSc patients" will need further study, this would be acceptable. 6. Original Comment: It also appears that the authors are claiming that optimal test assembly is a methodology for taking validated scales and shortening them using only the items in that scale, which was not the original purpose of the methodology. The authors should first consider the developmental methodology of the original SAAS scale (Hart and Flora, Development and Examination of the Social Appearance Anxiety Scale, Assessment, 2008) to ensure that the same 5-items would have been selected if the purpose was to use it in this scleroderma population, or to at the outset only select 5 items. For this analysis, they have assumed that the "item bank" is the 16 items, but the rationally derived method used to develop the SAAS, also did not use the original item content of the predicate scales from which the SAAS items were taken. Even if they started with just the 16-item data pool, at the very least, they need to conduct more performance testing than was the case here. a. REVIEWER'S RESPONSE: This comment was not addressed explicitly. Again, the major concern is that this shortened version once published will be picked up by investigators searching for body image and social anxiety instruments applicable for SSc patients for monitoring disease symptoms over time and changes due to treatment. There are very specific appearance concerns for patients with scleroderma especially regarding face, skin, fingers, toes, etc. Treatments which may improve these concerns might not be detected in the original 16-item form because only the general construct of social appearance anxiety was addressed in normal college students. However, the generable areas do seem to have face validity. With the 5-item version, the number of tick marks on this generic social appearance anxiety scale to detect clinically meaningful changes is reduced by a factor of 70%. Again, this potentially limits the use in the SSc for evaluative research. 7. Original Comment: Typically, optimal test-design methods are applied to rule-based item generation for test assembly from a pool of pre-generated, calibrated items; test generation on the fly from a pool of calibrated item families; and test generation on the fly directly from calibrated features defining the item families. It is possible that while the "optimal test" is in fact the selected 5 items based upon 16-item data pool ,there was still limited test performance analysis conducted in this study. The same five items might not be optimal, for the full item bank based upon the item content in the original scales, or one derived from a scleroderma expert panel and focus group qualitative analysis. a. Can be addressed as cited above.
Original Comment: In summary, the authors need to state the intent of the purpose of shortening the scale more clearly. Is it to simply show that the optimal test methods can be used to shorten a pre-existing instrument, or to use to describe the states of social appearance anxiety in this population? Is the intent to examine differences between groups (discriminant measure), or predict outcomes (predictive measure), or detect changes over time due to treatment and programs (evaluative measure)? The analysis of the performance metrics (validity, reliability, responsiveness and sensitivity) need to be appropriate for the purpose of the measure. Limitations of the shortened scale, and the disease-specific versus generic uses also need to be discussed. b. REVISION ACCEPTABLE: The authors added the sentence in Paragraph 4 of the Discussion: "The purpose in the present study was to illustrate the use of OTA in creating a shortened version of a full-length form, and to propose a new shortened version of the SAAS. Future studies should investigate these properties in order to assess the discriminant, predictive, and evaluative characteristics of the SAAS-5." which addresses this concern.
The authors have clarified my points of concern appropriately. However, Reviewer 2 who appears to be more knowledgeable in the methodology had some valid concerns, especially about the appropriateness of the way OTA was used. I would suggest for the authors to remove the third point in the "Strength and Limitations of this Study" that refers to "guaranteed to be similarly reliable and valid to the full length version." This statement is misleading, especially to the less methodologyinformed reader.
We thank Reviewer 1 for their comment, and have removed the third point in the "Strengths and Limitations of this Study" section.

Reviewer 2:
For the manuscript entitled "Shortening Patient Reported Outcome Measures through Optimal Test Assembly: Application to the Social Appearance Anxiety Scale in the Scleroderma Patient centered Intervention Network Cohort" I have reviewed the revisions in the current submission, and I believe that they have addressed many of the following concerns in the Discussion section. I would like to offer these comments in light of the revised manuscript. Please note that I believe that Comment 2 and related Comment 5 still need to be addressed, because I feel that the intent of the original comment was probably not fully interpreted by the authors.
We thank the Reviewer for clarifying these original comments and respond point-by-point below.
2. Original Comment: Taking an existing fixed 16-item, scale and "reducing" it to 5 items based primarily on the Cronbach's alpha, convergent validity with one of the original scales used by the original developer of the SAAS (BFNE II), and a 6-item version of the SAAS-6, does not provide enough evidence that this newly developed 5-item scale maintains high reliability and validity, especially in the population with scleroderma. It is also of concern that the development sample population was quite different from the scleroderma population. The development sample population "included three different samples of participants from two large public universities, who volunteered for the study in return for credit toward their research requirement for an introductory psychology course", and "Samples 1 and 3 were recruited at Temple University, whereas Sample 2 was recruited at the University of Buffalo (SUNY)". a) NEEDS FURTHER REVISION: This comment was not addressed adequately in the revision.
In the final Paragraph of the Discussion section, the authors added a sentence "An analysis based on one sample of SSc patients may not be sufficient for the derivation of a diseasespecific measure.  (1):48-59 ) was conducted in "three different samples of participants from two large public universities, who volunteered for the study in return for credit toward their research requirement for an introductory psychology course". The fundamental issues that are important to SSc patients relating to Social Anxiety construct are most likely different than those of healthy college students volunteering for an instrument development study in order to get credit for their research requirement. The comment did not refer to the fact that the authors shortened form is only valid in patients with SSc. It referred to that fact that the original 16-item instrument was not developed as a disease specific form for SSc, but rather tapped the "coverage" of the construct explicitly for individuals who did not have SSc. The statement that "The results of this study are only applicable for patients with SSc." Seems to convey the opposite. That is, it gives the reader the impression that the instrument is a disease-specific instrument for SSc only. b) It would be useful to add the fact that "the original SAAS developmental sample consisted of three different samples of participants from two large public universities, who volunteered for the study in return for credit toward their research requirement for an introductory psychology course of the original SAAS instrument and as such may not provide sufficient coverage in terms of content validity to serve as a disease-specific shortened version for SSc patients.
We thank the reviewer for clarifying this comment. We now note that: "The results of this study are only as applicable for patients with SSc as the original full-length SAAS. It should be noted that the original SAAS was developed based on three different samples of volunteers from introductory psychology courses at large public universities. Therefore, even the original SAAS instrument might not provide sufficient coverage in terms of content validity for patients with SSc." 5. Original Comment: The analysis does not qualify this 5-item scale to be a condition-specific instrument for scleroderma, nor does it provide evidence that it is an evaluative measure capable of detecting changes due to an intervention. In the manuscript, there was no discussion of coverage, test-retest reliability, discriminant validity or the ability to retain enough precision to be able to detect longitudinal changes due to treatments over time. This latter performance metric would seem to be the one of most interest to this group. a) NEEDS FURTHER REVISION: If Comment 2 above is addressed along with mentioning that clinically meaningful "longitudinal changes" due to treatments in a clinical trial of SSc patients" will need further study, this would be acceptable.
We now note that: "Furthermore, the assessment of longitudinal changes that are clinically meaningful due to, for example, treatment in a clinical trial of SSc patients, would need further study." 6. Original Comment: It also appears that the authors are claiming that optimal test assembly is a methodology for taking validated scales and shortening them using only the items in that scale, which was not the original purpose of the methodology. The authors should first consider the developmental methodology of the original SAAS scale (Hart and Flora, Development and Examination of the Social Appearance Anxiety Scale, Assessment, 2008) to ensure that the same 5-items would have been selected if the purpose was to use it in this scleroderma population, or to at the outset only select 5 items. For this analysis, they have assumed that the "item bank" is the 16 items, but the rationally derived method used to develop the SAAS, also did not use the original item content of the predicate scales from which the SAAS items were taken. Even if they started with just the 16-item data pool, at the very least, they need to conduct more performance testing than was the case here. a) REVIEWER'S RESPONSE: This comment was not addressed explicitly. Again, the major concern is that this shortened version once published will be picked up by investigators searching for body image and social anxiety instruments applicable for SSc patients for monitoring disease symptoms over time and changes due to treatment. There are very specific appearance concerns for patients with scleroderma especially regarding face, skin, fingers, toes, etc. Treatments which may improve these concerns might not be detected in the original 16-item form because only the general construct of social appearance anxiety was addressed in normal college students. However, the generable areas do seem to have face validity. With the 5-item version, the number of tick marks on this generic social appearance anxiety scale to detect clinically meaningful changes is reduced by a factor of 70%. Again, this potentially limits the use in the SSc for evaluative research.
We thank the reviewer for this clarification. We note that: "…the assessment of longitudinal changes that are clinically meaningful due to, for example, treatment in a clinical trial of SSc patients, would need further study. Therefore, this may limit the utility of the SAAS-5 as an evaluative measure in patients with SSc." 7. Original Comment: Typically, optimal test-design methods are applied to rule-based item generation for test assembly from a pool of pre-generated, calibrated items; test generation on the fly from a pool of calibrated item families; and test generation on the fly directly from calibrated features defining the item families. It is possible that while the "optimal test" is in fact the selected 5 items based upon 16-item data pool ,there was still limited test performance analysis conducted in this study. The same five items might not be optimal, for the full item bank based upon the item content in the original scales, or one derived from a scleroderma expert panel and focus group qualitative analysis. a) Can be addressed as cited above.
To address this comment, we note that: "The method used in this study does not include content validity or expert assessment of the items selected into the shortened form. Had an expert panel or focus group of patients been convened, they may have selected a different subset of items into the shortened form. An expert panel may have been able to use their knowledge to select items that were appropriate for the detection of clinically meaningful changes of worsening or improvement. However, such a procedure would not rely directly on patient data, may not be replicable, and may result in reduced measure validity based on imperfect clinical intuition 20 . More resource-intensive methods for developing short forms, such focus groups and content experts, along with replicable statistical criteria, would be ideal. However, the resources necessary to complete these procedures may represent a substantial barrier to the development of shortened forms.
The OTA method provides a replicable method that maintains performance standards based on objective criteria and provides a more feasible method." As well as: "Furthermore, the OTA method treats the 16 items of the SAAS as if they represented a full item bank of possible items. It is possible that if other items were considered, that a different set of items would have been selected into the final shortened version." We believe that these comments, as well as the ones noted above, adequately respond to this concern.