{'fraction_non_alphanumeric': 0.04637428599089941, 'fraction_numerical': 0.02594636460451157, 'mean_word_length': 4.865701306076093, 'pattern_counts': {'":': 0, '<': 4, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A psychometric evaluation of the National Stressful Events Survey for PTSD-Short Scale (NSESSS-PTSD) among Korean psychiatric outpatients

ABSTRACT The National Stressful Events Survey for PTSD-Short Scale (NSESSS-PTSD) is a brief screening measure for DSM-5 PTSD that has not been evaluated for its psychometric properties in clinical population. We developed a Korean version of the original English scale through translation-back translation process and examined its reliability and validity among treatment-seeking adults at a psychiatric outpatient unit of a university-affiliated hospital in South Korea. The sample comprised adults diagnosed with PTSD (n = 100) and other psychiatric disorders (n = 134). The NSESSS-PTSD, the PTSD Checklist for DSM-5 (PCL-5), the Beck Depression Inventory-II (BDI-II), and the Beck Anxiety Inventory (BAI) were used to determine validity and reliability. The findings show modest test-retest reliability (r = .43), good internal consistency (Cronbach’s α = .81), high convergent validity (r = .78) with PCL-5 and good concurrent validity with the BDI (r = .55) and BAI (r = .50), respectively. A cut-off score of 16 best predicted PTSD from other psychiatric disorders with specificity of .90 and sensitivity of .87. This study reveals sound psychometric properties of the Korean version of the NSESSS-PTSD and supports its use in the clinical population.

To reflect these changes in the diagnostic criteria of posttraumatic stress disorder (PTSD) in DSM-5 (American Psychiatric Association, 2013), adults' self-reported questionnaires from previous DSM-IV versions were upgraded. Furthermore, the psychometric properties, including the PTSD Checklist for DSM-5 (PCL-5) (Weathers et al., 2013), the PTSD Diagnostic Scale for DSM-5 (Foa et al., 2016), and the Primary Care PTSD Screen for DSM-5 (Prins et al., 2016) were tested.
Additionally, a new PTSD instrument called the National Stressful Events Survey for PTSD (NSES-PTSD) was developed in accordance with DSM-5 (Kilpatrick et al., 2011). The original scale comprised 20 items that encompassed each symptom of the DSM-5 criteria for PTSD and yielded stable construct validity, including a high Cronbach's alpha of 0.94 (LeBeau et al., 2014). In the same paper, authors developed a reduced version, a 9-item subset of 20 PTSD symptoms that can be used to screen for PTSD or to measure symptom change over time. This brief scale, the National Stressful Events Survey PTSD Short Scale (NSESSS-PTSD) was developed by examining the reduction of internal consistency upon corresponding item deletions. The nine items included two intrusive symptoms (flashback and emotional reactivity), avoidance (avoidance of internal triggers), negative cognition or mood (distorted blame, negative emotions, and loss of interest), and three elements of arousal or reactivity (aggression, hypervigilance, and startled reactions).
The original NSESSS-PTSD was validated from 318 online respondents who met probable DSM-5 PTSD criteria: internal consistency was excellent (Cronbach α = .90) with the 20-item scale being 0.94 (LeBeau et al., 2014). Results from another independent sample of 66 undergraduate students confirmed its high internal reliability (Cronbach α = .91) and good convergent validity (r = .84) with the PTSD Checklistcivilian version (LeBeau et al., 2014). Other language versions of the NSESSS-PTSD that have undergone psychometric evaluations include the Turkish version for university students and inpatients with alcohol use disorder (Evren, Dalbudak, Aydemir, et al., 2016;Evren, Dalbudak, Umut, et al., 2016) as well as the Persian version for Iranian earthquake survivors (Rafiey et al., 2017).
However, there is a lack of psychometric data regarding the scale's application to clinical settings, most importantly concerning screening properties for adults with DSM-5 PTSD. Under-diagnosis of PTSD in mental health outpatient setting has been reported as a comorbid condition to other principal diagnoses (Kiefer et al., 2020) or as a principal diagnosis (false-negatives) by psychiatrists in training (da Silva et al., 2019). In fact, PTSD was highly prevalent among psychiatric outpatients in these studies 17.4% and 20.5%, respectively. Psychiatric outpatients with unrecognised PTSD also showed negative outcomes such as lower improvement of symptoms, poor perceived health status, and less employment when they were re-evaluated after 3-4 years (Al-Saffar et al., 2002). In short, screening instruments with good psychometric properties for PTSD are needed for initial assessment in mental health facilities.
As such, we designed our study to include comprehensive psychometric properties. First, as the NSESSS-PTSD was not available in Korean, we did a standard translation and back-translation process to adapt the scale to the Korean language. Second, we sought to evaluate the scale's psychometric properties including internal consistency, test-retest reliability, and construct validity among psychiatric outpatient populations. Based on prior findings on online responders with probable PTSD and patients with alcohol use disorders, we hypothesised that the NSESSS-PTSD would show good reliability and validity among mental health outpatients with DSM-5 PTSD.

Participants and procedures
Participants were treatment-seeking adults at the psychiatric outpatient unit of Hanyang University Guri Hospital, Gyeonggi Province, South Korea, who were initially given DSM-5 diagnosis of PTSD by attending psychiatrists and recruited by consecutive sampling from March 2017 to February 2020. When the process of written informed consent is completed, the questionnaires were given and one of a research psychiatrist (DK) and clinical psychologists (YM & HL) administered the Structured Clinical Interview for DSM-5 Disorders (First et al., 2016;Oh et al., 2017) in a blind condition to the results of the questionnaire. The inter-rater reliability among assessors was not examined. The completors of interview and questionnires received 30 US dollars as an imbursement for participation.
For comparison group, we used medical records and questionnaire data from consecutive sample of other psychiatric diagnoses but acute stress disorder. Each DSM-5 diagnosis in the comparison group was made by clinical interviews at the first appointment by attending psychiatrists. We did not confirm their diagnoses with the SCID or examine the inter-rater reliability of diagnoses.
The exclusion criteria were (1) intellectual disability and neurocognitive disorders, (2) severe medical conditions, and (3) comorbid psychotic illnesses. All participants provided written informed consent, and the study was approved by the hospital's Research Ethics Committee (IRB No. 2015-05-013).
In the PTSD group, of 106 subjects responded to questionnaires, 6 (5.7%) were excluded due to incomplete or missing data; in the comparison group, 417 questionnaires were gathered, and 107 were disregarded due to incomplete or missing data (25.7%). As we did not investigate PTSD comorbidity with other diagnostic group, we excluded an additional 176 (42.2%) participants who scored higher than 33, a cut-off score for possible PTSD on the PTSD Checklist for DSM-5 . Therefore, the final sample comprised 100 participants with PTSD and 134 with other psychiatric diagnoses.
To account for test-retest reliability, another convenient sample (n = 48) with PTSD at the same outpatient unit was administered with the NSESSS-PTSD between four-week interval. All of them were under active treatment, either psychotherapy alone (n = 24, 45.8%) or both psychotherapy and psychotropic medications (n = 26, 54.2%). Compared to the original sample, those tested for temporal stability had significantly younger age (median = 27.5 vs. 39.5, Mann-Whitney U = 1841.0, p = .022), longer period of time since the index trauma (median 96.0 vs. 6.5 months, Mann-Whitney U = 1138.5, p < .001), and more interpersonal trauma (58.3% vs. 41.0%, chi square = 3.92, p = .048). There were no significant differences in regard to sex and initial PTSD scores.

NSESSS-PTSD
The National Stressful Events Survey PTSD Short Scale (NSESSS-PTSD) is a nine-item abridged version of the original scale, which originally comprised 20 items that were designed to measure the 20 symptoms of DSM-5 PTSD (LeBeau et al., 2014). It the test, respondents were instructed to list the traumatic event they had experienced and rate how much they were bothered by each problem during the past week on a 5-point scale from 0 (not at all) to 4 (extremely). Thus, the total raw score ranged from 0 to 36, and this total score was to assess the severity of posttraumatic stress symptoms. Another use of the scale is the average total score, which is calculated by dividing the total score by the number of items. This average score easily and reliably categorised overall severity of PTSD as follows: none (0), mild (1), moderate (2), severe (3), and extreme (4) (Kilpatrick et al., 2013).
The NSESSS-PTSD has not been psychometrically evaluated in the clinical population, although it had been tested with college students to determine its internal consistency and validity compared to other PTSD instruments (LeBeau et al., 2014). Test-retest reliability, discriminant validity, cross-cultural adaptability, and establishing cut-off score have been suggested as future evaluation by the authors of the original version and those are addressed in our study.

PCL-5
The PTSD Checklist for DSM-5 (PCL-5) is a Likerttype self-report instrument that measures PTSD symptoms (Weathers et al., 2013). This scale has 20 items that address the DSM-5 diagnostic criteria for PTSD, and each item is rated on a Likert scale ranging from 0 (not at all) to 4 (extremely). The respondents were asked to rate each question how much they were bothered by the worst event they had experienced during the past month.
The original version proved excellent internal consistency (α = .96), high four-week test-retest reliability (r = .84), and good convergent validity with DSM-IV PCL (r = .84) in 468 treatment-seeking veterans . The Korean version used in this study showed high internal consistency, high test-retest reliability, and good concurrent validity among elderly Vietnam veterans (Kim et al., 2017). Additionally, it resulted in high internal consistency, good convergent validity with other PTSD measures, and discriminant validity against anxiety and depression among earthquake survivors (Seo & Cho, 2021). The Cronbach's alpha for this study was .92 for PTSD group.

BDI-II
The Beck Depression Inventory-II is a self-questionnaire designed to measure the degree of depressive symptoms present over the past two weeks. The scale comprises 21 items that ask respondents to choose similar states on a four-point Likert-type scale ranging from 0 to 3, which reflects intensity or frequency of the problem. The soundness of the psychometric properties of the BDI-II has been proven in many studies on psychiatric, clinical, non-clinical and medical populations (Wang & Gorenstein, 2013). The Korean version showed high internal consistency, excellent criterion validity with other depression measures, and good sensitivity and specificity among DSM-IV depressive disorders (Lim et al., 2011). The Cronbach α obtained in this study was .92 for PTSD group.

BAI
The Beck Anxiety Inventory is a 21-item instrument used to assess the severity of anxiety symptoms over the past month (Beck et al., 1988). It asks individuals to rate how affected they were by diverse symptoms of anxiety (e.g. 'terrified or afraid' and 'fear of worst happening') on a four-point scale ranging from 0 (not at all) to 3 (severely). The Korean version of the BAI has shown excellent internal consistency and discriminant validity for clinical anxiety disorders (Yook & Kim, 1997). The Cronbach's alpha for this study was .94 for PTSD group.

Data analyses
The PTSD and other diagnostic groups were compared using the Mann-Whitney U test (due to the violation of normal distribution in continuous variables) and chi-square tests. The Pearson's correlation coefficient was calculated to evaluate test-retest reliability, whereas Cronbach's alpha was calculated to determine internal consistency. To test convergent and discriminant validity, we conducted Spearman's correlation analyses between the NSESSS-PTSD and other measures. To obtain the best cut-off scores for identifying a participant with PTSD, a receiver operator characteristic (ROC) curve analysis was conducted with the NSESSS-PTSD assigned as the criterion variable (1 = PTSD, 0 = comparison group).
All statistical analyses were performed using SPSS 25 for Windows (IBM Co., Armonk, NY, USA) and the statistical significance was set to α = .05 bidirectionally.
Approximately two-thirds of the participants were women (62.0%) and had an annual income of less than 40,000 US dollars (67.0%). Slightly less than half of the participants were married (48.0%) and employed (47.0%), and most (86.0%) had high school education or higher. The mean the NSESSS-PTSD score was 24.5 (SD = 7.17, 95% CI = 23.1-25.9).

Comparison with the comparison group
As shown in Table 1, there were no statistically significant differences between the PTSD and comparison group regarding age, sex, marital status, employment, education and annual income (Table 1).

Convergent and discriminant validity
The NSESSS-PTSD and PCL-5 demonstrated a high correlation (ρ = .78, p < .001) between the two scales and significant correlations were found between the NSESSS-PTSD and the BDI (ρ = .55, p < .001) and between the NSESSS-PTSD and the BAI (ρ = .50, p < .001). The NSESSS also showed significant higher scores when compared with those of other psychiatric diagnoses: the mean scores of 24.5 (SD = 7.1) vs. 7.6 Table 1. General characteristics of participants with PTSD (n = 100) and other psychiatric disorders (n = 134). National Stressful Events Survey PTSD Short Scale; PCL-5 PTSD checklist (PCL)-5; U: Mann-Whitney U. Owing to missing data, the total sum of subjects may differ by variables.

ROC analysis
The ROC curve analysis yielded an area under curve (AUC) value of .95 (95% CI: .92-.98), indicating excellent accuracy when it comes to predicting PTSD cases (Figure 1). A cut-off score of 16 corresponded to a sensitivity of .87, a specificity of .90, a kappa of .74, a positive predictive power of .90, and a negative predictive power of .86.

Discussion
Our results suggest that the Korean-translated version of the NSESSS-PTSD has adequate to good psychometric properties, particularly its diagnostic utility against other diagnostic group, supporting its use for screening purposes in mental health outpatient setting. Our research hypothesis that the NSESSS-PTSD will provide favourable reliability and validity among clinical population has been confirmed.
In this study, the cut-off score of 16 for the identification of PTSD vs. other psychiatric disorders was identical to one Turkish study applying the NSESSS-PTSD to inpatients with alcohol use disorder (Evren, Dalbudak, Umut, et al., 2016) and lower than another Turkish study reporting 24 in the case of university students (Evren, Dalbudak, Aydemir, et al., 2016). Both studies determined cases with probable PTSD by PCL-C (DSM-IV based) scores while our study used a structured diagnostic interview (SCID) for PTSD cases and non-PTSD case definition by PCL-5 scores. The ROC curve demonstrated that the Korean version of NSESSS-PTSD had high predictive validity shown by excellent sensitivity, specificity, and the AUC.
Next, the Cronbach's α of the NSESSS-PTSD was good (.81) in our study. This is comparable to the original English version, which initially reported a Cronbach's α of .90 based on 318 adults from a large national sample with probable DSM-5 PTSD, as well as that of .91 from 66 college students who had experienced at least one traumatic event (LeBeau et al., 2014). Compared to versions in other languages, the Cronbach's α in our study was lower than those reported for Iranian earthquake survivors (α = .88, n = 600) (Rafiey et al., 2017) and Turkish inpatients with alcohol use disorder (α = .88, n = 190) (Evren, Dalbudak, Umut, et al., 2016). However, higher than those of impoverished urban residents in Bangladesh (α = .70, n = 435) (Islam et al., 2021). These differences between studies may depend on the population studied; in this case, the sample comprised treatment-seeking psychiatric outpatients with DSM-5 PTSD diagnoses from diverse traumatic events. It should be noted that values between .80 to .90 are ideal for a psychological instrument because a value higher than .9 may indicate that the scale has too many items (i.e. redundancy of the scale) (Cortina, 1993). Additionally, the NSESSS-PTSD is a brief test with relatively fewer items because longer tests tend to constitute more reliability (Tavakol & Dennick, 2011). Our study confirmed the internal consistency of the Korean version of the NSESSS-PTSD.
In our study, the correlation coefficient was lower (r = .43), which is considered to be modest for a psychometric questionnaire (Cicchetti, 1994). Only one study reported the retest reliability of the NSESSS-PTSD (Pearson r = .60) in Turkish inpatients with alcohol use disorder although the time interval between evaluations and any PTSD treatments given during the period were not mentioned. We have set the interval of four weeks for convenience of sampling between outpatient appointments, which is longer as intervals of one or two weeks are typical in health research (Polit, 2014).
Additionally, all participants for test-retest study received intensive treatments, i.e. psychotherapy and/or medication; changes in the time interval may reflect true changes in clinical status (Matheson, 2019). Thus, low correlation between treatment interval may in fact suggest usefulness of the NSESSS-PTSD in measuring symptom change over time.
Next, the NSESSS-PTSD had good convergent validity (ρ = .78) with other DSM-5 PTSD measure and moderate correlations with anxiety (ρ = .50) and depression (ρ = .55). Previous studies have reported correlation coefficients ranging from .76 to .84 using the previous DSM-IV version measure (PCL), thus showing similar results as ours. Nevertheless, the NSESSS-PTSD is a DSM-5 criteria-based instrument and should correspond to the DSM-5 PTSD measures as PCL-5 used in our study. Our findings confirmed the convergent validity of the NSESSS-PTSD.
Discriminant validity has been shown in part to have a lower, albeit moderate correlation with anxiety and depression, which often coexist and have overlaps with PTSD symptoms (Spinhoven et al., 2014). This is also in line with the re-positioning of the diagnostic entity of PTSD from anxiety disorder to trauma-and stressor-related disorders in the DSM-5. To further validate the discriminant validity, we additionally compared the scores of the NSESSS-PTSD between PTSD and other psychiatric disorders and found the significantly higher scores in PTSD.
This study has some limitations. First, the sample in our study was drawn from a single psychiatric centre and individuals with acute PTSD (41.7%) and single accident (46.0%) are overrepresented; thus external validity of our findings should be considered with caution. Second, comorbid psychiatric conditions were not investigated in both groups although those with probable PTSD (high self-reported symptoms over cut-off score) had been excluded in other diagnostic group. Possible concurrency of other psychiatric disorders in PTSD group may act as a confounding factor for discrimination of diagnoses. Third, we cannot rule out any influence of subtle cultural nuisance in translation and back-translation processes, although we tried to meet the principles of the cross-cultural adaptation of psychological instruments (Mason, 2005).
For future research, psychometric properties of the Korean version of the NSESSS-PTSD should be investigated in non-clinical population including diagnostic utility for screening. Conducting confirmatory factor analyses and expanding these finding to more heterogeneous trauma population are also needed.
To our knowledge, this is the first study to prove the reliability and validity of the NSESSS-PTSD in clinical population with DSM-5 PTSD. We have provided basic and encouraging psychometric data that show that the NSESSS-PTSD, which is both brief and accessible, can be used as a first-line evaluation tool for PTSD patients.

Disclosure statement
No potential conflict of interest was reported by the author(s).