{'fraction_non_alphanumeric': 0.030100585497673023, 'fraction_numerical': 0.012573187209127758, 'mean_word_length': 4.81260907504363, 'pattern_counts': {'":': 0, '<': 1, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Telephone triage in general practices: A written case scenario study in the Netherlands

Objective General practices increasingly use telephone triage to manage patient flows. During triage, the urgency of the call and required type of care are determined. This study examined the organization and adequacy of telephone triage in general practices in the Netherlands. Design Cross-sectional observational study using a web-based survey among practice assistants including questions on background characteristics and triage organization. Furthermore, practice assistants were asked to assess the required type of care of written case scenarios with varying health problems and levels of urgency. To determine the adequacy of the assessments, a comparison with a reference standard was made. In addition, the association between background characteristics and triage organization and the adequacy of triage was examined. Setting Daytime general practices. Subjects Practice assistants. Main outcome measures Over- and under-estimation, sensitivity, specificity. Results The response rate was 41.1% (n = 973). The required care was assessed adequately in 63.6% of cases, was over-estimated in 19.3%, and under-estimated in 17.1%. The sensitivity of identifying patients with a highly urgent problem was 76.7% and the specificity was 94.0%. The adequacy of the assessments of the required care was higher for more experienced assistants and assistants with fixed daily work meetings with the GP. Triage training, use of a triage tool, and authorization of advice were not associated with adequacy of triage. Conclusion Triage by practice assistants in general practices is efficient (high specificity), but potentially unsafe in highly urgent cases (suboptimal sensitivity). It is important to train practice assistants in identifying highly urgent cases. Key points General practices increasingly use telephone triage to manage patient flows, but little is known about the organization and adequacy of triage in daytime practices. Telephone triage by general practice assistants is efficient, but potentially unsafe in highly urgent cases. The adequacy of triage is higher for more experienced assistants and assistants with fixed daily work meetings with the general practitioner.


Introduction
Triage is the process of determining the level of urgency and type of healthcare required in requests for help: telephone advice, consultation or home visit with a general practitioner (GP), or referral to the emergency department or ambulance care. Telephone triage is a vulnerable part of the care process: the assessment is made without visual input and a balance has to be found between efficiency (giving patients the lowest effective level of care) and safety (identifying patients in need of immediate care). [1,2] Telephone triage is increasingly used to manage workload in primary care. [3] In the Netherlands, the quality of telephone triage in out-of-hours primary care services, GP cooperatives, has received relatively much attention. [4][5][6][7][8] In these large-scale organizations, triagists are qualified after professional training and use decision-support systems. [2] Previous research on the adequacy of urgency assessments by triagists at GP cooperatives and emergency departments showed that the urgency was over-estimated in 1-19% of telephone contacts (inefficient triage) and under-estimated in 7-41% (potentially unsafe triage). [2,[5][6][7][8][9][10] Relevant characteristics of the Dutch healthcare system are listed in Figure 1.
In daytime general practices, telephone triage is performed by practice assistants, who generally have followed an intermediate vocational medical education of three years. A minority of the assistants are educated as a nurse. In contrast to out-ofhours GP cooperatives, most practice assistants in daytime practices have had no additional training as a triagist. To our knowledge little is known about the adequacy of triage assessments in daytime general practices.
To guide policy decisions aiming to optimize patient safety, it is relevant to know what background and organizational factors are associated with the adequacy of triage. Long working experience and extensive triage training of practice assistants are likely to be associated with more adequate triage, as experience and education are known to influence performance in general. The use of a triage decision-support system is also expected to have a positive effect [11]. The NHG Triage Index is a Dutch triage decision support system that is generally used in GP cooperatives and increasingly in general practices [12,13]. Moreover, getting feedback on performance might also be associated with the adequacy of triage. Feedback on performance can be obtained during daily work meetings with the GP and after authorization of contacts in which an assistant gave patients self-care advice.
Our study aim was to investigate the organization of telephone triage in general practices, the adequacy of the assessments of the required type of care (i.e. over-and under-estimation, sensitivity, specificity, and predictive values), and factors (i.e. characteristics of General practitioner (GP) care:

Design
We performed a cross-sectional observational study using a web-based survey among practice assistants workings in daytime primary care.

Population and setting
The survey was conducted among 2369 practice assistants working in general practices in the Netherlands.
Half of the assistants (n ¼ 1184) were asked to assess the required type of care in case scenarios. The contact details of the assistants were obtained from the Dutch Association of Medical Assistants (NVDA). About 30% of all practice assistants in the Netherlands are members of the NVDA. The members are a mix of new and inexperienced assistants and experienced assistants. We excluded members known not to be working (any longer) as practice assistants in primary care.

Questionnaire and case scenarios
The web-based survey contained questions concerning background characteristics of the practice assistants, the organization of triage in the general practice in which they were employed, and case scenarios. The background questions were based on an existing questionnaire for general practices [14] and were checked for completeness and relevance by the management of the NVDA and two practice assistants. Based on an earlier study in emergency departments, [13] on the medical experience of the researchers (including one GP and two general physicians), and on descriptions of health problems in triage systems, 36 case scenarios were written. The case scenarios varied in the degree of urgency and required type of care. The required care categories were based on the NHG Triage Index: [12,13] (i) immediate warning of GP and dispatch of ambulance if necessary, (ii) appointment for urgent consultation with GP within one hour, (iii) appointment for consultation with GP within three hours the same day, (iv) appointment for consultation with GP without time pressure, and (v) telephone advice by assistant.
To be able to evaluate the safety of triage (i.e. potential unsafe triage decisions), we deliberately included more highly urgent case scenarios than actually occur. All case scenarios were presented to an expert panel and field panel to determine the ''reference standard'' regarding the required type of care. The expert panel consisted of three triagists from GP cooperatives and three GPs, who were asked by e-mail to give for each case scenario their assessment of the appropriate type of care to be chosen by a practice assistant. We used triagists from GP cooperatives instead of practice assistants to determine the ''correct answer'' because they are formally trained and registered as telephone triagists and can be seen as experts in telephone triage. The field panel consisted of nine professionals (GPs and triage assistants) from one GP cooperative. The members of the field panel were asked to assess the case scenarios during a workshop and the most common score per case scenario counted as one ''vote'' in determining the reference standard. The six experts on the expert panel all had an individual vote, resulting in a total of seven votes per case scenario. The case scenarios were usable if there was a consensus of over 70%: at least five out of seven votes were for the same type of required care. Eventually, 19 of the 36 case scenarios were included in the study, of which six (32%) were highly urgent (category 1 or 2) and 13 (68%) low urgent (category 3, 4 or 5) (see Appendix 1 for the 19 case scenarios used).

Procedure
The practice assistants received an e-mail with a personal link to a secure website to complete the questionnaire. All assistants received questions regarding their background and the organization of telephone triage in the general practice. Half of the assistants were asked to assess the required type of care of a random set of four or five out of the 19 eligible case scenarios, as if they were telephone calls from patients in their practice. The number of cases per assistant varied, in order to present each case scenario equally often. The other half of the assistants received other questions that are beyond the scope of this article. The data collection took place in April-May 2013 during a period of 21 days, with a reminder on the tenth day.

Data analysis
Descriptive statistics were used to examine the background characteristics of practice assistants and organizational characteristics of the general practices. To calculate the percentages of correct estimation, and under-and over-estimation of the required care for the case scenarios, the assessments of the assistants were compared with the reference standard, including a calculation of the number of categories by which the assistants varied from the reference. In addition, sensitivity, specificity, and predictive values were calculated. Sensitivity was used as a measure for potential unsafe assessments, whereas specificity was used as a measure for inefficient assessments. For this purpose, the required care categories were dichotomized into high urgent care (category 1: GP consultation within one hour and category 2: direct help) and low urgent care (category 3: GP consultation within three hours; category 4: GP consultation without time pressure and category 5: telephone advice by assistant). Healthcare problems that can wait more than one hour (category 3, 4, or 5) were not considered highly urgent, because there is no chance that the patient's condition will soon deteriorate or that delaying treatment will cause serious and irreparable damage. [5,15] To examine which factors related to the adequacy of triage, we calculated the percentage of errors per practice assistant (error rate) in cases where three or more case scenarios had been assessed. Each deviation from the reference standard, no matter how large, counted as an error. The association between characteristics of the practice assistant and triage organization and the error rate was examined using a multiple regression analysis with forced entry in two blocks. In step 1, working experience and triage training were entered in the model. In step 2, use of triage tool, authorization of telephone advice, and work meetings were added. The error rate was the outcome. The data were checked for influential cases, linearity, multicollinearity, homoscedasticity, and normality of errors. Two cases were outliers and had deviating test scores. These influential cases were deleted from the regression analysis.
For each of the predictors we present the unstandardized regression coefficient (B) with 95% confidence interval (95% CI) and standard error (SE) and the standardized coefficient (beta). The analyses were performed using the statistical software package IBM SPSS 20 TM (IBM Corp, Armonk, NY, USA). Results were considered significant at p < 0.05.

Background characteristics practice assistants and general practices
The response rate was 41.1% (n ¼ 973). All responding practice assistants were female and their mean age was 42.4 years. The respondents worked an average of 25.1 hours per week in a general practice and had an average of 13.3 years of experience as a practice assistant. About a third of the respondents (32.5%) had not followed any triage training. The NHG Triage Index was used in most contacts by 4.1% of the respondents. The majority of respondents (80.7%) used it in less than half of the telephone contacts.
In the majority of the general practices where the respondents worked, the GP and the assistants (92.4%) had daily work meetings, either during the (coffee) break (39.7%), at fixed times (37.7%), and/or in between seeing patients (33.3%). Almost all respondents gave telephone advice (99.5%) and 37.1% indicated that the advice is mostly authorized by the GP ( Table 1).

Adequacy of triage
The response rate was 40.0% (n ¼ 474). In 63.6% (1424/ 2240) of cases the assessment of the required care was  Table 2). The sensitivity of the assessments of the required care was 76.7% (550/717) and the specificity was 94.0% (1431/1523). The positive predictive value was 85.7% (550/642); this is higher than the a priori probability of a case requiring highly urgent care in this study (32.0%; 717/2240). The negative predictive value was 89.5%, while the a priori probability of a case requiring low urgent care was 68.0% (1523/2240). Table 3 gives the results of the multiple regression analysis. The error rate in the assessment of required type of care was significantly lower for more experienced assistants (B ¼ -0.003): for each year's increase in experience, the error rate reduced by 0.3%. Visual inspection of the scatterplot indicated that the error rate decreased a little faster in the first years of experience than in later years. However, adding a quadratic term to the model showed this term was not significant.

Association with characteristics of practice assistant and triage organization
The error rate was significantly lower for assistants with fixed daily work meetings with the GP (B ¼ -0.059). Assistants with fixed daily work meetings  had an average error rate of 34% (n ¼ 174) compared with 38% for assistants without a fixed work meeting (n ¼ 295) (not in table). Triage training, use of a triage tool, and authorization of advice did not predict the error rate. The amount of variance explained by the models was low, respectively 1.9% and 3.2%.

Principal findings and interpretation
Practice assistants made an adequate assessment of the required care in 64% of cases, while 19% were over-estimated and 17% under-estimated. The sensitivity was not optimal (77%), which means that the practice assistants missed a significant number of the highly urgent help requests. However, since we overrepresented the number of highly urgent case scenarios in the sample, the potential risk for patient safety in triage of real contacts in general practices will be lower. The specificity was high (94%), so practice assistants worked efficiently, rarely over-estimating low urgent help requests. In other words, they do not often give a ''false alarm''.
The adequacy of the assessments of the required type of care was significantly higher for more experienced assistants and assistants with fixed daily work meetings with the GP. However, the clinical relevance of these findings is low: the absolute decrease in the error rate per year of experience was not substantial and also the difference in error rates between the groups with and without fixed work meetings was small. Triage training, use of a triage tool, and authorization of advice were not associated with the adequacy of triage.
The organization of telephone triage and training of practice assistants in Dutch general practices is not uniform. Only a small proportion of the practice assistants had been trained as qualified triagists and most assistants do not regularly use a triage tool. Almost all practices have daily work meetings between GPs and assistants, but usually not at a fixed time. Nearly all assistants give patients telephone advice by themselves, but this is not regularly authorized by the GP.

Strengths and limitations
This study is one of the first in its field and was conducted among a large group of practice assistants across the Netherlands.
The response rate was mediocre. However, the actual response rate is probably higher, because we do not know if the e-mail addresses of the non-respondents were in use; a number of the invitations to the survey probably did not reach the intended receivers. We included incomplete questionnaires in the analyses to avoid selection bias, as assistants who did not fully complete the questionnaire might have found the cases more difficult to assess than assistants who assessed all cases. However, we set a minimum number of three completed cases to maintain enough (variation in) cases for the calculation of the respondent's error rate.
The error rate was the percentage of errors per practice assistant. It depends on the healthcare problem and time limits for possible treatments as to whether a one-step error in the upper care categories is more important than a one-step error in the lower care categories. However, we decided not to give a different weight to each possible error of each case scenario. We believe that an unweighted error rate over all cases gives the most objective and interpretable results.
Performing a survey with case scenarios instead of using simulated calls enabled us to reach a large group of assistants. Moreover, by using case scenarios instead of studying real patient contacts we could over-represent cases requiring highly urgent care and thus study the safety of triage. Written case scenarios have previously been used in research into triage and have proved to be a useful method. [9,16] A limitation of written case scenarios is that there is no possibility for the practice assistant to ask the patient additional questions. Furthermore, all information is presented at once, assuming that the practice assistant would collect this information. There was also no time limit to assess the cases.
Finally, the 19 included case scenarios had a panel consensus of 70% or more concerning the type of care required. Applying a stricter criterion would have decreased the number of eligible case scenarios. A post hoc analysis for the set of case scenarios with (almost) perfect agreement between the panel members showed similar results, but had less statistical power and less variation in case scenarios.

Comparison with previous studies
The percentage of agreement with the reference standard (64%) in our study was within the range (49-78%) of other studies in out-of-hours primary care that used simulated patients or case scenarios. [5,6,9,17] The percentages of under-and over-estimation were similar to those found by Giesen et al., [5] but different from the results of Derkx et al., [6] who found under-estimation to be 41% (our study 17%) and over-estimation was 1% (our study 19%). This latter finding might be explained by differences in methodology and case description.
Giesen et al. reported the same sensitivity (76%) and specificity (95%) as in our study, when comparing highly urgent and low urgent cases; [5] the other studies did not report these outcomes.
We did not find clinically relevant associations between the background of the practice assistants or triage organization and the adequacy of triage. This is in line with other studies regarding factors affecting triage decisions: neither the clinical background of the triagist [5,9,17] nor the length of experience [9,18] affected triage decisions. However, contrary to our findings, triagists trained in the use of a triage tool were found to have a lower rate of under-estimation of the urgency. [5] Recommendations for practice and research Because of the potential lack of safety of triage in highly urgent cases, practice assistants should be trained in recognizing alarm symptoms in the health problems patients present.
Moreover, in the context of patient safety, standards for authorizing telephone advice by practice assistants in daytime general practice are recommended. Furthermore, the NHG Triage Index is only being used moderately in general practices. Possibly, the assistants are not familiar with it or the triage tool is less appropriate in general practice. Examination of the suitability of the triage decision-support tool in general practice is required.
Finally, we could only explain a very small part of the variation in the error rate between practice assistants. Further studies in this area are recommended, for instance into psychological features and personality characteristics of practice assistants. Assistants might have individual approaches to risk that influence their triage decisions. [

Case: Lisa Martens
Thirty-year-old Lisa Martens phones the practice. She woke up last night because her knee was very painful, which made her decide to take paracetamol. When she got up this morning her knee was red and swollen, while moving about was very painful. She feels awful as well as feverish. Lisa has not been ill in the past few days and her medical history is blank. She only takes microgynon 30. Required care: Appointment for consultation with GP within three hours the same day

Case: Roos Janssen
The partner of Roos Janssen calls the practice; 27-year-old Roos is six weeks pregnant with her first child. Since this morning she has been suffering from severe abdominal pains, accompanied by convulsions and some vaginal blood loss. The pain is now radiating towards the shoulder. She is not running a temperature, but feels very dizzy when she gets up and she is afraid that she might faint. Roos has a blank medical history and is not on any medication.

Case: Arjan Cruijssen
Arjan Cruijssen (aged 25) calls the practice to make an appointment, because he feels absolutely rotten. He is constantly sneezing and he has a runny nose. His eyes are itchy and swollen. The symptoms only seem to be getting worse.
In the past Arjan took pills for this. The problem list mentions that he is allergic to household dust, cats, and dogs. Required care: Telephone advice

Case: Mrs Nelissen
Mrs Nelissen (aged 43) calls because for the past two hours she has been suffering from severe abdominal pain, something she has never had before. The pain is located in the upper part of her belly, radiating to the shoulder blades. Mrs Nelissen feels a need to move around, she feels nauseous but has not vomited yet. Her bowel movement was without problems this morning, she does not know if she is running a fever. Mrs Nelissen is not on any medication and her medical history is blank. Required care: Appointment for urgent consultation with GP within one hour

Case: Femke Jaspers
Femke Jaspers (aged 25) phones the practice: she would very much like to make an appointment for today. Yesterday at work she gave a presentation, and at one point she began to feel unwell. She also suffered from palpitations, dizziness, sweating, and she was no longer able to express herself. Femke is very concerned that something is wrong with her heart and she fears that this might happen again. Her medical history is blank. Last year Femke's father suffered a severe coronary. Required care: appointment for consultation with GP without time pressure

Case: Mrs Aarts
The husband of Mrs Aarts (aged 50) calls: all of a sudden his wife is suffering from a red and very painful right eye. She feels absolutely miserable, nauseous, and has a headache. The vision in her right eye has deteriorated, but she is not suffering from flashes of light. They do not think that something has got into her eye. Mrs Aarts does not have any symptoms in her left eye and she has never experienced this before. She has a blank medical history and is not on any medication. Required care: Appointment for urgent consultation with GP within one hour

Case: Mr Van de Put
Mr Van de Put (aged 45) phones to make an appointment with his GP, because for two days he has been suffering from a red left eye. In the mornings his eyelashes are glued together on account of the pus. His vision is good and the eye is not painful. He states that he does not think that anything has got into his eye. He is bothered by the pus and would like to be rid of it as soon as possible. Mr Van de Put is not on any medication and is in good health.

Case: Kees Gerrits
The girlfriend of 32-year-old Kees Gerrits calls because she is worried. Kees is very absent-minded and she finds it difficult to get through to him. He is fidgety and he has vomited. He keeps moving his limbs, although the movements are not jerky. Kees has experienced a lot of stress at work recently and has had trouble sleeping well. He is not on any medication. His electronic health record states an excess intake of alcohol.
Required care: Appointment for urgent consultation with GP within one hour

Case: Tessa Hendriks
The father of nine-year-old Tessa Hendriks phones, because Tessa has been having abdominal pains since yesterday. She also experiences pain every time she has to urinate. Moreover, she has to urinate a lot. She is not running a fever (T 36.8). Tessa does not feel ill and she has not often been ill in the past. She is not on any medication. Required care: appointment for consultation with GP without time pressure

Case: Mrs De Haan
The husband of Mrs De Haan (aged 79) phones the practice because his wife's left leg has become swollen and painful. There is a warm, red, and painful spot on the leg. Mrs De Haan is not feeling very well; at this stage it is not clear whether she is running a temperature. A few years ago she had the same problem, which was easily cured with a course of antibiotics. Her husband would like to know if it is possible to send a prescription to the chemist. Mrs De Haan suffers from diabetes mellitus type 2 and takes metformin and simvastatin. Required care: Appointment for consultation with GP within three hours the same day

Case: Mrs Van Kesteren
Mrs Van Kesteren (aged 52) phones to make an appointment. For two days she has been suffering from diarrhoea, which varies from being watery to being mushy. She finds the diarrhoea very unpleasant, because she needs to go to the toilet at least 10 times a day. She does not feel ill, although she has lost her appetite somewhat. Mrs Van Kesteren suffers from hypertension, for which she takes hydrochlorothiazide. Required care: Telephone advice