{'fraction_non_alphanumeric': 0.022879321791532608, 'fraction_numerical': 0.010213982942648485, 'mean_word_length': 5.422433584781896, 'pattern_counts': {'":': 0, '<': 8, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 57, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Utilization and utility of clinical laboratory reports with graphical elements

Background: Graphical reports that contain charts, images, and tables have potential to convey information more effectively than text-based reports; however, studies have not measured how much clinicians value such features. We sought to identify factors that might influence the utilization of reports with graphical elements postulating that this is a surrogate for relative clinical utility of these graphical elements. Materials and Methods: We implemented a pilot project at ARUP laboratories to develop online enhanced laboratory test reports that contained graphical elements. We monitored on-demand clinician access to reports generated for 48 reportable tests over 22 months. We evaluated utilization of reports with graphical elements by clinicians at all institutions that use ARUP as a reference laboratory using descriptive statistics, regression, and meta-analysis tools to evaluate groups of similar test reports. Results: Median download rate by test was 8.6% with high heterogeneity in download rates between tests. Test reports with additional graphical elements were not necessarily downloaded more often than reports without these elements. Recently implemented tests and tests reporting abnormal results were associated with higher download rates (P < 0.01). Higher volume tests were associated with lower download rates (P = 0.03). Conclusions: In select cases graphical information may be clinically useful, particularly for less frequently ordered tests and in on reports of abnormal results. The utilization data presented could be used as a reference point for other laboratories planning on implementing graphical reporting. However, between-test heterogeneity was high and in many cases graphical elements may add little clinical utility, particularly if these merely reinforce information already contained in text based reports.


INTRODUCTION
Several studies have shown that formatting can improve the clarity and clinical utility of pathology reports. [1][2][3] As the complexity of laboratory testing increases there is a growing need for well designed, intuitive reports that convey complex information to clinicians. Graphical trending of serial laboratory measures is well established in laboratory information systems and electronic medical records and its utility has been widely acknowledged; [4][5][6] however, additional graphical elements are being incorporated into both paper and electronic test result reports such as specimen photographs, karyotype images, flow cytometry plots, gel images, charts of ancillary information, links to outside web-based information, and even interactive calculators. [7][8][9] Although graphical reports have potential to convey information more effectively than standard text-based reports, studies have not measured how much clinicians value novel graphical features in laboratory reports. [9,10] Despite the lack of data on their clinical utility; color, charts, images, and tables have become standard for reports provided by small specialty laboratories that circumvent standard interfaces between laboratory information systems and electronic medical records. [11] High-quality, color reports are often used as a selling point, although in some cases reports used as marketing tools overstate the clinical utility of novel testing modalities and actual test performance may be equivalent to or worse than standard testing. [12,13] Local and regional reference laboratories may need to implement higher quality reporting to compete; however, it is not clear if extra images and charts add clinical value, or if they are merely distractions that look nice but contribute little to clinical care. Understanding factors that influence utilization of clinical information is important to laboratories because developing and implementing each individual test report requires time and effort from laboratory directors, managers, and informatics personal in addition to infrastructure development to produce and maintain reporting systems. We sought to identify factors that might influence the utilization of reports with graphical elements beyond simple trending information postulating that this may be a surrogate for relative clinical utility of these graphical elements.

MATERIALS AND METHODS
ARUP laboratories, a national reference laboratory affiliated with the University of Utah, established pilot projects to test an enhanced electronic laboratory reporting system (EELRS) in November 2009. This was implemented as a method of providing clinicians graphical elements that cannot be transmitted or displayed with current interface systems, as a means to integrate information from multiple test reports, and as a location for links to sources of additional information. [8] This system supplements and does not replace the standard mechanisms for delivering test results. Each time a test result is verified within the laboratory information system (LIS), a pdf-format chart is generated and stored on a server. A URL (http:// www.aruplab.com/Lab-Reports) along with chart id number and password are provided as a footnote to the LIS-version test result. This allows all individuals with access to standard test reports to access enhanced versions containing the same test results with graphical elements via web download. Protected health information contained within reports is stored on secure servers, and can only be accessed by those that have access to the password transmitted with reports via current interfaces. Login information linked to specific test reports allowed us to track how often these reports were accessed. We evaluated utilization of enhanced electronic laboratory reports from institutions across the United States. We evaluated instances of access to enhanced reports viewed between November 2009 and September 2011 to determine if there were characteristics of the result presentation or result content that determined how frequently clinicians accessed enhanced reports. This study was reviewed and deemed human subjects exempt by University of Utah Institutional Review Board.  Table 1]. We categorized these into groups of reports on similar tests. There were five genomic microarray tests, six maternal screening tests, and 24 cytogenetic/FISH tests. We further divided the cytogenetic/FISH tests into four subgroups: seven constitutional cytogenetic tests, four constitutional FISH panels, four oncology cytogenetic tests, and nine oncology FISH panels. 13 individual tests did not lend themselves to any obvious groupings. Beyond enabling complex reports that are not interface compatible, electronic delivery of report documents can eliminate the need to mail hard-copy forms to clients. For several tests, specifically cytogenetics tests, enhanced reports eliminated hard-copy mailing. Other enhanced reports were created specifically for online graphical presentation.
We tracked the total number of enhanced reports generated and the reports accessed at the individual report level for all institutions that ordered relevant tests throughout the study period. We only included the initial access of a report in our analysis, so multiple visits to the same report were not part of our analysis. We initially used descriptive statistical methods (median download rates for individual tests and groups) to explore the proportion of generated reports accessed for each individual test and to determine the average access rate for each test group.
It appeared that tests brought online more recently were downloaded more frequently than older tests and that higher volume tests were downloaded less often  than low volume tests, so we used linear regression to test if these factors influenced download rates using months from implementation as one variable and the natural log of the test volume as another continuous variable. We then used the Freeman-Tukey double arcsine transformation method for meta-analysis of proportional data as implemented by the metaprop function in the R-meta package to synthesize data, determine average download rates within group, and evaluate within-group heterogeneity. Data on chart content that could be classified as normal or abnormal was available for 31 of the 48 tests, so we also evaluated whether report content influenced download rate by comparing download rates for reports with abnormal results to overall download rates for these tests.
Enhanced reports for the 13 FISH tests were initially implemented with the intention of adding images of cells with representative marker patterns on the enhanced reports; however, due to technical issues these images were not added to the online reports during the study period. Thus, these reports presented the same information presented in traditional reports transmitted over electronic interfaces with only minor differences in formatting. So, we used two groups of FISH test reports (oncology FISH and constitutional FISH) as controls and compared these groups with other groups of tests. We used 95% confidence intervals from meta-analysis to ascertain whether download rate for groups differed significantly and differed from "negative control" FISH testing download rates.
We evaluated categories of graphics presented in enhanced reports in order to better understand betweentest heterogeneity in download rates. We thought that different types of graphics might explain why certain reports were downloaded more than others. We present several examples of enhanced reports to illustrate examples with high and low download rates.

RESULTS
During the study period starting at the implementation of EELRS on November 9, 2009 and ending September 30, 2011 around 174,170 enhanced reports were generated for the 48 reportable tests evaluated [ Table 1]. The majority of enhanced reports generated were maternal screening reports (66.4%), after this the next most represented group was oncology cytogenetics (6.2% of reports), followed by constitutional cytogenetics reports (5.6%), constitutional fish panel reports (2.5%), oncology fish (2.1%), and genomic microarray reports (1.4%). 15.8% of generated reports were for the 13 individual tests which not lend themselves to any obvious groupings [ Table 1]. For all generated reports 8,947 were downloaded for a total download rate of 5.1%. However, the highest volume tests had low download rates, so the median test download rate was 8.6% and several tests had much higher download rates so the estimated average test download rate was 13.3% [ Table 2].
Both time from implementation and number of total tests ordered were significantly associated with download rate. More recently implemented tests were associated with higher download rates, with download rates declining by 5% to 10% in the first six months after a new report is implemented (P < 0.001 for trend). Higher volume tests were downloaded at a rate lower than low volume tests, with download rates declining by about 2% per order of magnitude of test volume (P = 0.03). Reports indicating abnormal test results were also downloaded on average 8% more often than reports indicating normal results (P < 0.01).
All groups of tests except the Maternal Serum Screening group had significantly higher download rates than the Oncology FISH control group (P < 0.05). However, only the "other" group had a significantly higher download rate than the Constitutional FISH control group (P < 0.05). There was striking overall heterogeneity in download rates between tests (P < 0.001 for heterogeneity). Even when we examined tests in the same group, there was substantial within group heterogeneity (P < 0.001 for within group analysis of heterogeneity for all groups except Oncology Cytogenetics, P = 0.198, see Table 2). This high level of within group heterogeneity tempers any conclusions that might be drawn from between group comparisons. Although download rates differed from test to test and group to group, there were no clear factors beyond those noted above that correlated with download rate. We present and discuss selected examples of representative reports below.

Maternal serum screening reports
Maternal serum screening tests were the highest volume tests with enhanced reports. The enhanced report graphically displays likelihood ratios compared to decision cutoffs [ Figure 1]. This group of reports had among the lowest download rates (4.7% average by meta-analysis, Table 2). A positive or abnormal serum screen was associated with a substantially higher report download rate (13.2%). The most frequently ordered test was the maternal serum screen with AFP, hCG, EST, INH (quad screen) which had an overall download rate of 1.1%, and a download rate of 5.0% for tests reporting positive screening.

FISH reports
PDF reports for FISH testing displayed the standard information that would be transmitted over an electronic interface with standard formatting and a color logo [ Figure 2]. Despite this lack of additional information, there was substantial heterogeneity in download rates for FISH reports (P for heterogeneity < 0.001); with rates varying between 23.4% and 0.5% [Tables 1 and 2]. Reports on FISH for suspected malignancies were downloaded less than FISH reports for constitutional abnormalities (4.6% and 13.8% respectively). Overall oncology FISH reports with abnormal results were downloaded only slightly more often than those with normal results (7.1% and 4.6% respectively), and reports with abnormal constitutional FISH results were downloaded more often than reports with normal constitutional FISH findings (19.2% and 13.8% respectively), but neither of these differences were significant.

Cytogenetic test reports
In the past, images of karyotypes were mailed or faxed to clinicians. Enhanced cytogenetics test reports replaced paper images of karyotypes with PDF reports accessible online [ Figure 3]. Cytogenetic test results for constitutional chromosomal abnormalities were downloaded as often as FISH reports for constitutional abnormalities (14.5% and 13.8% respectively). Although cytogenetic reports with karyotypes for oncology testing were downloaded more often than similar FISH reports for oncology testing (14.7% and 4.6% respectively), download rates for abnormal tests were not significantly higher than overall download rates in this group [ Table 2].

Drug screening reports
Several ungrouped tests were ordered most frequently [ Table 1]. Among those with the highest download rates were drug screening test reports that presented information on metabolic pathways [ Figure 4], BRC-ABL testing reports showing historical trend information, and calculi analysis reports with photographs of the calculi.

DISCUSSION
Understanding factors that influence utilization of graphical elements on laboratory reports can facilitate efficient and effective communication of information. [10] Our study attempts to determine the potential clinical importance of graphical information by using report access as a surrogate for report utility. We believe utilization is a reasonable, objective surrogate for utility; however, we acknowledge that there are major limitations to this assertion. Our assumption could be confounded by several common practices such as office staff downloading all enhanced reports or instructions for access being lost, hidden, or difficult to find because of interface formatting. Furthermore, our study evaluated utilization across many institutions that use AURP as a reference laboratory so much of the heterogeneity in utilization may be due to institution specific practices.
As with any measure of subjective value, utility is context dependent and difficult to measure accurately. However, to our knowledge, there are no other published studies that examine the utility or utilization of graphical report elements in the clinical laboratory setting, beyond reports that evaluate the utility of trending information for numeric tests, suggesting that these measures of utility and utilization could serve as a useful baseline for other laboratories planning on implementing graphical reporting. We have received comments from clinicians that they would like to know what additional benefit is provided in the enhanced report so they can determine whether it is worthwhile to download the report, suggesting that clinicians are thinking about laboratory information in a way that is consistent with utilization being a surrogate for clinical utility.
Our experience with 48 custom build reports, as presented here, shows that custom report access rates were often low, but varied widely from test to test. Despite this heterogeneity, we observed several significant trends in download rates that might shed light on clinician behavior and help laboratory medical directors evaluate how often the addition of a graphical element might be clinically useful. Recently implemented reports were downloaded more often than older reports, reports for more frequently ordered tests were downloaded less than reports for more frequently ordered tests, and reports with abnormal results were downloaded more often than reports with normal results. Each of these observations may illustrate clinical learning patterns. Many clinicians are curious about new information, and once they have ascertained what is in a report they do not take the time to access it again if it is unlikely to add value. However, knowing that the information is available, clinicians may then access it again when they come across an abnormal or unexpected result. This suggests that in most cases implementing graphical reports for novel tests may be better received than adding graphical elements to existing test reports.
Download rates for the majority of individual reports generated were below 10% suggesting that for most clinicians formatting and color images are not important enough to justify the time necessary to download the reports. On the other hand, several reports were accessed at much higher rates suggesting that there are instances where graphical elements on reports present valuable information that may be impossible to convey over current interfaces. Also, the fact that reports associated with positive or abnormal tests are accessed more than negative reports suggests that the need for added information may be situational and case specific.
In conclusion, our analysis suggests that color images and other graphical elements in many cases do not enhance the utility of clinical laboratory reports. However, in some cases additional graphical information may be useful to clinicians, particularly for less frequently ordered tests where the interpretation of text results may not be as apparent from text base reports. The utilization data presented could be used as a reference point for estimating future utilization rates for laboratories planning on implementing graphical reporting. However, substantial observed heterogeneity suggests laboratory directors should not be surprised when graphical elements are not received as they expected. Laboratory directors should also consider carefully the expected clinical benefit of information display methods before redesigning reports in the face of presumed market pressure. In addition, non-laboratory clinicians could be more aware of what information from laboratory testing they are using for clinical decisions and base their choice of laboratory testing primarily on the quality of the information rather than the presentation of this information.