{'fraction_non_alphanumeric': 0.023977695167286244, 'fraction_numerical': 0.009014869888475836, 'mean_word_length': 4.896438356164383, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 2, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Trust, but verify – Accuracy of clinical commercial radiation Treatment Planning Systems

Computer based Treatment Planning Systems (TPS) are used worldwide to design and calculate treatment plans for treating radiation therapy patients. TPS are generally well designed and thoroughly tested by their developers and local physicists prior to clinical use. However, the wide-reaching impact of their accuracy warrants ongoing vigilance. This work reviews the findings of the Australian national audit system and provides recommendations for checks of TPS. The Australian Clinical Dosimetry Service (ACDS) has designed and implemented a national system of audits, currently in a three year test phase. The Level III audits verify the accuracy of a beam model of a facility's TPS through a comparison of measurements with calculation at selected points in an anthropomorphic phantom. The plans are prescribed by the ACDS and all measurement equipment is brought in for independent onsite measurements. In this first version of audits, plans are comparatively simple, involving asymmetric fields, wedges and inhomogeneities. The ACDS has performed 14 Level III audits to-date. Six audits returned at least one measurement at Action Level, indicating that the measured dose differed more than 3.3% (but less than 5%) from the planned dose. Two audits failed (difference >5%). One fail was caused by a data transmission error coupled with quality assurance (QA) not being performed. The second fail was investigated and reduced to Action Level with the onsite audit team finding phantom setup at treatment a contributing factor. The Action Level results are attributed to small dose calculation deviations within the TPS, which are investigated and corrected by the facilities. Small deviations exist in clinical TPS which can add up and can combine with output variations to result in unacceptable variations. Ongoing checks and independent audits are recommended.


Introduction
Radiation Therapy relies heavily on computer based Treatment Planning Systems (TPS) to design and calculate treatment plans for most patients treated. Commercial TPS are generally well designed and thoroughly tested by their developers. Each system undergoes acceptance testing after installation. Medical Physicists commission and verify the calculations of a TPS before the system is used to plan patient treatments at a Radiation Therapy facility. Medical Physicists also perform ongoing quality assurance (QA) of a TPS and additional checks when updates are installed. However, no check can cover all aspects of the system, and the wide-reaching impact of the accuracy of TPS calculations compels the use of additional QA measures.
The Australian Clinical Dosimetry Service (ACDS) has been created by the Australian federal government as a joint initiative between the Department of Health and Ageing and the Australian Radiation Protection and Nuclear Safety Agency. The ACDS is nearing the end of a three year test period during which it designed and implemented a three level national audit system. Audits are provided free of charge to Radiation Oncology facilities throughout Australia.
[1] This work reviews findings of the ACDS Level III audit and provides recommendations for checks of TPS.

Methods:
The ACDS Level I audit checks the output of the linac for reference conditions. Level II covers part of the treatment chain using a synthetic CT data set as the basis for planning and a planar array as measurement device. The Level III audit represents an end-to-end test that covers the entire chain of procedures a patient experiences at a Radiation Therapy facility from imaging through planning, checks, setup, delivery and record. Radiation Therapists conduct each of the steps in keeping with routine clinical practice so that the audit assesses the actual patient process.
The treatment plans are prescribed by the ACDS and all measurement equipment is brought in for independent onsite measurements by ACDS auditors. In this first version of audits, plans are comparatively simple, involving asymmetric fields, wedges and inhomogeneities. The audit uses an anthropomorphic thorax phantom (IMRT Phantom Model 002LFC CIRS, Norfolk, VA, USA) which contains materials with radiological properties of inhale lung and bone as inhomogeneities (figure 1). The phantom features ten cylindrical ports which, in the default configuration, are filled with solid plugs. For the audit measurements the solid plugs are replaced with plugs that hold Farmer type ionization chambers. Three cases with 6 MV photons are planned and delivered (figure 2). The first case is a measurement near linac reference conditions, which serves as a "sanity" check. A 10 cm x10 cm field is delivered using a 100 cm Source to Surface Distance (SSD) setup with the prescription being in point 1, at 3 cm depth. A second measurement is performed at point 10, which is at 15 cm depth. The chamber is in a water equivalent plug, which is surrounded by a cylindrical shell of bone equivalent material. Cases 2 and 3 have been adopted from an international test case publication [IAEA-TECDOC-1583]. Small changes were made to the field size to allow for a shorter (superiorinferior direction) version of the phantom, which simplified its transport in commercial carriers. Also, in addition to the measurement points suggested in IAEA-TECDOC-1583, additional measurement points were selected. Case 2, which is IAEA-TECDOC-1583 Case 2, uses a lateral wedged field, isocentrically located around point 1, which is also the prescription point. Additional measurement points are in the build-up region behind the lung material (point 4) and at an out of field location in the lung (point 7). Case 3 is based on IAEA-TECDOC-1583 Case 7 with additional measurements taken at points 8 and 10. The measure used to compare plan and measurement is "Variation from ACDS", defined as: (facility planned dose -ACDS measured dose) / ACDS measured dose. Measurement points in low dose areas (outside the field) and points in the lung equivalent material are reported to the facility but not scored (RNS). A measurement point is considered passed at the "Optimal Level" if the "Variation from ACDS" is within 3.3%. It is considered passed at the "Action Level" if the variation is between 3.3% and 5%. The point is considered at the "Outside Tolerance Level" if the variation is outside 5%. The overall audit outcome is equal to the lowest result for an individual measurement point. This approach is in principle based on the ACDS strategy to use 2σ for the limit of the Optimal Level and 3σ for the threshold to Outside Tolerance, with σ being the uncertainty of the measurement. However due to the complexity of the Level III audit, the uncertainty of the measurements could not be assessed with sufficient accuracy. Therefore a clinically acceptable tolerance of 5% [3] was chosen to equal the Outside Tolerance Level (3σ) and the Action Level tolerance was derived from there.
The Level III audit was tested in four field trials between February and April 2012. Radiation Oncology Facilities across Australia with a diverse mix of equipment (TPS, Record and Verify System, Linac) were selected for the field trials. All measurement results in the field trials were at "Pass Optimal Level". Feedback from the facilities was incorporated into the procedures and the Level III audit was deployed clinically in July 2012.

Results
The ACDS has performed 14 Level III audits to-date. Six audits returned at least one measurement at Action Level, indicating that the measured dose differed more than 3.3% (but less than 5%) from the planned dose. Two audits failed (difference exceeded 5%).  Figure 3 shows the results for points 1 and 4 of Case 2. The "Variation from ACDS" is plotted for all audited facilities, including the field trials and one international comparison (random order). The ACDS measured dose is not corrected for Linac output for the analysis, because of the principle of the end to end test used for the Level III audit. The left diagram shows this data. Using the output of the Linac can be helpful in understanding the data and in troubleshooting it. Therefore Linac output, measured by the physics staff of the facility, is also recorded during the audit. Results with the ACDS measurements corrected for the Linac output are shown on the right graph.
Both above mentioned audit fails occurred for Case 2. In one of the fails an incorrect field size had been transmitted to the Record and Verify system coupled with QA not being performed. This result corresponds to the first data point of both plots, where the difference for point 4 was > 100% (not shown). The second fail (see data point with a Point 4 variation of above +6.5% in the graphs) was investigated and reduced to Action Level with the onsite audit team finding phantom setup at treatment a contributing factor.

Discussion and Conclusions
The Level III audit verifies the accuracy of a beam model of a facility's TPS through comparison of measurements with calculation at selected points in an anthropomorphic phantom. As the audit comprises of several steps, any deviations found could be attributed to several sources. Potentially, errors could also cancel each other out. The treatment plans selected for the audit have been chosen to support troubleshooting of any deviations found. Additionally collected data, such as the complete 3D dose information and radiographic images taken to verify phantom setup at the treatment machine, is available to the audit team in this effort. If further clarification is needed, the ACDS Level II audit can be deployed. The Level II audit uses a synthetic CT data set and a 2D array to verify planar dose delivered to a rectilinear phantom made of water-equivalent material with lung-equivalent inhomogeneities. The cases of the Level II audit represent components of the beams of the Level III audit, helping pinpointing the problem within the TPS or delivery system.
The found Action Level and Fail results are attributed to: 1.
Not following provided instructions 2.
Failing to observe internal protocols and QA procedures 3.
Dose calculation deviations in the TPS for wedges, in particular for off axis positions 5.
Dose calculation deviations in the TPS for Reference conditions. All of the above causes can be avoided or rectified in clinical practice. Staff should be trained and supervised to follow instructions and internal protocols and procedures. Tests should be implemented to check TPS and delivery system thoroughly at commissioning and regularly as well as with upgrades and major repairs of any component of the treatment chain.
An independent audit program is warranted.