{'fraction_non_alphanumeric': 0.02781052777585131, 'fraction_numerical': 0.007975587766141896, 'mean_word_length': 5.228941684665227, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
High performance on-demand de-identification of a petabyte-scale medical imaging data lake

With the increase in Artificial Intelligence driven approaches, researchers are requesting unprecedented volumes of medical imaging data which far exceed the capacity of traditional on-premise client-server approaches for making the data research analysis-ready. We are making available a flexible solution for on-demand de-identification that combines the use of mature software technologies with modern cloud-based distributed computing techniques to enable faster turnaround in medical imaging research. The solution is part of a broader platform that supports a secure high performance clinical data science platform.


Background:
In order to support secondary use of data generated during the course of clinical care to advance the science and practice of medicine, Stanford Medicine is developing its next generation clinical data platform for research and related data science ecosystem [Datta2020]. This platform is an ecosystem consisting of a data lake (STanford medicine Research data Repository or STARR), a suite of analytical data warehouses and cohort building applications and a secure data science facility. A sampling of medical imaging research shows that researchers have been successful at Stanford [Varma2019, Patel2019, Irvin2019, Park2019, Dunnmon2019, Bien2018, Banerjee2018a, Banerjee2018b, Rajpurkar2018, Esteva2017] and elsewhere [Peng2020, Zhang2020, Gulshan2016], in applying AI approaches to imaging data to derive scientific insights. In parallel, national experts are charting the path of AI in medical imaging [Allen2019]. We expect AI research in imaging to become ubiquitous resulting in even greater demand for imaging data at Academic Medical Centers like ours. As a result, our new ecosystem has a strong focus on supporting our imaging community.
Modern hospital environments are generating hundreds of terabytes of new imaging data per year. At Stanford, the radiology imaging data is growing at the rate of 450 terabytes annually representing various different modalities ( Figure 1)  Traditional approaches for de-identifying radiology medical images involve the use of one or more software applications directly connected to a clinical imaging archive. When imaging studies are identified for research use, the anonymization software retrieves and de-identifies the images from the clinical archive. This approach quickly breaks down in scale when multiple researchers are requesting billions of images and hundreds of terabytes of data. Recently cloud computing providers have begun offering API services for text and medical image de-identification, however they were incompatible with our requirement for reproducibility and control over the de-identification process. Vendor API's operate as a "black box" that changes without notice, affecting reproducibility. These APIs are in early stages, and can lack functionality like complete control over the replacement of DICOM attributes. These APIs are reassessed periodically as they mature and provide greater flexibility, however their proprietary nature, lack of change control and vendor lock-in will always present a major challenge for implementation.
Another challenge to delivering medical image data for research purposes is that Stanford University Privacy Office (UPO) determines that algorithmically de-identified DICOM images have the potential for leaking incidental Protected Health Information (PHI). In order to confirm that the de-identified data has no incidental PHI, UPO requires the algorithmic output to be scanned by humans. While such expert determination is used for broader data sharing (e.g., https://aimi.stanford.edu/research/public-datasets ), it is not feasible to scan each image by eye for internal research use with hundreds of terabytes of data. To address security and patient privacy at Stanford, we have built a secure data science facility [Datta2020] to support downstream data processing on petascale data. To support research needs including the use of modern deep learning approaches, we present a new de-identification system that combines the reliability of existing de-identification software with the scale of cloud-based distributed computing. We also decouple the data processing system from the clinical imaging archive to reduce any adverse impact on clinical operations. The new imaging pipeline acquires data on-demand from the clinical radiology archive, algorithmically de-identifies the data in parallel and delivers to the researcher's secure data science workspace.

Method:
When de-identifying patient data, patient identifiers must be replaced by unique, anonymized codes, a process known as pseudonymization [Noumeir2007]. Our de-identification pipelines support two separate research stages. First one is pre-IRB non-human subject research. The de-identification aggressively strips everything that may have HIPAA identifiers. In this mode, the anonymized codes generated for non-human subject research can never be reversed and linked to identified patient data. Second is for post-IRB study, the researcher can request de-identified images which meet the HIPAA minimum necessary for the study and can additionally request links between the anonymized images and the original patient identifiers.
We have developed an on-demand highly scalable de-identification solution in lieu of a single archive of permanently de-identified images to support the two distinct use cases. The on-demand pipeline also allows us to make continuous improvements to de-identification quality (i.e., reduce PHI leak) unlike a single permanently de-identified image archive. We will show in the Results section that our cost of de-identification is extremely low to support this on-demand workflow.
Specifically, we have written a distributed software application (STARR-Radiology aka STARR-Radio) which a) forwards fully-identified DICOM image data from on-premise clinical systems to STanford medicine Research data Repository (aka STARR data lake), specifically, to an encrypted and distributed cloud object storage service, b) listens for de-identification requests using a publish/subscribe messaging model, c) instantiates an appropriate number of de-identification compute instances based on the size of the message queue, and d) delivers the de-identified DICOM images to an object store accessible to the researcher for downstream data science. Although CTP is designed for installation as a client-server application, it's highly extensible nature allowed us to extract only the DICOM filtering and anonymizing components for our use. Additionally, MIRC CTP uses the dcm4che [Dcm4che2019] libraries, which provide support for acquiring, manipulating, and transmitting DICOM files. These two software components were integrated with the cloud platform that underlies the STARR data lake.
A central database and server component was created to store workflow information relevant to the lifetime of a de-identification request. When a request for images has IRB approval, an entry for the approved study is inserted in the database along with associated accession numbers (representing specific imaging studies for that study). The accession numbers are first validated as eligible for research, and if validated a new anonymized accession number, patient MRN, and randomized date jitter specific to the specific research study are created. These identifiers, combined with anonymization rules appropriate for the associated IRB, are transmitted to a central messaging queue. An auto-scaling compute pool which is subscribed to the messaging queue creates an appropriate number of compute instances based on the total number of outstanding messages in the queue and the expected delivery window. Each worker retrieves messages from the queue, downloads and de-identifies the DICOM files using the MIRC anonymization engine, and uploads the de-identified images to an object store accessible to the researcher. Compute instances are deleted once the message queue is empty, and a manifest file is created which indicates the transformations applied to each image, along with success or failure states.
The greatest challenge encountered was creating and validating rules for reliably de-identifying DICOM imaging studies. While the MIRC anonymizer includes a set of rules intended as a starting point for further development, they are not acceptable in their default state [Aryanto2015]. The DICOM anonymization process (Figure 2a) is executed in three discrete stages: filtering, scrubbing, and anonymization. The filtering stage accepts or discards the image based on metadata rules. For example, the filtering stage removed encapsulated PDF reports, scanned intake forms, outside study alerts etc. If the image is not discarded, it is then passed to the scrubbing stage, which removes rectangular regions from the image which are known to contain PHI. Here, the regions of the image containing PHI are replaced by black pixels and recompressed using the JPEG Lossless syntax. The rules used for pixel blanking can be found on GitHub ( https://github.com/susom/mirc-ctp/blob/master/stanford-scrubber.script ). The final stage removes or replaces image metadata which is known to contain PHI. filtering, anonymization and pixel scrubbing rules were created using an iterative process. Images for each modality are categorized by make, model and image resolution. A sample for each combination is executed against the pipeline, which is modified as necessary to remove PHI. After every iteration the image is added to the regression suite, which is then executed to ensure new rules do not break existing ones.
During development regression tests were created as new rules were added to ensure new rules did not break existing rules. The regression tests are written using the Cucumber testing framework ( https://cucumber.io/ ) and are human readable ( Figure  2b). The maturity of the CTP software itself presented unique challenges. While CTP is written in the cross-platform Java language, it requires libraries which utilize the Java Native Interface, and only run on x86 or SPARC platforms. These libraries are no longer supported by Oracle, and posed problems during development, requiring the use of Docker containerization as a compatibility layer. Additionally, the CTP software uses older Java technologies, such as file references instead of stream buffers. This requires the de-identification workers to store images in block storage attached to the OS, instead of networked direct data streams. The negative performance impact of block storage was mitigated by mounting system RAM as block storage (aka ramdisk) which is used as ephemeral storage during de-identification.

Results:
In this section, we present the performance achieved for the de-identification pipeline. A selection of random studies (containing one or more images) for three different imaging modalities were passed through the de-identification pipeline consisting of 8 compute instances running Google Container Optimized OS, each with 32 CPUs and 208GB of RAM. The de-identification application is executed within docker, using a 'tmpfs' mount as a high-performance ramdisk. Anonymized instances which have not been filtered are uploaded to Google Cloud Storage and deleted from ramdisk. The low cost of de-identification allows us to store a single copy of the identified image and provide on-demand de-identification as needed.

Studies / Modality
Discussion: During the iterative development process it was found that certain types of images needed to be excluded completely as the probability for PHI leakage was high. Such examples include: 1. Analog x-ray film which has been digitized by a film scanner. For example, all images with manufacturer "Vidar" are filtered by the pipeline. Scanned analog x-rays are highly problematic as PHI may be present at any location on the film (in the form of a stick or hand-written notes) and the image may be the incorrect orientation, eg. flipped or upside-down. 2. DICOM files containing: a. Encapsulated PDF documents b. Structured Report (SR) documents c. Presentation state objects d. Uncommon modality attributes (eg. RAW) e. Secondary capture objects* f. Burned-in annotation is set to "Yes"* g. ConversionType attribute is an empty-string h. ImageType attribute contains DERIVED or SECONDARY* 3. DICOM from video-capture devices * May be bypassed by specific whitelisting rules based on other attributes.
Ultrasound devices in particular were found to be the most complex for reliable de-identification, followed by digital x-ray. While other modalities may occasionally contain PHI in the pixel data (for example CT radiation dose exposure screens), ultrasound almost always contains PHI in the pixel data. It is common for multiple regions within a single image to have pixel data that needed to be removed, which may be different even within the same make and model of an ultrasound device depending on the resolution of the image. For this reason ultrasound devices are only supported via a whitelist: if there is no rule for a specific make, model, and image resolution for an ultrasound image, the image is filtered. It should be noted that certain data types, such as head images, even when de-identified fully (i.e., no incidental PHI) could be combined with other publicly available data to re-identify the original patient [Schwarz2019]. There are advanced techniques for skull stripping [Kalavathi2016] that should be applied if research requires public sharing of head imaging data.

Future work:
Future work focuses on several different areas. Firstly, we are building a DICOM metadata store using Google BigQuery [Datta2020] that will host metadata from raw identified DICOM images. A pre-IRB de-identified version of this store will be made accessible for cohort development. Secondly, we are leveraging our clinical text mining framework [Datta2020] to augment cohort development e.g. text mining radiology reports. Thirdly, we hope to integrate OCR and other machine learning approaches to improve image de-identification. Finally, we plan to generate a pre-IRB non-human subject dataset containing a stratified subset of imaging data that is linkable to the pre-IRB OMOP clinical data [Datta2020] for research acceleration.