{'fraction_non_alphanumeric': 0.023514043109079032, 'fraction_numerical': 0.016002612671456563, 'mean_word_length': 5.138276553106213, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The Evolution of Newspaper Digitization at the Washington State Library

In 2005 the Washington State Library began a pilot project to create the Pioneer Newspaper collection, a digital collection of the state’s earliest newspapers available online. The project began with Washington’s first newspaper, The Columbian from Olympia, published in 1852. Other newspapers were also selected from the territorial period, 1852–1889 and represent diverse geographic regions of the state. Approximately 11,000 pages have been digitized and made available as part of this collection. Many of Washington’s early newspapers were filmed in the 1950s, before established microfilming standards and long before the United States Newspaper Project (USNP), a cooperative national effort among the states and the federal government to locate, catalog, and preserve on microfilm newspapers published in the United States. Because of the lack of standards much of this early film is not optimal, with large reduction ratios, low resolution, bleed through, clipped pages, and a lack of consistent collation practices during filming. The Pioneer Newspaper project was meant to be a pilot project that the State Library could use to see what could be accomplished in terms of newspaper digitization with the tools on hand. The original conversion process was done in-house with a Nikon Super CoolScan 9000 ED film scanner. Though a high-end consumer-grade film scanner, the Nikon is not a production level, dedicated roll film scanner so it required some extra processing and care. For example, without roll film capability the operator had to cut and feed the frames in smaller groups, rather than keep them on continuously rolling reels. This requirement significantly slowed down the process. Also, with an optical resolution of 4000 pixels per inch (ppi), only those reels imaged at a reduction ratio of up to 13x would ensure at least 300 ppi output (4000 ppi / 300 ppi = 13.33 reduction ratio). As very little of the film from this time period was imaged at such low reduction ratios or consistent resolution, the scans rarely produced usable Optical Character Recognition (OCR) results. While slow, some impediments to speed became assets to quality. Sometimes referred to as “boutique” scanning, this approach allowed adjustments during scanning to improve tonal contrast, and ultimately the legibility of the words on the film. The image quality of this early film varied enough that it was often impossible to have baseline settings that worked for an entire roll. Frame-by-frame scanning allowed the operator the time to adjust settings more frequently as exposures and other conditions changed throughout the roll. The images were then indexed by a dedicated group of volunteers and made searchable through article level indexing (dates, titles, names, keywords and subjects). The data was then input into an SQL database which currently feeds the search interface for researchers, students, genealogists and other users who access the Pioneer Newspapers collection online.