{'fraction_non_alphanumeric': 0.019064911484339538, 'fraction_numerical': 0.0, 'mean_word_length': 5.369942196531792, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
A new process for documenting and checking archival data

The Inter-university Consortium for Political and Social Research (ICPSR) is a data archive and repository for social science data. A major function of the ICPSR is to disseminate the data holdings in a reasonably standard format. For holdings that will be extensively used, additional effort is made to prepare comprehensive, machine-readable documentation, to cross-check the documentation against the data for accuracy and consistency, and to correct or document any inconsistencies discovered.In the past, "cleaning" and documenting the data involved using a number of different computer programs. A great deal of human time was expended on procedural matters: which programs to use, when to use them, and how to coordinate the various stages of the cleaning process. As staff costs and the number of new acquisitions skyrocketed, and computers increased in power and decreased in cost, it became imperative to automate as much as possible the procedure for preparing data for distribution. The GIDO software was developed to meet this need.GIDO is an interactive multi-function program package that guides staff members through the procedure for documenting and cleaning social science data. A cohesive history of the processing operations performed on the data is maintained automatically in machine-readable form. Video terminals are used to display "forms" which the staff fill out with the textual and technical documentation for the data. GIDO immediately verifies the contents of each form and provides an opportunity to make corrections. The forms allow the input of information without requiring knowledge of specialized syntax and conventions. After all documentary materials have been entered, GIDO checks the data for consistency with the original documentation, corrects or flags discrepancies encountered, reformats the data using uniform conventions, and produces machine-readable documentation in a form ready for dissemination.Use of GIDO enables the ICPSR archive to perform its data processing functions more efficiently and at lower cost, thus permitting the organization to meet ever-increasing demands on its resources.