{'fraction_non_alphanumeric': 0.026267785479751914, 'fraction_numerical': 0.003283473184968989, 'mean_word_length': 4.922246220302376, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Discovering and Sharing Sensitive Text

There is an increasing realization of the need to share sensitive information across organizational boundaries; much of it in textual form. While search and information retrieval technology has progressed significantly, it assumes an open environment. Providing these capabilities when the information, and the need for that information, are sensitive and cannot be openly disclosed poses significant research challenges. We give an example showing how such challenges can be addressed, and outline some of the open research challenges in secure sharing of textual data. A key finding of the 9-11 commission was that the U.S. government needs to move from an information access policy of need to know to need to share. Unfortunately, while the report views this as a straightforward (although not simple) change in culture, the details prove more difficult. For example, the report suggests: ... when a report is first created, its data be separated from the sources and methods by which they are obtained. The report should begin with the information in its most sharable, but still meaningful, form. Therefore the maximum number of recipients can access some form of that information. If knowledge of further details becomes important, any user can query further, with access granted or denied according to the rules set for the network-and with queries leaving an audit trail in order to determine who accessed the information. [10] The problem is that it is likely that the sensitive information in the “further details” may be precisely what makes the report relevant, e.g., the name of a source. This is recognized in the Markle foundation report that the 9-11 commission cites as evidence of the technological feasibility: The network, too, must not only enable users to push information to others, it must also enable users to pull it on demand, or at least give each user pointers to a person who can determine whether the user is authorized to access the information. [8] It is clear that techniques are needed to determine if users need information without disclosing sensitive data (unless the need for it exists.) “Need to share” moves this onus from the information consumer to the information producer, but as the need for information (e.g., based on the identity of an individual under investigation) may be as sensitive as the information to be shared, the fundamental problems remain. This is not purely a government problem; the ever increasing public information sharing facilities raise similar issues. Imagine, for example, an employee of Google with an entrepreneurial bent using publicly available resources of Google (search, docs, calendar) to research and develop a new idea