{'fraction_non_alphanumeric': 0.018273471959672338, 'fraction_numerical': 0.003780718336483932, 'mean_word_length': 5.481632653061224, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Design of a microprogrammed lexical microprocessor

A lexical processor accepts a string of characters and delivers a string of symbols. This paper presents the design of a vertically microprogrammed processor, specially for lexical processing of a subset of Algol. The design includes the processor organization, the microprogram control organization, and the algorithm for the microsubroutines. This microprogrammed lexical processor could be implemented as a microprocessor in a multi-microprocessor system for high-level languages.
 The advent of microprocessors in 1971 has created a new era in computer technology. Since then, there are over 20 microprocessors that have become available, and there are numerous applications that have been made possible by these microprocessors.
 However, these microprocessors are essentially arithmetic processors. As a result, conventional software such as compilers, assemblers, linkage editors and loaders are required. The advancement in semiconductor technology has not been equally matched by the advancement of computer architecture and computer languages.
 In order to make better use of microprocessors, it is highly desirable that the layers of software be simplified or eliminated, and that high-level languages be designed with both the technology and the users in the proper perspective. This paper presents the design of a lexical processor, which is vertically microprogrammed for easier programming. This processor could be implemented as a microprocessor to be a member of a multi-microprocessor system for high-level languages.