{'fraction_non_alphanumeric': 0.022241231822070145, 'fraction_numerical': 0.000855431993156544, 'mean_word_length': 5.287634408602151, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Multisensory Human Experience: Measurement, Synthesis, and Interaction

Human interaction with the physical world is inherently multisensory. When we place a coffee cup on a table, our eyes help us to locate an empty spot, our skin’s mechanoreceptors help us to determine how hard to grasp the cup, and our ears help us to decide that contact has been made with the surface of the table. Multisensory integration can occur very early in neural information processing. For instance, multisensory cells in the superior colliculus of the brain stem respond if visual and auditory stimuli arise from the same spatial location. In addition, if the visual and auditory stimuli occur simultaneously, the cells will respond more strongly. Thus, to take advantage of these features of human information processing, computer graphics systems and human interfaces must also provide correlated multisensory input, which is tightly synchronized. At Rutgers, we are building a unique facility, the Rutgers haptic, Auditory and Visual Environment (the RAVE), for measuring real human multisensory interaction, constructing multisensory simulation models, and interacting with multisensory environments with low latency. This work builds on our previous work with the UBC Active Measurement facility, ACME [1], which was designed to build multisensory models of small objects. The long term goal our research is to construct virtual environments that respond just like the real world, producing correlated multisensory stimuli when humans interact with them. The RAVE is a highly controlled environment with excellent conditions for multisensory measurement and rendering. Fig. 1 shows a schematic overview of the system. The instrumentation is located inside an acoustically and optically isolated chamber in which humans interact with everyday objects and with each other. As they interact, their motion, deformation, contact forces, and contact sounds can be simultaneously measured using a variety of sensors located on the walls, on the ceiling, on the table top, and also attached to manipulated objects and to a human’s hand. This vast amount of information is digitized and processed in real time on a low latency network of computers. The RAVE is designed to rapidly perform a variety of modeling and simulation tasks, at low cost. For instance: