{'fraction_non_alphanumeric': 0.029529635098080573, 'fraction_numerical': 0.007804260704492723, 'mean_word_length': 5.632167832167832, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 4, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
The cognition of comics: What "comics" can tell us about the mind

The cognition of comics: What “comics” can tell us about the mind Neil Cohn (neilcohn@visuallanguagelab.com) Center for Research in Language, University of California, San Diego David Wagner (david.wagner@uis.no) Department of Cultural Studies and Languages, University of Stavanger-Norway Tom Foulsham (foulsham@essex.ac.uk) Department of Psychology, University of Essex John E. Drury (john.drury@stonybrook.edu) Department of Linguistics, Stony Brook University research addressing these questions, along with illustrating the potential that visual narratives hold for the cognitive sciences more generally. Keywords: comics; narrative; visual narrative; grammar; eye-tracking; event-related potentials. Introduction Tracking the eyes across text-image relationships in comic pages Drawing is a fundamental aspect of human communicative expression, and sequential images have been used to convey information since at least cave paintings. However, only until recently has increased research turned towards studying the cognition of visual narratives—especially those found in comics. The most examined question about visual narratives has asked: How do readers make meaning out of a sequence of images? Such an inquiry must balance the understanding of meaning within and across individual images, the generation of inferences from such content, and the engagement of such phenomena within the physical layouts of a page, possibly in multimodal interactions. Recent research has posited that sequential images use a “grammar” that organizes categorical roles into hierarchic constituents. However, rather than a grammar of “syntax” like in sentences, sequential images employ a “narrative grammar” at a discourse level (Cohn, 2013b) to order “panels”—the encapsulated image units of a visual narrative. Such structures are separate from, yet interface with, the rules organizing the composition and navigation of images arranged in a page’s layout (Cohn, 2013a). Ongoing research has demonstrated that, like the relationship between syntax and semantics, this narrative grammar is separate from meaning (Cohn, 2012; Cohn, Paczynski, Jackendoff, Holcomb, & Kuperberg, 2012), and is organized using a constituent structure (Cohn, 2012; Gernsbacher, 1985). In addition, manipulation of these structures yields the same brain responses as comparable paradigms from sentence processing (Cohn, 2012; Cohn et al., 2012). This initial research has raised several questions: 1) How do readers negotiate the interaction of text, images, and page layouts in their comprehension of both sequential images and individual panels? 2) What aspects of individual images might influence the comprehension of sequential images? 3) How does the comprehension of visual narratives overlap with the cognition of other domains? The presentations in this symposium will report on ongoing collaborative While eye-tracking has been a popular method for studying the comprehension of both text and visual scenes, surprisingly few studies have used this technique for studying visual narratives with any theoretically guided significance. Eye-tracking seems especially useful as a technique to study the complex interaction between text and image, and the complex layouts used in whole comic pages. In this talk, David Wagner describes the results of an experiment examining the eye-movements of readers as they progressed through 20 pages of Jeff Smith's Bone. These findings show that readers engage comic pages in systematic ways. These include the dedicated navigation through a page’s layout, the engagement of content within and between panels, and the allocation of time to text and image depending on which carries more semantic weight. Overall, the results of this experiment demonstrate that readers are consistent in their engagement with comic pages across multiple levels of structure. Such results allow us to form insights on the structure and comprehension of visual narratives as well as the ways in which readers engage with graphic media beyond isolated text. Prediction and interpretation in comic reading Beyond the understanding of whole pages, eye-movements can inform us about the structure of attentional processing in sequential image comprehension more specifically. In this talk, Tom Foulsham describes two experiments which begin to investigate how attention is allocated in the reading of wordless visual narratives. In the first experiment, participants’ eye movements changed systematically based on manipulations to the order in which panels appeared. Participants fixated on features important to the meaning of the panel relative to their context within the broader