{'fraction_non_alphanumeric': 0.03230012342888931, 'fraction_numerical': 0.010847834779131826, 'mean_word_length': 5.266230936819172, 'pattern_counts': {'":': 1, '<': 2, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Beyond Being Real: A Sensorimotor Control Perspective on Interactions in Virtual Reality

We can create Virtual Reality (VR) interactions that have no equivalent in the real world by remapping spacetime or altering users' body representation, such as stretching the user's virtual arm for manipulation of distant objects or scaling up the user's avatar to enable rapid locomotion. Prior research has leveraged such approaches, what we call beyond-real techniques, to make interactions in VR more practical, efficient, ergonomic, and accessible. We present a survey categorizing prior movement-based VR interaction literature as reality-based, illusory, or beyond-real interactions. We survey relevant conferences (CHI, IEEE VR, VRST, UIST, and DIS) while focusing on selection, manipulation, locomotion, and navigation in VR. For beyond-real interactions, we describe the transformations that have been used by prior works to create novel remappings. We discuss open research questions through the lens of the human sensorimotor control system and highlight challenges that need to be addressed for effective utilization of beyond-real interactions in future VR applications, including plausibility, control, long-term adaptation, and individual differences.

: Movement-based VR interactions showcased on a continuum from high to low degree of verity (meaning true to life): reality-based, illusory, and beyond-real interactions, with sensory mismatch created through body-centered (egocentric) or world-centered (allocentric) warping.

INTRODUCTION
The idea of leveraging VR beyond the replication of reality dates back to the early days of this technology. In a 1965 article, "The Ultimate Display," Ivan Sutherland proposed that "there is no reason why the objects displayed by a computer have to follow the ordinary rules of physical reality" and that "such a display could literally be the Wonderland into which Alice walked" [171]. Over the years, other researchers have shared a similar perspective about VR interaction design and have highlighted potential benefits of designing VR interactions beyond reality, including improving human performance [109] and making interactions more efficient, ergonomic, and accessible [72]. For example, the Go-Go interaction is an arm-extension technique that stretches the user's arm during reach, enabling them to grasp and manipulate distant objects [122]. These interactions are designed not to overcome the limitations of VR technology, but to overcome the limitations of our reality.
In "Beyond Being There" (1992), Hollan and Stornetta made a parallel argument in response to telecommunication and computer supported collaborative work research and development at the time.

arXiv:2204.08566v1 [cs.HC] 18 Apr 2022
They argued that when comparing telecommunication to face-toface communication "the imitation will never be as good as the real thing. This is true by definition if one is strict in using the old medium as the standard of measurement. . . requiring one medium to imitate the other inevitably pits strengths of the old medium against weaknesses of the new" [69]. They presented a framework around needs, media, and mechanisms, "to ask the question: what's wrong with (physically proximate) reality?" and explore new mechanisms that leverage the strengths of the new medium to meet our needs.
Towards similar goals, we believe there is a need to more systematically investigate what we call beyond-real VR interactions: movement-based interactions that are not possible in the real world. By "real world" we do not imply that there exists an objective physical world independent of the user's subjective mental world. Instead we are referring to what others have called the "actual world": what can be experienced by the human sensory system without the use of technology [87]. VR enables full-body interactions with digital content where users can move and act in the virtual world. These interactions do not have to resemble users' movements in the real world, as VR presents an opportunity to construct imaginative interactions by remapping users' movements and altering the resulting sensory feedback. Due to the plasticity of the human sensorimotor system, users have the ability to learn and perform motor tasks under new remappings [117]. As HCI practitioners, we are interested in exploring VR interactions that are usable and lead to such motor skill acquisition given novel dynamics. Thus, we propose describing VR interactions through the lens of the sensorimotor system, as transformations applied to tracking and sensing inputs from the real world. We believe such perspective highlights considerations around action and perception that are key for understanding the potential, as well as challenges, of beyond-real interactions.
In this paper, we present a framework based on sensorimotor control for categorizing virtual reality interactions as reality-based, illusory, or beyond-real, as shown in Figure 1. We further utilize this framing for describing beyond-real interactions as a set of transformations applied to real-world input. We apply the framework to a survey of VR interactions and systematically identify and categorize beyond-real interactions based on their underlying transformations. This survey provides an overview of more than 30 years of beyond-real interaction techniques and identifies key types of transformations that have been explored. Finally, we use the lens of sensorimotor control to map out open research questions central to better understanding the effective use of beyond-real interactions.
In this work, we contribute: • Beyond being real, a framework based on the human sensorimotor control to describe movement-based VR interactions as transformations applied to input from the real world. • A literature survey to categorize existing VR interactions (at CHI, IEEE VR, VRST, UIST, and DIS conferences) as realitybased, illusory, or beyond-real, apply the framework to isolate beyond-real transformations in these selected works, and describe transformation categories that have been explored by prior research for creating beyond-real interactions. • A discussion of challenges and open research questions that require further investigation of beyond-real VR interaction design through the lens of sensory integration.

BACKGROUND
In VR users can move in virtual spaces and perform full-body interactions. We focus on these movement-based VR interactions [56] and action execution in VR (p. 40) [114]. We approach interaction from a control and optimal behavior perspective [70], and study interaction techniques, such as selection, manipulation, and locomotion, that require motor performance [22]. In this section, we first present our categorization of VR interactions as either reality-based, illusory, or beyond-real. We then provide a high-level background on the human sensorimotor system and control theory. Through this lens, we describe how VR interactions can be thought of as transformations that directly map or remap the user's movements in the real world to renderings in the virtual world. This insight is central to our work and we believe to research that follows. Here, we use this framing to differentiate between reality-based, illusory, and beyond-real interactions based on whether the transformation applies a remapping and whether that remapping is noticeable by users.
We begin by situating these three categories within the context of prior research. Thurman and Mattoon [176] describe different dimensions of VR, including what they call the verity, meaning true to life, dimension. They then use verity to denote "a continuum of simulation experiences that range from recreations of the physical world as we know it to depictions of abstract ideas which have no physical counterparts." Along this continuum, VR interactions range from interactions with high degree of verity that follow natural laws of the real world to interactions with low degree of verity that follow novel, original laws. Similarly, Slater and Usoh discuss interactions on a spectrum from the mundane to the magical [155] which map closely to the verity continuum.
Our three categories of movement-based VR interactions range from high to low degree of verity: (1) reality-based interactions that match the user's real-world movements, (2) illusory interactions that create remappings between the user's movements and the virtual renderings that remain unnoticed by users, and (3) beyondreal interactions that create novel remappings between the user's movements and the renderings in the virtual world (see Figure 1).

Reality-Based Interactions
Highly realistic VR environments that seek to replicate our realworld experiences are used for practical applications, such as training [68], exposure therapy for treating phobias [119,131], and post-traumatic stress disorders [70,132]. These environments also facilitate user interactions that closely resemble interactions in the real world. Jacob et al. proposed the notion of Reality-Based Interactions (RBI) to describe such interactions that employ themes of reality and leverage users' pre-existing knowledge of the everyday, in VR and more broadly [72]. They highlight the benefits of RBI, including accelerated learning, reduced mental effort, facilitated improvisation, and improved performance, particularly in situations involving information overload, time pressure, or stress. They also note that despite the advantages of RBI, designers may explicitly give up realism to gain desired qualities by allowing users to perform many tasks within an application (expressive power) or across different applications (versatility) and to do so rapidly (efficiency), without fatigue or risk of physical injury (ergonomics), and using a varied range of abilities (accessibility). In this work, we focus on VR interactions in which designers explicitly give up realism by creating novel remappings between user inputs and the rendered outputs in VR to overcome the limitations of our experiences in the real world. However, it should be noted that there are many advantages associated with reality-based interactions, and extending interactions beyond reality is not always beneficial, nor is it suitable for all VR applications.

Illusory Interactions
As Lanier highlights, our most important canvas in VR is the user's sensorimotor loop [84]. This technology offers a unique opportunity for manipulating senses, as arbitrary mappings can be created between the user's movements and the rendering of their virtual body. Movement-based VR illusions are remappings that result in a subtle mismatch between the sensory feedback from the virtual system and the sensory feedback from the real world; however, the discrepancy is below the user's perceptual thresholds and is resolved such that the sensory feedback aligns with what the user expects (i.e., the predictions of their internal model). For example, slightly extending the length of the user's arm (Figure 1b) or slightly misplacing the user's hand ( Figure 1c) in VR are illusions that will go unnoticed by users. Gonzalez-Franco and Lanier present a model of illusions in VR that describes these processes in more detail [59].
Illusions have been explored by researchers to redirect the user's hand while tracing surfaces [1,81,212] or reaching in midair [11,30,57] to provide an improved perceived haptic sensation and overcome the current limitations of VR technology. In these visuohaptic illusions the mismatch between the visual and proprioceptive feedback is resolved by visual dominance [64]. Another example of movement-based VR illusions is redirected walking where the rotational movement of the user's head during turns is remapped to a different rotational angle in VR such that their perceived walking path is altered. When studying VR illusions, researchers are concerned with identifying users' perceptual thresholds to ensure that the illusion remains unnoticed [1,163]. While these illusory interactions are important for improving the perception of realistic (high degree of verity) VR environments, prior research has shown that our cognitive system can adjust to repeated exposure to conflicting stimuli [20]; thus, there are opportunities for exploration of overt forms of such remapping techniques that go beyond reality.

Beyond-Real Interactions
For decades, scholars have emphasized the need for further exploration of virtual experiences beyond replication of reality. In 2003, Schneiderman highlighted that there are many opportunities for enhancing 3D interfaces "if designers go beyond the goal of mimicking 3D reality" [144]. In 2005, Casati et al. argued that efforts should be directed towards "creation of virtual perceptual objects that have no equivalent in the hard reality" [27]. Gaggioli suggested, in Human Computer Confluence, that "the possible uses of VR range from the simulation of plausible possible worlds and possible selves to the simulation of realities that break the laws of nature and even of logic" and that VR can provide "a subjective window of presence into unactualized but possible worlds" [54]. Bailenson, in Experience on Demand, proposed that the reality bending properties of VR allow us to create experiences "unbound by the law of the real world, to do impossible things in virtual settings" and that "VR is perfect for things you couldn't do in the real world" [12].
From an interaction design perspective, while beyond-real VR interactions can offer benefits, such as making movement-based input more efficient [109] and ergonomic [72], they create noticeable incongruencies between the sensory feedback from the real world and the virtual environment. This sensory mismatch has important implications for designing usable beyond-real interactions that people can learn, adapt to, and feel in control of. Therefore, in our work, we carefully consider the human sensorimotor system and approach interaction from a control and optimal behavior perspective [70]. Under this assumption, the human is a goal-directed control system that receives feedback about the state of the world through virtual renderings and behaves so as to change the control signal towards a desired output. The human pursues this goal optimally and adapts to the constraints of the virtual environment. In the next section, we present a simplified model of the human sensorimotor system and optimal control theory that we believe is key in the discussion of beyond-real interactions. We use this theoretical lens throughout the paper to describe beyond-real VR interactions as transformations applied to real-world input. We conduct a survey of beyond-real transformations that have been utilized by prior research and highlight open research questions that remain in the design and evaluation of usable beyond-real VR interactions.

Sensorimotor System and Control Theory
Human performance may be modelled at various levels of behavior: skill-based, rule-based, and knowledge-based behaviors [124]. Optimal Feedback Control (OFC) theory focuses on skill-based behavior (e.g., catching a ball) and has been used to predict how the human brain plans and controls movement [140] by studying the link between high-level goals and real-time sensorimotor control strategies [178]. This theory suggests that the Central Nervous System (CNS) acts as a feedback controller, continuously converting sensory input into motor output [182] and it does so optimally, based on a performance metric, such as obtaining minimal uncertainty in the state estimate [183]. Researchers have also proposed using a Mini-Max Feedback Control (MMFC) model, an extension of the OFC model that minimizes energy consumption under the assumption of worst-case uncertainty [182]. Figure 2 shows how the CNS interacts with the body and the VR system during movement-based interactions. In this diagram blocks represent key components, and arrows denote the flow of control signals, clockwise from the top left. The optimal controller outputs motor commands based on the discrepancy between the desired and estimated states [200]. These motor commands lead to movements in the real world that are then subject to body dynamics and the effects of the environment, such as external forces. The VR system includes sensing and tracking devices that capture the user's movements. Movement-based VR interactions can be thought of as transformations applied to these signals captured from the real world. In reality-based interactions the transformations create a 1:1 mapping to the VR renderings; in illusory interactions the transformations create subtle remappings that are unnoticed by users; and in beyond-real interactions the transformations create novel remappings. The human sensory apparatus receives sensory feedback from both the real world and the virtual system (shown in orange). The state estimator receives the sensory feedback through the sensory apparatus as well as an efference copy of the original motor signal [18]. While noise is present at all stages [183], we only show discrete examples in this diagram. Figure 3 shows the subcomponents of the CNS and the flow of control signals. The feedback controller outputs motor commands based on the discrepancy between the desired and estimated states, which is then combined with the output of an adaptive inverse model [200]. An efferent copy of motor signals is sent to a forward model that predicts the result of motor commands [18]. The forward and inverse models are collectively referred to as the internal model and capture information about the context and the properties of the sensorimotor system.

Sensory integration.
Multisensory integration is a complex process that modifies the original signal based on low-level sensory information, top-down influences of the internal model, and a range of cognitive factors; therefore, it is perhaps more accurately described as multisensory interaction [172]. This interaction is taskdependent and may be affected by the modality of the stimulus as well as the information content of the feedback [157]. Multisensory processing is also influenced by attention [172] and human emotional responses to stimuli [139]. Finally, the central nervous system minimizes uncertainty by refining sensory signals based on prior knowledge and memory [182]. Redundancy in the sensorimotor system ensures robustness [46] such that elimination of a feedback has minor effects on motor behavior [40]; however, perturbations of the same signal may significantly alter movement [35]. For example, while reaching without sight results in minor errors, visual distortions have been shown to lead to drastic compensatory movements [134,137]. Therefore, sensory integration predominantly addresses unexpected changes based on prediction errors [172]. Note that OFC theory is concerned with errors that are referred to as slips, and not mistakes that arise from incorrect intentions (p. 414) [115].

Learning and adaptation.
Prediction errors drive simultaneous perceptual and motor learning [46,117]. While both the forward and inverse models are adapted [200], it has been shown that prediction learning precedes the learning of new control policies [46]. Beyond adaptation to perturbations, humans can learn to synthesize movement under entirely novel dynamics [61]. An example of sensorimotor learning is prism adaptation in which an individual performs perceptual motor tasks while wearing goggles that shift their visual field [128]. An interesting characteristic is that the effects of adaptation after removing the goggles, known as aftereffects, are not global, and only result in a systematic movement bias for the specific, practiced task [6].
We use this theoretical background to first present a descriptive framework for beyond-real interactions and then discuss open research questions and challenges that remain in the context of sensory integration and adaptation.

THE BEYOND BEING REAL FRAMEWORK
We present beyond being real, a framework using a sensorimotor control perspective for investigating movement-based VR interactions that have no equivalent in the real world. The framework, shown in Figure 4, provides a scaffolding for describing beyond-real interactions in three stages, by describing the sensing and tracking data in the real world, the set of transformations applied to the real-world input for creating novel remappings in VR, and the VR renderings that provide signifiers and feedback to improve the usability of beyond-real interactions. In this section, to highlight this descriptive power, we return to the example of the Go-Go interaction technique [122]. Note that while we provide a highlevel description of the Go-Go interaction here, the framework is better suited to describing a specific implementation of the interaction technique that includes more details. For a more in-depth walk-through example, please refer to Appendix B.

Real-World Sensing and Tracking
The virtual system has limited information about the state of the real world. For example, in most commercial VR systems the position of the headset is known; however, the system often does not have direct knowledge of the user's body pose. The real-world information often describes the environment state (e.g., room dimensions, obstacles), users' physical state (e.g., head position, hand pose), and their cognitive state (e.g., attention, workload). In the first stage of the framework, we identify what sensing and tracking devices are available, what limitations they have (e.g., range, rate, accuracy), and what real-world data from those devices is being utilized by the virtual system. In the Go-Go interaction technique, the inputs from the real world are the user's physical arm length ( ) and the user's real hand position ( ì ) in the user's egocentric frame of reference. In this case, the origin is the user's chest position, approximately determined based on the position of the VR headset.

Beyond-Real Transformations
The input passed on to the virtual system is then either disregarded, mapped directly, or remapped. Remappings can be fixed throughout the interaction or may dynamically change based on users' actions captured by the input data [85]. These mappings can be described as transformations that take input from the real world and modify the space, the user's body representation, or time parameters.
In the second stage of the framework, we focus on the goal of the beyond-real interaction (i.e., interaction task). To describe the sets of transformations applied to the real-world data, we identify what parameters are modified as a result of the remappings (space, body, or time) and what the mapping type is (direct, fixed, or dynamic). In the Go-Go interaction, the beyond-real transformation is a dynamic remapping that modifies the user's body representation for manipulation of distant objects. More specifically, the user's virtual hand ( ì ) is extended during the last 1/3 of their reach range, for a given coefficient k (0 < < 1): While our framework focuses on VR interactions, such transformations are not unique to VR and may be used to describe remappings between movement-based user inputs and outputs of computing systems more broadly. For example, by modifying the control/display ratio, the movements of the mouse can be remapped to the movements of the cursor. However, for beyond-real VR interactions specifying these transformations can be especially useful for determining the sensory mismatches that users experience as a result of the remapping, which we discuss in more detail in section 5.

VR Renderings and Remapping Signifiers
The virtual system renders information through output devices, such as the VR headset and headphones. While some renderings are mapped to the input (i.e., they directly result from user actions), others are independent of the real-world input and the applied transformations. Independent renderings may communicate to users what mappings exist, prior to the execution of actions. These signifiers may be egocentric (e.g., visible features of the body representation) or exocentric (e.g., features in the environment or specialized objects). The concept of "User Representation" defined by Seinfeld et al. [142] is closely related to egocentric signifiers. More specifically, User Representations are virtual elements that extend users' physical bodies and they "may have signifiers that communicate the actions they support. " In the third stage of the framework, we identify the aspects of the renderings that are independent of the real world and communicate the remapping to users (invisible, egocentric, or exocentric signifiers). Remapping signifiers have important implications for learnability and adaptation to novel remappings (see discussion in section 5). In the Go-Go interaction, there are no visible signifiers that communicate the remapping to users independent of the user's actions. Therefore, users can only discover the remapping after they extend their arm more than 2/3 of their arm length.
In the following section, we present a survey of beyond-real interactions previously presented at HCI conferences. We apply this framework to those interactions to isolate and group the beyondreal transformations that have been explored by prior works.

SURVEY OF BEYOND-REAL INTERACTIONS
We conducted a systematic review of literature, following PRISMA guidelines [102], to (1) understand past research trends with respect to reality-based, illusory, and beyond-real movement-based VR interactions, (2) evaluate to what extent VR interaction research has explored beyond-real transformations, and (3) explore whether or not researchers have considered the human sensorimotor loop in their exploration of beyond-real interactions.
In this survey, we focused on action execution and more specifically, on motor performance (p. 40) [114]. 3D interaction techniques have been categorized into selection, manipulation, wayfinding, locomotion, system control, and symbolic input [22]. We were particularly interested in selection, manipulation, and locomotion, as they require users to act on the world. We chose to exclude symbolic input and system control through which users change the mode or state of the system, as they do not have a counterpart in our physical reality and fall outside the scope of our work.
Navigation is conceptualized as having two components: wayfinding refers to the cognitive component of navigation and locomotion describes the movement from one place to another [107]. Due to its cognitive nature, wayfinding cannot be fully captured through the lens of sensorimotor control. While we use the term "navigation" as part of our search query to capture all locomotion papers, we do not focus on works that only address wayfinding.

Method
4.1.1 Phase 1: Identification. We searched the ACM Digital Library for full papers targeting the following venues: the ACM Conference on Human Factors in Computing Systems (CHI), the ACM Symposium on Virtual Reality Software and Technology (VRST), the ACM Symposium on User Interface Software and Technology (UIST), and the ACM Conference on Designing Interactive Systems (DIS). Additionally, we searched IEEE Xplore targeting the IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR). We focused our search on VR interaction techniques, allowing terms for common interaction techniques focused on action (selection, manipulation, locomotion, navigation) to appear in either the title or abstract. As we sought to understand trends of research attention to reality-based, illusory, and beyond-real interactions over time, papers placed in our identification phase date back to 1988. Note that we did not use keywords in our search query. An example of the way our queries were structured: Title:((interaction* OR select* OR manipulat* OR locomot* OR navigat*) AND (virtual OR VR)) OR Abstract:(((interaction* OR select* OR manipulat* OR locomot* OR navigat*) AND (virtual OR VR)) This phase found a total of 1268 full papers for further screening.
4.1.2 Phase 2: Screening. We excluded papers that were not focused on virtual reality (326). This excluded augmented reality and other non-immersive platforms such as tabletop displays. Furthermore, we excluded papers that were not focused on interaction techniques (271). We defined interaction techniques as means by which the user engages with the virtual content through movement -as opposed to novel infrastructure, rendering techniques, collision detection algorithms, visualizations, descriptions of input devices or haptic devices. Screening reduced our set to 671 papers.

Phase 3: Eligibility.
We excluded papers that were analyses or experimental evaluations of existing interaction techniques (283), applications of interaction techniques to real-world problems (127), surveys of interaction techniques (8), and revisions of the same interaction techniques produced by the same authors (5). This step built our final study set of 248 papers.
4.1.4 Phase 4: Dataset and coding. We coded each included paper along four dimensions: (1) type of interaction: reality-based, illusory, or beyond-real, and if beyond-real (2) interaction task: selection, manipulation, locomotion, or wayfinding, (3) remapping parameter: space, time, or body, and (4) consideration of sensorimotor loop: yes/no. Often papers developed techniques that leveraged multiple transformations and could be applied to multiple interaction tasks. We applied multiple labels in these cases. Note that while we have coded for all interaction tasks that appeared in our dataset, we focus on interactions that require skilled motor actions and not higher-level cognition (e.g., wayfinding) in the results.

Interaction tasks.
Of the 97 beyond-real interaction tasks, we found: 51 selection (39%), 43 manipulation (33%) and 37 locomotion (28%), shown in Figure 7. These numbers do not sum to 97 because, as mentioned, some interaction techniques are multi-purpose. For example, beyond-real techniques that leverage miniature reconstructions of virtual environments allow for both selection and manipulation of occluded objects [89].

Beyond-Real Transformations Explored
Here we focused on the subset of papers in our survey that explore beyond-real interactions. We applied our framework (described in section 3) to these prior works in order to isolate the beyond-real transformations they utilize and organized these transformations into three groups (space, body, or time) based on their remapping parameter. Using a combination of inductive and deductive coding, we then identified subcategories of transformations that we describe in this section. It should be noted that these transformations can be described in multiple ways. Additionally, a user may reason about these transformations differently than how the transformations are implemented in practice. Therefore, the choice of transformation functions may depend on the specifics of the design, context of the interaction, or the aim of the analysis.

Space Transformations. Space transformations create remappings of movement in 3D space.
Translation: Prior work has explored space translation for locomotion tasks in VR, specifically to augment walking. Translational gain amplifies the shift of the virtual ground under the user's feet to enable users to walk more rapidly in VR [125,195].
Scaling: The scale of rendered content can be altered in VR to enable novel forms of interactions with virtual objects and the surrounding environment. For example, users can scale down the environment to obtain a high-level overview and more easily manipulate large virtual objects [204]. Scale-based remappings have also been leveraged for locomotion. For example, the virtual world can be scaled down with the center of scaling at the midpoint between the user's eyes, allowing them to walk rapidly through a miniature world [2].  [135], c: [204], d: [96]).
Duplication: A fixed mapping can be created between the real-world space and multiple copies of the environment in VR. Poros is one such interaction technique that displays proxies of remote regions of the virtual environment [121]. The changes made to the miniature VR space propagate to the full-size virtual environment, enabling users to utilize the proxy to manipulate objects. Another technique, vMirror, uses strategically placed mirrors and their reflections to allow users to manipulate obscured objects [89].

Body Transformations.
Beyond-real interactions may alter users' body representation, which has been defined in a variety of ways. Given our focus on action, in this paper, we refer to the categorization of bodily representation by Martel et al. [95] which includes body image, body structural description, and body schema. Body image relies heavily on visual input and refers to the conscious representation of the body's shape and size. Body structural description is a conscious spatial map of the body parts and their relationships, informed primarily by somatosensory and visual systems. Body schema refers to the unconscious and highly plastic representation of the body parts, including posture, shape, and size. It should be noted that while some beyond-real interactions may be described in other ways, such as space transformations, they are perceived by users as transformations of body representation. Due to the plasticity of body representation in the human brain, this egocentric perspective is necessary to more accurately capture users' expectations and actions.
Alternate Morphologies: In VR, users can embody avatars with novel sizes, forms, and structures. For example, Ninja Hands maps the movement of a single hand to multiple hands to ease distant target selection [135]. Another paper iteratively adjusts the length, and therefore range of motion, of the avatar's forearms and fingers to achieve better performance on specific tasks [96]. Note that users may perceive space scaling transformations as body scaling and a form of alternate morphology.
Movement Remapping: Movement of the user's body in the real world can be altered virtually to represent another type of movement. Shake-Your-Head maps lateral and vertical head movements to walking and jumping, thus enabling in-place locomotion [175]. Walking by Cycling maps real-world pedaling motions to the walking of a virtual avatar [50]. Movement remappings can also be used for object selection and manipulation. For example, users can move and rotate objects by grasping an imaginary handle bar skewering virtual objects [160]. One prominent form of movement remapping comes in the form of gaze-based interactions, which translates eye movement to hand input such as grabbing or shifting objects [206].
Speed Change: Users of virtual reality applications can develop skills with gentler learning curves with the help of time manipulation -for instance, slower motion of a tennis ball so that beginner players can successfully return it [75]. The motion of avatars in VR can also be accelerated or decelerated to change the user's perception of time [127].

Consideration of sensorimotor loop.
We found that 23 of the 97 beyond-real papers consider the effect of sensory conflict (24%); only 4 of these were published before 2017. Usually discussions of the sensorimotor loop center around simulator sickness evaluated on study participants with the standard Simulator Sickness Questionnaire (SSQ). Primarily SSQ scores are one of several metrics, such as frustration or movement instability, used to assess the effectiveness of a given interaction technique. We found limited examination of causal factors in favor of a more empirical treatment. Deeper model-based analysis such as that enabled by control theory may position researchers to design interaction techniques that do not induce simulator sickness at the outset, as well as iterate more efficiently upon recognition of factors responsible for sensory mismatch. The results of this survey additionally suggest a strong opportunity for the VR community to explore sensorimotor issues with interaction techniques beyond simulator sickness. Our survey identified that over the years, a number of realitybased and beyond-real interaction techniques have been developed and presented at research conferences. While a clear research agenda has been articulated for reality-based interactions (see [72]), as a research community, we are missing a road map for the design and development of beyond-real interactions. Our survey also identified that while there are many new beyond-real interaction techniques developed, only a small percentage of them consider sensorimotor control issues beyond motion sickness. Our proposed framework, beyond being real, allows for a uniform way to describe such interactions from this sensorimotor control perspective, but lacks any predictive ability. We believe a research agenda is needed to help bridge this gap towards better understanding and modeling of beyond-real interactions. Here we discuss some of the open research questions for beyond-real interactions from this perspective.

Control and Usability
In the real world, once learned, we perform skilled motor actions automatically and without conscious awareness of how they are executed [114,124]. Initially, beyond-real remappings are unfamiliar, as they have no counterpart in our physical reality, and the forward model cannot predict the outcome of motor actions. These unexpected prediction errors shift the user's attention in the virtual environment [6,172] and lead to breakdowns that require conscious reflection during interaction, as described by the Heideggerian notion of presentness-at-hand [65,196]. Motor learning can be described as an experience-driven, systematic update to the internal model that enables users to predict the outcome of motor commands and develop new control policies [46] (see Figure 3). For beyond-real interaction designs to be usable [113], users need to learn to synthesize movement under the new dynamics [61]. Meaningful feedback plays a key role in these adaptations, as has been shown for body transformations in VR [36]. However, the best methods for supporting these adaptations are not known.

Drawing from Prior Experiences.
The time it takes for users to learn beyond-real remappings depends on their familiarity with the motor task [61]. To address this lack of familiarity, designers have leveraged users' prior experiences by using themes from science fiction literature, or more broadly books, movies, and other narratives [109]. Another approach is to design interactions that indirectly utilize skills users have already developed in the real world. For example, eye gaze as a mechanism for selecting distant objects leverages a skill we have developed as a result of making eye-contact with others during conversation [72].

Learning
Timescales. Repeated interactions are needed for users to learn new control policies over time. For example, in a study where the range of motion of the participants' arms and legs were swapped, it took around 10 minutes for participants to learn the remapping and to utilize the range of motion of their avatar's body parts [201]. Motor learning is driven by different processes at multiple timescales and it often involves quick approximations, followed by slow adjustments that enable fine tuning [67]. Beyondreal interactions change how we perceive the affordances of objects and properties of the world around us. For example, body scaling influences users' perception of size and distance [16,82,184]. This movement context is captured by the internal model, and compared to the state updates, it changes at a much slower time-scale [200]. Moreover, "motor learning undergoes a period of consolidation, during which time the motor memory is fragile to being disrupted"; therefore, dynamic remappings that may interfere with one another should not be presented in quick succession.

Exploration.
The uncertainty associated with outcomes of motor action directly relates to exploratory behavior [182]. When users suspect that their knowledge of the environment could be improved, they make exploratory choices to increase their learning rate [40]. While the specifics of these learning processes in the brain is not fully understood [88], prior research has proposed different strategies for encouraging exploration. For example, providing lower quality visual feedback may increase uncertainty, promote exploratory risk-taking, and lead to more accurate internal models under the new dynamics [67]. This approach is at odds with the sentiment of effective feedback in interaction design, and more research is needed to unpack this interplay.
Due to these gaps in our understanding, more predictive models are needed to determine the boundaries of usable beyond-real VR interactions and how far we can push those boundaries [201]. More specifically, new methods are needed for evaluating the usability of a beyond-real interaction design and predicting how much time is necessary for users to learn the remapping.

Long-Term Use and Aftereffects
It has been shown that our cognitive system can adjust to repeated exposure to conflicting stimuli [20,35]. This adaptation, which is driven primarily by the forward model [18], has been studied both in the context of illusions and interactions beyond reality. The VR user experience begins when users choose to engage with the virtual content and put on their headset and continues as users exit VR [80]. Therefore, adaptation aftereffects that may carry over into the real world need to be carefully considered. Aftereffects are often task-dependent [6] and may also depend on the speed of the movement [35]. These adaptations also affect other aspects of how we perceive space [82]. However, not all adaptations lead to aftereffects, and this may depend on the modality of the sensory feedback. For example, in a between-subjects study where users experienced either visual or proprioceptive distortions (created by vibration), the effects of proprioceptive adaptations disappeared afterwards [17]. Many research questions remain unanswered in the context of long-term use of beyond-real interactions. Can we train users, through extended practice, to perform effectively under novel remappings? How will the dynamics of the interaction change after the novelty of the beyond-real experience diminishes? Will users maintain their ability to perform under similar remappings the next time they return to the virtual experience?

Individual Differences
Individual differences play a significant role in how users perceive and act in virtual environments [58]. These differences also influence sensory integration and the thresholds at which users become aware of novel remappings. As a result, the categorization of interactions as either illusory or beyond-real is user-dependent. For example, a user that notices a reach redirection illusion [11] will perceive this interaction as a beyond-real space transformation. In the context of beyond-real VR interactions, various factors may contribute to these individual differences, including users' age [66], physique [23], prior experiences [165,194], familiarity with science fiction [109], or gaming frequency [197].

Physiological Responses.
Beyond-real interactions are susceptible to negative physiological responses, as users often receive incongruent sensory feedback from the real world and the virtual system. For example, some users report symptoms of motion sickness when there is a mismatch in sensory feedback from the vestibular and visual systems. Individuals may have different physiological responses to a virtual experience. With regards to flying, researchers have found that "a lot of people find it an endless source of fun, but other people report tired arms and motion sickness" [24]. The user's physiological response is an important consideration from both safety and usability standpoints [12,47].

Emotional
Responses. VR has been recognized as a powerful, immersive media that can evoke strong emotions in users [12]. In particular, experiences beyond reality may lead to profound emotional responses, both positive [54,165], often involving the feeling of awe [31,77], and negative, such as the feelings of fear [21], distress [141,166], and regret [51]. It has been shown that incongruent sensory stimuli result in negative emotions [139]. Conversely, emotional responses influence multisensory processing in ways that can be reflected in action [85]. For example, expectations of high reward release dopamine in the brain, such that it may no longer operate as an optimal controller [40].
At a high-level, individual differences influence sensorimotor control and consequently, how users respond to beyond-real VR interactions. However, many open research questions remain regarding how individual differences should be accounted for in the design of such interactions. How can we capture individual differences from real-world input data? How might we better model user behavior based on individual user's interactions with the system over time? Can we offer adaptive, personalized experiences that account for individual differences? How might we then evaluate beyond-real interaction designs at a large scale?

Presence and Plausibility
Presence, or place illusion, has been defined as the psychological experience of being there: "the extent to which an individual experiences the virtual setting as the one in which they are consciously present" [156]. Designers, often guided by implicit or explicit theories, seek presence in the hopes of improving other attributes of the virtual experience, including learning or task performance [19]. However, researchers have found inconsistent results when studying the correlation between the sense of presence and such attributes [154,156], which perhaps is expected, as these are influenced by many factors, including users' abilities and prior experiences. Due to individual differences, in practice, it may be challenging to evaluate the effects of beyond-real interactions on users' subjective sense of presence. Witmer and Singer [197] have proposed a collection of factors hypothesized to contribute to presence, including control, sensory, distraction, and realism factors. It should be noted that the realism factor does not require content that replicates reality, but relates to continuities, connectedness, and coherence of the virtual experience.
Plausibility illusion, the illusion that what is occurring is actually happening [152], is another psychological dimension that has been attributed to realistic responses to virtual environments. Plausibility illusion also does not require physical realism and is related to causal relationships between the user's actions and the resulting sensations. While in this paper we do not discuss presence and plausibility directly, human sensorimotor control naturally lends itself to discussions around these contributing factors.

Accessibility
Beyond-real interactions must also be considered from the perspective of accessibility. Mott et al. [110] discuss the potential in VR for increased "interaction accessibility" and equity for all people, including people with disabilities, given that in VR people can have abilities no person can experience in the real world, what they call "superpowers" (e.g., flying). Sadeghian and Hassenzahl extend this concept of superpowers into a VR interaction design methodology [133]. While assistive technology in the physical world has many limitations in terms of its ability to adapt, VR and specifically beyond-real interactions might support more adaptive and ability-based interactions [198]. However, researchers have also warned about the potential to amplify differences in ability [110] and that the inherent body-centric perspective of VR poses substantial issues for people with physical disabilities. Gerlig and Spiel [55] highlight the importance of considering minority bodies while designing VR interactions and more importantly including people with disabilities in the design of new interaction paradigms. Additionally, while some VR accessibility research, including the papers we surveyed, focus on manipulating visual feedback (e.g., [101]), there are opportunities for exploration of beyond-real interactions through other sensory feedback, which would increase accessibility for blind and visually impaired VR users, who primarily access VR through audio [34,143] or haptics [150,211].

Ethical Implications
Slater et al. present a detailed discussion of the ethics of realism in virtual reality, and many of the discussion points are highly relevant to beyond-real experiences [153]. Throughout the paper, we have also alluded to some ethical implications of beyond-real VR interactions, such as motion sickness; however, it is necessary to explicitly acknowledge the importance of ethical considerations from a sensorimotor control perspective. Beyond-real remappings may result in motor behavior changes during the interaction. For example, when embodying an avatar with a flexible tail-like appendage, users changed the way they moved their hips [202]. When walking around a virtual environment as a giant, users took bigger steps in the real world [2]. "Presence in VR leads to absence in the physical world" [12]; therefore, not accounting for behavior changes, especially with regards to users' movements in the real world, could have serious consequences. Moreover, long-term use of beyond-real VR interactions may have aftereffects that alter motor behavior in the real world [17]. For example, in a study where users' virtual eye position was offset from the position of their real eyes, it was shown that, after removing the VR headset, participants' hand-eye coordination was altered, as evidenced by their inability to accurately point to a target [20]. Such sensorimotor adaptations pose safety concerns that need to be carefully considered.

LIMITATIONS
In the following section, we will acknowledge some of the limitations of our work and highlight opportunities for future work related to the study of beyond-real virtual reality interactions.

Completeness
The human sensorimotor system is incredibly complex and many details are not captured by the simplified model presented here. For example, multisensory integration happens at multiple stages, utilizing both bottom-up and top-down processes [172], and the sensory signals involve considerable delays that are largely missing in our model [200].
Moreover, our goal for conducting a survey was to apply the framework to selected work exploring beyond-real VR interactions. One limitation is our choice of query; a broader query may be needed for an exhaustive categorization. For example, some VR interaction papers, such as [90], did not appear in our search because neither "VR" nor "virtual" was mentioned in the title or the abstract of the paper. Future work may also consider analyzing other venues, such as the journal of Virtual Reality, and including VR interactions that were not in the scope of our work, such as system control and symbolic input [22].

Embodiment
Previous research has studied the sense of embodiment, which has been defined as having three components: the sense of self-location, the sense of agency, and the sense of body ownership [78]. Our work lacks a coverage of this extensive body of research and how beyondreal interactions may influence users' sense of embodiment in VR. For example, it has been shown that in arm-extension techniques the sense of body ownership declines as the length of the virtual arm increases [79]. The effects of beyond-real transformations on body ownership have mainly been studied in isolation, and our understanding of how different transformations might interact with each other in more complex scenarios is limited. Consider the beyond-real interaction technique Ninja Hands [135], where the movement of the user's hand is mapped to multiple virtual hands in space. In Ninja Hands, these virtual hands are visually disconnected from the user's body (see Figure 8b). While prior research has shown that virtual limbs that are visually connected to the user's body increase the user's sense of body ownership (as measured through their physiological responses) [177], it remains unclear if many connected limbs, as in Ninja Hands, would have a similar effect. Users' sense of body ownership, which is subject to individual differences [94], may have implications for learning and adaptation of beyond-real interactions. Further research is needed to unravel the effects of beyond-real transformations on embodiment, including body ownership, agency, and self-location.

Focus on Action
While we focus on VR experiences that require users to act on the world, applications that do not focus on action could also be beneficial. For example, beyond-real experiences can be utilized for demonstration in educational applications, as they provide a unique opportunity for learning abstract concepts [24]. They can facilitate transformative experiences that evoke strong emotions and elicit new insights [54]. Beyond-real experiences can ignite one's imagination and foster creativity [84] and encourage positive behavior changes that may even transfer to the real world [12].

Social Interactions
In our framework, we have taken an ego-centric approach, focusing on a single user's interactions; however, VR is well suited for socialization and collaboration. When describing how reality is reflected in the word virtual reality, Lanier highlights that "VR functioned as the interstices or connection between people; a role that had been previously taken only by the physical world . . . A reality results when a mind has faith that other minds share enough of the same world to establish communication and empathy" (p. 240) [84].
Bailenson et al. describe techniques beyond reality that change the nature of social interaction in collaborative virtual environments, including manipulation of self representation, sensory capabilities, and the temporal/spatial context [13]. They argue that in VR, unlike face-to-face interaction, the user's rendered behavior can deviate from their actual behavior. The system can leverage this characteristic to, for example, improve communication by altering the user's rendered behavior such that it mimics the nonverbal behavior of others, referred to as the Chameleon Effect [29]. While beyond-real social interactions have been explored [129,130], many research questions remain. For example, how might users with dramatically different scales interact [204]? How does that influence their perception of interpersonal distance?

Other Sensory Modalities
Our work mainly addresses users' visual and somatosensory systems. However, the current state of VR technology enables rendering of audio, and perhaps in the future commercial VR headsets may be able to engage our other sensory input channels, such as olfactory [111]. Virtual experiences beyond reality are not limited to vision and touch, and can span other sensory modalities. For example, in VR, we may gain the ability to smell the scent associated with others, in ways that we could know when someone familiar enters a room without seeing them, or whether that person has previously been in the same room.

Mixed Reality Spectrum
Finally, while we have specifically focused on virtual reality, beyondreal interactions may be integrated into other experiences on the mixed reality spectrum [98]. For example, the Go-Go arm-extension technique has been applied in augmented reality to enable interactions with distant objects in the real world [45]. While similar transformations may be used to describe such interactions, further study of the implications and considerations is needed.

CONCLUSION
In this paper, we first described a simplified model of the control signal flow during movement-based interactions and situated VR interactions within this model. We explained how intent is converted to motor commands in the central nervous system resulting in movements in the real world. These movements are tracked by the VR system and transformed into virtual renderings. Users receive sensory feedback from both the real world and the virtual system. In most cases, the brain operates as an optimal controller and with the use of the state estimator, responds accordingly to perform the intended actions in VR. Using this simplified model, we partitioned the space of VR interactions into reality-based, illusory, and beyond-real based on the magnitude of the resulting sensory conflict. We then presented beyond being real, a framework for describing beyond-real interactions as a set of transformations applied to real-world input. We conducted a survey of prior HCI literature (at CHI, IEEE VR, VRST, UIST, and DIS conferences) with a focus on selection, manipulation, locomotion, and navigation in VR. We applied our framework to extract and categorize the beyond-real transformations in these works and highlighted a gap: research that carefully considers the consequences of sensory conflict resulting from beyond-real transformations. Lastly, we discussed challenges and opportunities for future research towards the goal of better understanding and evaluating interactions beyond reality.