{'fraction_non_alphanumeric': 0.03997603310427472, 'fraction_numerical': 0.023255378508435225, 'mean_word_length': 4.977392277560157, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Mixed Reality Interaction Techniques

This chapter gives an overview of interaction techniques for mixed reality including augmented and virtual reality (AR/VR). Various modalities for input and output are discussed. Specifically, techniques for tangible and surface-based interaction, gesture-based, pen-based, gaze-based, keyboard and mouse-based, as well as haptic interaction are discussed. Furthermore, the combination of multiple modalities in multisensory and multimodal interaction, as well as interaction using multiple physical or virtual displays, are presented. Finally, interaction with intelligent virtual agents is considered.


INTRODUCTION
This chapter gives an overview of interaction techniques for mixed reality (MR), including augmented and also virtual reality (AR/VR). Early research in the field of MR interaction techniques focused on the use of surface-based, tangible as well as gesture-based interaction, which will be presented at the beginning of this chapter. Further modalities, such as pen-based, gaze-based or haptic interaction have seen recent attention and are presented next. Further, with the move towards productivity-oriented use cases, interaction with established input devices such as keyboard and mice has seen interest in the research community. Finally, inspired by the popularity of conversational agents, interaction with intelligent virtual agents are discussed. The development of interaction techniques is closely related to the advancements in input devices. Hence, the reader is invited to study the according book chapter as well. While this chapter follows the above mentioned structure, further possibilities to structure interaction techniques include organizing according to interaction tasks [41] such as object selection [4,80,302] and object manipulation [219], navigation [163], symbolic input [89] or system control [79]. Further, interaction techniques for specific application domains have been discussed such as music [27] games [273] or immersive analytics [51,98]. Interested readers are also referred to further surveys on 3D interaction techniques [145] or interaction with smart glasses [196].

TANGIBLE AND SURFACE-BASED INTERACTION
This section presents the concepts of Tangible user interfaces (TUIs) and their applicability in AR. It covers the effects of output media, spatial registration approaches for TUIs, tangible magic lenses, augmenting large surfaces like walls and whole rooms, the combination of AR with shape-changing displays as well as the role of TUIs for VR-based interaction. Figure 1 depicts an overview about output and input devices typically found in TUI-based interaction for MR.
TUIs are concerned with using physical objects as medium for interaction with computers [325] and have seen substantial interest in human-computer interaction [295]. Early prototypes utilized tabletops, on which physical objects were placed to change properties of digital media. For example Underkoffler and Ishii introduced a simulation of an optical workbench using tangible objects on a tabletop [326] as well as an application for architectural planning [327].
In AR, this concept was introduced by Kato et al. [170] as Tangible Augmented Reality (TAR). They used a paddle as prop, equipped with a fiducial, to place furniture inside a house model. Fjeld et al. [108] introduced further tangibles such as a booklet and a cube for interacting within an educational application for chemistry.
TAR is typically used for visualizing digital information on physical objects while concurrently using those physical objects as interaction devices. Billinghurst et al. [33] state TAR characteristics as realizing spatial registration between virtual and physical objects and the ability for users to interact with those virtual objects by manipulating the physical ones. For example, Regenbrecht et al. [264] utilized a rotary plate to allow multiple co-located users to manipulate the orientation of a shared virtual object. This way, the gap between digital output (e.g., on a flat screen) and physical input (e.g., using a rotary knob) can be reduced as the digital information is directly overlaid over the physical content. Lee et al. [194] described common interaction themes in TAR applications such as static and dynamic mappings between physical and digital objects. They describe a spacemultiplexed approach, where each physical tool is mapped to a single virtual tool or function as well as a time-multiplexed approach in which the physical object is mapped to different digital tools dependent on the context-of use. However, the effect of this overlay depends also on the output medium. For example, when using projection-based systems [34] or video see-through (VST) head-mounted displays (HMDs) (c.f. chapter 10 in [143]), the distance between the observer to the physical and virtual objects is the same. In contrast, when using commodity optical see-through (OST) HMDs with a fixed focal plane, there can be a substantial cost of perceiving virtual and physical objects at the same time. Specifically, Eiberger et al. [94] showed that when processing visual information jointly from objects within arms reach (in this case a handheld display) and information presented on a OST HMD at a different distance, task completion times increased by approximately 50% and error rate increased by approximately 100% compared to processing this visual information solely on the OST HMD.
For spatially registering physical and virtual objects, early works on TAR often relied on fiducial markers, such as provided by ARToolKit [169] or ARUCO [279]. While, easy to prototype (i.e. fiducials have to be simply printed out and attached to objects), these markers can inhibit interaction due to their susceptibility to occlusions (typically through hand and finger interaction). Hence, it is advised to use modern approaches for hand-based interaction [16,203] with spatially tracked rigid and non-rigid objects [141,259,346]. Also, when using OST HMDs, the calibration between the HMD and the users' eyes can impact the interaction [120,134].
Evolving from the magic lens [30] and tangible interaction concepts [325] tangible magic lenses allow to access and manipulate otherwise hidden data in interactive spatial environments. A wide variety of interaction concepts for interactive magic lenses have been proposed within the scope of information visualization (c.f. [319,320]).
Within AR, various rigid shapes have been explored. Examples include rectangular lenses for tabletop interaction [305] or circular lenses [306]. Flexible shapes [308] have been utilized as well as multiple sheets of paper [156]. In their pioneering work, Szalavári and Gervautz [314] introduced the personal-interaction-panel in AR. The two-handed and pen-operated tablet allowed for selection and manipulation of virtual objects as well as for system control. Additionally, transparent props have been explored (e.g., a piece of plexiglass) both for tabletop AR [44,242,270] as well as VR [283]. Purely virtual tangible lenses have been proposed as well [207]. Brown et al. [45] introduced a cubic shape which could either perspectively correct render and manipulate 3D objects or text. This idea was later revisited by Issartel et al. [161] in a mobile setting.
Often, projection-based AR has been used to realize tangible magic lenses, in which a ceilingmounted projector illuminates a prop such as a piece of cardboard or other reflective materials [61,305] and (typically RGB or depth) cameras process user input.
Mobile devices such as smartphones and tablets are also commonly used as a tangible magic lens [131,199], see Figure 2, and can be used in conjunction with posters [131], books [178], digital screens [199] or maps [226,267].
When using the tangible magic lens metaphor in public space, one should be aware about the social acceptability, specifically due to the visibility of spatial gestures and postures [263,272]. For example, in a series of studies across gaming and touristic use cases, Grubert et al. [127,132] explored benefits and drawbacks of smartphone-based tangible lens interfaces in public settings and compared them to traditional static peephole interaction, commonly used in mobile map applications. They found that user acceptance is largely dependent on the social and physical setting. In a public bus stop inside a large open space used at transit area, participants favored the magic lens over a static peephole interface despite tracking errors, fatigue and potentially conspicuous gestures. Also, most passersby did not pay attention to the participants and vice versa. However, when deploying the same experience in a different public transportation stop with other spatial and social context (waiting area, less space to avoid physical proximity to others), Fig. 2. Two magic lens applications in handheld AR. from left to right: Smartphone-based magic lens interface reveals virtual huts and prices on a physical ski map [131]. Tourist searches for prices in front of the map. Find and select game using a handheld magic lens interface. User playing the game in front of a physical poster. [127] Fig. 3. In device-perspective rendering (left) the scene is rendered from the point of view of the camera. In user perspective rendering (right), the scene is rendered from the point of view of the user's head [225]. participants used and preferred the magic lens interface significantly less compared to a static peephole interface. In the context of of evaluations of magic lens metaphors in handheld AR, the impact of tracking technology also has to be considered [228].
Further, when using smartphones or tablets as magic lenses, the default user's view is based on the position of the physical camera attached to the handheld device, see Figure 3, left and Figure 4, left. However, this can potentially negatively affect the user experience [74,75]. Hence, it can be advisable to incorporate user-perspective rendering to render the scene from the point of view of the user's head, see Figure 3, right and Figure 4, right. For example, Hill et al. [153], introduced user-perspective rendering as virtual transparency for VST AR. Baričević et al [15] compared uservs. device-perspective rendering in a VR simulation. Tomioka et al. [321] presented approximated user-perspective rendering using homographies. Grubert et al. [133] proposed a framework for enabling user-perspective rendering for augmenting public displays. Čopič et al. [74,75], quantified the performance differences between device-and user perspective rendering in map-related tasks and Mohr et al. [225], developed techniques for efficient computation of head-tracking techniques needed for user-perspective rendering.
Beyond handheld solutions, whole surfaces such as tables, walls or body parts can be augmented and interacted with. Often projector-camera systems are used for processing input and creating output on surfaces. Early works included augmenting desks using projectors to support office work of single users [211,338,339] or in collaborative settings [269]. Later, the Microsoft Kinect and further commodity depth sensors gave rise to a series of explorations with projector-camera systems. For example, Xiao et al. [348] introduced WorldKit, to allow users to sketch and operate user interface elements on everyday surfaces. Corsten et al. [76] proposed a pipeline for repurposing everyday objects as input devices. Henderson and Feiner also proposed to utilize passive haptic feedback from everyday objects to interact with virtual control elements such as virtual buttons [151]. Mistry and Maes [224] utilized a necklace-mounted projector-camera system to sense finger interactions and project content on hands or the environment. Following suit, Harrison et al., [147] introduced OmniTouch, a wearable projector-depth-camera system that allowed to project user interface elements on body parts, such as the hand (e.g., a virtual dial pad) or to augment paper using touch.
Beyond handheld solutions, whole surfaces such as tables, walls or body parts can be augmented and interacted with. Often projector-camera systems are used for processing input and creating output on surfaces. Early works included augmenting desks using projectors to support office work of single users [211,338,339] or in collaborative settings [269]. Later the Microsoft Kinect and further commodity depth sensors gave rise to a series of explorations with projector-camera systems.
For example, Xiao et al. [348] introduced WorldKit, to allow users to sketch and operate user interface elements on everyday surfaces. Corsten et al. [76] proposed a pipeline for repurposing everyday objects as input devices. Henderson and Feiner also proposed to utilize passive haptic feedback from everyday objects to interact with virtual control elements such as virtual buttons [151].
Mistry and Maes [224] utilized a necklace-mounted projector-camera system to sense finger interactions and project content on hands or the environment. Following suite, Harrison et al., [147] introduced OmniTouch, a wearable projector-depth-camera system that allowed to project user interface elements on body parts, such as the hand (e.g., a virtual dial pad) or to augment paper using touch.
Further, the idea of interacting with augmented surfaces was later expanded to cover bend surfaces [24], walls [167], complete living rooms [166] or even urban facades [40,107]. For example, in IllumiRoom [167], the area around a television was augmented using a projector, after initially scanning it with a depth camera, see Figure 5. Possible augmentations included extending the field of view of on-screen content, selectively rendering scene elements of a game or changing the appearance of the whole environment using non-photorealistic rendering (e.g., cartoon style or a wobble effect). In RoomAlive, multiple projector-depth camera units were used to create a 3D scan of a living room as well as to spatially track the user's movement within that room, see Figure 6. Users were able to interact with digital elements projected in the room using touch and in-air gestures. Apart from entertainment purposes, this idea was also investigated in productivity scenarios such as collaborative content sharing in meetings [105]. Finally, the augmentation of  . Using multiple projector-camera systems a whole room is turned into an immersive experience in RoomAlive [166]. Two different visualizations of the same physical room can be seen. Image courtesy of Microsoft Research.
shape changing interfaces was also explored [109,200,261]. For example, in Sublimate [200] an actuated pin display was combined with a stereoscopic see-through screen to achieve a close coupling between physical and virtual object properties, e.g., for visualizing height fields or NURBS surface modelling. InForm [109] expanded this idea to allow both for user input on its pins (e.g., utilizing them as buttons or handles) as well as manipulation of external objects (such as moving a ball across its surface).
In VR, tangible interaction has been explored using various props. The benefit of using tangibles in VR is that a single physical object can be used to represent multiple virtual objects [3], even if they show a certain extend of discrepancy. Simeone et al. [301], presented a model of potential substitutions based on physical objects such as mugs, bottles, umbrellas or a torch. Hettiarachchi et al. [152] transferred this idea to AR. Harley et al. [146], proposed a system for authoring narrative experiences in VR using tangible objects.

GESTURE-BASED INTERACTION
Touch and in-air gestures and postures make up a large part of interpersonal communication and have also been explored in depth in Mixed Reality. A driver for gesture-based interaction was the desire for "natural" user interaction, i.e. interaction without the need to explicitly handle artificial control devices, but to rely on easy to learn interaction with (to the user) invisible input devices. While many gesture sets have been explored by researchers or users [252], it can be debated how "natural" those gesture-based interfaces really are [236], e.g., due to the poor affordances.
Still, the prevalence of small sensors such as RGB and depth cameras, inertial measurement units, radars or magnetic sensors in mobile devices, AR and VR HMDs, as well as continuing advances in hand [16,203], head [229] and body pose estimation [54,63,67,81,117,205,281] gave rise to a wide variety of gesture-based interaction techniques being explored for Mixed Reality.
The additional modalities are either substituting or complementing the devices' capabilities. These approaches typically relied on modifying existing devices using a variety of sensing techniques, which can limit their deployment to mass audiences. Hence, researchers started to investigate the use of unmodified devices. Nandakumar et al. [231] proposed to use the internal microphones of mobiles to determine the location of finger movements on surfaces, but could not support mid-air interaction. Song et. al [303] enabled in-air gestures using the front and back facing cameras of unmodified mobile devices. With Surround See, Yang et al. [353] modified the front-facing camera of a mobile phone with an omnidirectional lens, extending its field of view to 360 • horizontally. They showcased different application areas, including peripheral environment, object and activity detection, including hand gestures and pointing, but did not comment on the recognition accuracy. In GlassHands, it was demonstrated how the input space around a device can be extended, by using a built-in front-facing camera of an unmodified handheld device and some reflective glasses, like sunglasses, ski goggles or visors [128]. This work was later extended to investigate the feasibility of utilizing eye reflections [288,289].
While being explored since the mid 90's in tabletop-based AR [46,78,87], for handheld AR, vision-based finger and hand tracking became popular since the mid 2000's [158,197,198,296]. Yusof et al. [356] provide a survey on the various flavors of gesture-based interaction in handheld AR, including marker-based and marker-less tracking of fingers or whole hands.
An early example for in-air interaction using AR HMDs is presented by Kolsch et al. [183], who demonstrated finger tracking with a head-mounted camera. Xiao et al. [350] showed how to incorporate touch gestures on everyday surfaces in to the Microsoft HoloLens. Beyond hand and finger tracking, full-body tracking using head-mounted cameras was also explored [60]. Also, reconstruction of facial gestures, e.g., for reenactment purposes, when wearing HMDs has seen increased interest [64,95,318,365]. Further solutions for freehand interaction were also proposed including a wrist-worn gloveless sensor [174], swept frequency capacitive sensing [282], an optical mouse sensor attached to a finger [352], or radar-based sensing [332].
Many AR and VR in-air interaction techniques rely on using arms not being supported by a surface (e.g. an elbow resting on a table). Hence, to facilitate reliable selection, targets are designed to be sufficiently large and spaced apart [304]. Also, while the addition of hand tracking to modern AR and VR HMDs allows for easy access to in-air gestures, the accuracy of those spatial tracking solutions still is significantly lower than dedicated lab-based external tracking systems [291].
Besides interaction with handheld or head-worn devices, also whole environments such as rooms can be equipped with sensors to facilitate gesture-based interaction [43,230,360]. In VR, off-the-shelf controllers were also appropriated to reconstruct human poses in real-time [55,165].

PEN-BASED INTERACTION
In-air interactions in AR and VR typically make use of unsupported hands or controllers designed for gaming. In addition, pens (often in combination with tablets as supporting surface) have also been explored as input device. Szalavári and Gervautz [314], and, similarly, Billinghurst et al. [31] utilized pens for input on physical tablets in AR respectively VR. Watsen et al. [336] used a handheld Personal Digital Assistant (PDA) for operating menus in VR. In the Studierstube framework, pens were used to control 2D user interface elements on a PDA in AR. Poupyrev et al. [255] used a pen for notetaking in VR. Gesslein et al. [116] used a pen for supporting spreadsheet interaction in Mobile VR.
Researches also investigated the use of pens for drawing and modelling, see Figure 7. Sachs et al. [277] presented an early system of 3D CAD modeling using a pen. Deering [83] used a pen for in-air sketching in a fishtank VR environment. Keeve et al. [173] utilized a brush for expressive painting in a Cave Automatic Virtual Environment (CAVE) environment. Encarnacao [97] used a pen and pad for sketching in VR on top of an interactive table. Fiorentino et al. [106] explored the use of pens in mid-air for CAD applications in VR. Xin et al. [351] enabled the creation of 3D sketches using pen and tablet interaction in handheld AR. Yee et al. [354] used a pen-line device along a VST HMD for in-situ sketching in AR. Gasquez et al. [114,115], Arora et al. [5], as well as Drey et al. [88] noted the benefits of supporting both free-form in-air sketching as well as sketching on a supporting 2D surface in AR and VR. Suzuki et al. [313] expanded previous sketching applications for AR with dynamic and responsive graphics, e.g. to support physical simulations.
The performance of pen-based input was also investigated in VR. Bowman and Wingrave [42] compared pen and tablet input for menu selection against floating menus and a pinch-based menu system and found that pen and tablet interaction was significantly faster. Teather and Stuerzlinger [317] compared pen-based in put to mouse input for target selection in a fishtank VR environment and found that 3D pointing was inferior to 2D pointing when targets where rendered stereoscopically. Arora et al. [6] compared pen-based mid-air painting to surface-supported painting and found supporting evidence that accuracy improved using a physical drawing surface. Pham et al. [250] indicated that pens significantly outperform controllers for input in AR and VR and is comparable to mouse-based input for target selection. Batmaz et al. explored different pen grip styles for target selection in VR [17].

GAZE-BASED INTERACTION
Besides utilizing touch input, in-air gestures or physical input devices, gaze has also been explored as input modality in Mixed Reality. Duchowski [91] presents a review of 30 years of gaze-based interaction, in which gaze-based interaction is categorized within a taxonomy that splits interaction into four forms, namely diagnostic (off-line measurement), active (selection, look to shoot), passive (foveated rendering or gaze-contingent displays), and expressive (gaze synthesis).
For VR, Mine [222] proposed to use gaze-directed steering and look-at menus, as early as 1995. Tanriverdi and Jacob [315] highlighted that VR can benefit from gaze tracking. They stated that physical effort can by minimized through gaze and that user's natural eye movement can be employed to perform interactions in VR (e.g., with distant objects). They also indicated that a proposed heuristic gaze selection technique outperformed virtual hand-based interaction in terms of task-completion time. Cournia et al. [77] found that dwell-time based selection was slower than manual ray-pointing. Duchowski et al. [92] presented software techniques for binocular eye tracking within VR as well as their application to aircraft inspection training. Specifically, they presented means for integrating eye trackers into a VR framework, novel 3D calibration techniques and techniques for eye-movement analysis in 3D space. In 2020, Burova et al. [49] also utilized eyegaze analysis in industrial tasks. They used VR to develop AR solutions for maintenance tasks and collected gaze data to elicit comments from industry experts on the usefulness of the AR simulation. Zeleznik et al. [358] investigated gaze interaction for 3D pointing, movement, menu selection and navigation (orbiting and flying) in VR. They introduced Lazy interactions that minimize hand movements, Helping Hand techniques in which gaze augments hand-based techniques as well as Hands Down techniques, in which the hand can operate a separate input device. Piumsomboon et al. [253] presented three novel eye-gaze-based interaction techniques for VR: Duo-Reticles, an eye-gaze selection techniques based on eye-gaze and inertial reticles, Radial Pursuit, a smooth pursuit-based technique for cluttered object and Nod and Roll, a head-gesture-based interaction based on the vestibulo-ocular reflex.

HAPTIC INTERACTION
Auditory and visual channels are widely addressed sensory channels in AR and VR systems. Still, human experiences can be enriched greatly through touch and physical motion. Haptic devices enable the interaction between humans and computers by rendering mechanical signals to stimulate human touch and kinesthetic channels. Research in haptics has a long tradition and incorporates expertise from various fields such as robotics, psychology, biology and computer science. Haptics also play a role in diverse application domains such as gaming [85], industry [347], education [223] or medicine [73,144,341]. Haptic interactions are based on cutaneous/tactile (i.e. skin-related) and kinesthetic/proprioceptive (i.e. related to the body pose) sensations. Various devices have been proposed for both sensory channels, varying in form factor, weight, mobility, comfort as well as the fidelity, duration and intensity of haptic feedback. For recent surveys, please see [26,245]. Also, in VR, using haptic feedback has a long tradition [218]. A commonly used active haptic device for stationary VR environment with a limited movement range of the users hands, is the PHANToM , which is a grounded system (or manipulandum) offering a high fidelity but low portability. Hence, over time substantial research efforts have been made in creating mobile haptic devices for VR [245], see Figure, 8.
In AR, a challenge for using haptics is that the display typically occludes real objects the user might want to interact with. Also, in OST displays, the haptic device is still visible behind virtual objects rendered on the display. When using VST displays, the haptic device might be removed by inpainting [280].
Besides active haptic systems, researchers have also investigated the use of low-fidelity physical objects to augment virtual environments in passive haptics. An early example of this type of haptic feedback is presented by Insko [160], who showed that passive haptics can improve both sense of presence and spatial knowledge training transfer in a virtual environment.
A challenge when using passive haptic feedback, besides a mismatch in surface fidality, is that the objects used for feedback are typically static. To mitigate this problem two strategies can be employed. First, the objects themselves can be moved during interaction by mounting them on robotic platforms such as robots [312,334] or by human operators [68,70]. Second, the movements of the user themselves can be redirected to a certain extend by decoupling the physical motion of a user from the perceived visual motion. This can be done with individual body parts such as hands [8,69], see Figure 9 or the whole body using redirected walking techniques [181,234].

MULTIMODAL INTERACTION
While, often, AR and VR system offer single input channels along with audio-visual output, rich interaction opportunities arise when considering the combination of further input and output modalities. Complementing the strengths of multiple channels can lead to enriched user experiences. While multimodal (or multisensory) output is typically concerned with increasing the immersion and sense of presence in a scene, multimodal input typically tries to increase the efficiency of user interaction with a AR or VR system. For overviews about mutlimodal interaction beyond AR and VR, please see works by Jaimes and Sebe [162] or Turk [324]. Nizam et al. also provide a recent overview about multimodal interaction for specifically for AR [235].
The use of multisensory output such as the combination of audiovisual output with smell and touch has been shown to increase presence and perceived realism in VR [57,155] and has been employed as early as in the 1960s [150]. Gallace et al. discussed both benefits and challenges when Fig. 9. A user has the illusion of touching three different virtual cubes (top row), but is solely touching a single physical cube, to which his arm movements are redirected (bottom row) [8]. Image courtesy of Microsoft Research.
In AR, Narumi et al. [232] showed that increasing the perceived size of a real cookie using AR also increased the feeling of satiety. Narumi et al. [233] also created a multisensory eating expeirence in AR by changing the apparent look and smell of cookies. Koizumi et al. [182] were able to modulate the perceived food texture using a bone-conducting speaker. Ban et al. [11], showed that it is possible to influence fatigue while handling physical objects by affecting the perceived weight of those objects through modulating their size using AR.
Regarding multimodal input in VR, the combination of speech and gestures is a commonly used input combination. In 1980, Bolt [37] introduced put-that-there. Users could immerse themselves in a Media Room to place objects within that environment through a combination of gestures and speech. In 1989, Hauptmann [148] showed that users preferred a combination of speech and and gestures for the spatial manipulation of 3D objects. Cohen et al. [72] used a handheld computer along with speech and gesture for supporting map-based tasks on a virtual workbench. LaViola [191], used hand-based interaction (sensed through a data glove) along with speech for interior design in VR. Ciger et al. [71] combined speech with pointing of a magic wand on an immersive wall to create "magical" experiences. Burdea et al. [48], present an early survey on VR input and output devices as well as an overview about studies that quantify the potentials of several modalities on simulation realism and immersion. Prange et al. [256], studied the use of speech and pen-based interaction in a medical setting.
In AR, Olwal et al. [243], combined speech and gestures for object selection. Kaiser et al. [168] extended that work by introducing mutual disambiguation to improve selection robustness. Similarly, Heidemann et al. [149], presented an AR system for online acquisition of visual knowledge and retrieval of memorized objects using speech and deictic (pointing) gestures. Kolsch et al. [183], combined speech input with gestures in an outdoor AR environment. Piumsomboon [251], studied  [220]. Left: View on a virtual map at a small scale. In a study, users were asked to select an active target (blue dot), with inactive targets visualized as red dots. The pink sphere indicates the fingertip position, the white disc below the pivot point for zooming. The touch screen area of the smartphone is shown as semi-transparent blue rectangle with red border. Right: View on the VR scene at 1:1 scale. The circular targets are complemented with 3D flag symbols to indicate that targets can be selected. The two fingertips used for zooming are indicated by yellow and pink spheres. The pivot point indicated by the white disc is visualized half-way between both fingertips. the use of gestures and speech vs gestures only for object manipulation in AR. They found, that the multimodal was not substantially better than gesture-only based interaction for most tasks (but object scaling). This indicates, that multimodality per se is not always beneficial for interaction, but needs to be carefully designed to suit the task at hand. Rosa et al. [274], discussed different notions of AR and Mixed Reality as well as the role of multimodality. Wilson et al. [343] used a projector-camera system mounted on a pan-tilt platform for multimodal inteaction in a physical room using a combination of speech and gestures.
The combination of touch and 3D movements has also been explored in VR and AR. Tsang et al. [322], introduced the Boom Chameleon, touch display mounted on a tracked mechanical boom and used joint gesture, speech and viewpoint input in a 3D annotation application. Benko et al. [23] combined on surface and in-air gestures for content transfer between a 2D screen and 3D space. Mossel et al. [227] as well as Marzo et al. [216], combined touch input and handheld device movement for 3D object manipulations in mobile AR. Polvi et al. [254] utilized touch and the pose of a handheld touchscreen for remided object positioning in mobile AR. Grandi et al. [118], studied the use of touch and the orientation of a smartphone for collaborative object manipulation in VR. Surale et al. [311] explored the use of touch input on a spatially tracked tablet for object manipulations in VR. In VR, Menzner et al. [220] utilized combined in-air and touch movements on and above smartphones for efficient navigation of multiscale information spaces, see Figure  10. Several authors combined pen input both in mid-air as well as on touch surfaces to enhance sketching in VR [88] and AR [5,114,115].
Also, the combination of eye-gaze with other modalities such as mid-air gestures and headmovements has seen recent interest for interaction in AR and VR. For example, Pfeuffer et al. [248] investigated the combination of gaze and gestures in VR, see Figure 11. They described Gaze + Pinch, which integrates eye gaze to select 3D objects, and indirect freehand gestures to manipulate those objects. They explored this technique for object selection, manipulation, scene navigation, menu interaction, and image zooming. Similarly, Ryu et al. [276] introduced a combined grasp eye-pointing technique for 3D object selection. Kyto et al. [190] combined head and eye gaze for improving target selection in AR. Sidenmark and Gellersen [299,300], studied different techniques combining eye and head pointing in VR. Gesslein et al. [116] combined pen-based input with gaze tracking for efficient interaction across multiple spreadsheets, see Figure 12. Biener et al. [29] utilized gaze and touch interaction for navigating virtual multi-display environments, see Figure  13. Fig. 11. Multimodal target selection in VR using a combination of gaze and gestures [248]. Image courtesy of Ken Pfeuffer. Fig. 12. Interacting with multiple sheets using a combination of pen-based and gaze-based interaction [116]. Initially, solely icons indicating the existent of additionally accessible sheets are visible (a). Neighboring sheets are expanded and each sheet the user gazes at is highlighted with a red frame (b). The user taps with his non-dominant hand on the tablet bezel, causing the selected sheet to slide towards the tablet (c), where the user can edit it using the tablet's touchscreen (d). Fig. 13. Left: a virtual multi-display environment [29]. Right: Users can navigate across displays by gaze (left, right, up, down) and touch (depth).

MULTI-DISPLAY INTERACTION
Traditionally, output of interactive systems is often limited to a single display. However, multidisplay environments from the desktop to gigapixel displays are also increasingly common for knowledge work and complex tasks such as financial trading or factory management as well as for social applications such as second screen TV experiences [125]. Surveys about multi-display systems and distributed user interfaces have been presented by Elmqvist [96], Grubert et al. [124,125,257] and Brudy et al. [47]. Augmented Reality has the potential to enhance interaction with both small and large displays by adding an unlimited virtual screen space or other complementing characteristics like mobility. However, this typically comes at the cost of a lower display fidelity compared to a physical panel display (such as lower resolution, lower contrast, or a smaller physical field of view in OST HMDs).
In 1991, Feiner et al. [104], proposed a hybrid display combining a traditional desktop monitor with an OST HMD and explored a window manager application. Butz et al. [53], combined multiple physical displays ranging from handheld to wall-sized with OST HMDs in a multi-user collaborative environment. Baudisch et al. [19] used a projector to facilitate focus and context interaction on a desktop computer. MacWilliams et al. [213] proposed a multi-user game in which players could interact with a tabletop, laptop as well as handheld displays. Serrano et al. [294] propose to use an OST HMD to facilitate content transfer between multiple physical displays on a desktop. Boring et al. [39] used a smartphone to facilitate content transfer between multiple stationary displays. They later extended the work to manipulate screen content on stationary displays [20] and interactive facades [38] using smartphones. Raedle et al. [258] supported interaction across multiple mobile displays through a top-mounted depth-camera. Grubert et al. [121,123] used face tracking to allow user interaction across multiple mobile devices, which could be dynamically re-positioned, see Figure 14, left. They also proposed to utilize face tracking [121,122] for creating a cubic VR display with user perspective rendering, see Figure 14, right. Butscher et al. [52] explored the combination of VST HMDs with a tabletop displays for information visualization. Reipschläger et al. [265,266] combined a high resolution horizontal desktop display with an OST HMD for design activities. Gugenheimer et al. [137] introduced face touch, which allows interacting with display-fixed user interfaces (using direct touch) and world-fixed content (using raycasting). This work was later extended to utilize three touch displays around the user's head [139], see Figure 15. Gugenheimer et al. also introduced ShareVR [138], which enabled multi-user and multi-display interaction across users inside and outside of VR, see Figure16.
A number of systems also concentrated on the combination of HMDs and handheld as well body-worn displays such as smartwatches, smartphones and tablets in mobile contexts. Here, typically the head-mounted display extends the field of view of the handheld display to provide a larger virtual field of view. In MultiFi [119], an OST HMD provides contextual information for higher resolution touch-enabled displays (smartwatch and smartphone), see Figure 17. The authors explored different spatial reference systems such as body-aligned, device-aligned, and side-by-side modes. Similar explorations have followed suit using video-see-through HMDs [237], an extended set of interaction techniques [364], using smartwatches [209,340,345], or with a focus on understanding smartphone-driven window management techniques for HMDs [271].  15. Multiple Displays around a users head for enhanced interaction in VR [139]. Image courtesy of Jan Gugenheimer. Fig. 16. Left: A controller mounted dipslay to allow users outside of VR to interact with a user wearing an immersive head-mounted display in ShareVR [138]. Right: The ShareVR environment resembling a living room. Image courtesy of Jan Gugenheimer.
Purely virtual multi-display environments have also been explored in AR and VR. In 1993, Feiner et al. [103] introduced head-surrounding and world reference frames for positioning 3D windows in VR. In 1998, Billinghurst et al. [32] introduced the spatial display metaphor, in which information windows are arranged on a virtual cylinder around the user. Since then, virtual information displays have been explored in various reference systems, such as world-, object-, head-, body-or device-referenced systems [192]. Specifically, interacting with windows in body-centered reference systems [330] has attracted attention, for instance to allow fast access to virtual items [66,202], mobile multi-tasking [99,101] and visual analytics [100]. Lee et al. [195] investigated positioning a window in 3D space using a continuous hand gesture. Petford et al. [247] compared the selection performance of mouse and raycast pointing in full coverage displays (not in VR). Jetter et al. [164] proposed to interactively design a space with various display form factors in VR.

INTERACTION USING KEYBOARD AND MOUSE
Being the de facto standard for human-computer interaction in personal computing environments for decades, standard input peripherals such as keyboard and mouse, while initially used in projection-based CAVE environments, were soon replaced by special purpose input devices and associated interaction techniques for AR and VR (see previous sections) . This was partly due to the constraints of those input devices making them challenging to use for spatial input with six degrees of freedom. Physical keyboards typically support solely symbolic input. Standard computer mice are restricted to two-dimensional pointing (along with button clicks, and a scroll-wheel). However, with modern knowledge workers still relying on the efficiency of those physical input devices, researchers revisited how to use them within AR and VR.
With increasing interest in supporting knowledge work using AR and VR HMDs [130,140,204,275], keyboard and mouse interaction drew attention by several researchers.
The keyboard was designed for rapid entrance of symbolic information, and although it may not be the best mechanism developed for the task, its familiarity that enabled good performance by users without considerable learning efforts kept it almost unchanged for many years. However, when interacting with spatial data, they are perceived as falling short of providing efficient input capabilities [28], even though they are successfully used in many 3D environments (such as CAD or gaming [309]), can be modified to to allow 3D interaction [246,335] or can outperform 3D input devices in specific tasks such as 3D object placement [25,310]. Also for 3D object manipulation in AR and VR they found to be not significantly slower than a dedicated 3D input device [187].
In VR, a number of works investigated the costs of using physical keyboards for standard text entry tasks, see Figure 18. Grubert et al. [135,136], Knierim et al. [180] as well as McGill et al. [217] found physical keyboards to be mostly usable for text entry in immersive head-mounted display-based VR but varied in their observations about the performance loss when transferring text entry from the physical to the virtual world. Pham et al. [249] deployed a physical keyboard on a tray to facilitate mobile text entry. Apart from standard QWERTY keyboards a variety of further text entry input devices and techniques have been proposed for VR, see [89].
Besides using unmodified physical keyboards, there have been several approaches in extending the basic input capabilities of physical keyboard beyond individual button presses. Specifically, input on, above and around the keyboard surface have been proposed using acoustic [171,189], pressure [86,208,357], proximity [316], capacitive sensors [35,102,142,268,298,323], cameras [175,260,344], body-worn orientation sensors [50] or even unmodified physical keyboards [193,Fig. 18. Text entry in VR using standard physical keyboards using different hand reprensentations [135]. From left to right: no representation, inverse-kinematic model, finger tip representation using spheres, video pass-through of the user's hands. 359]. Besides sensing, actuation of keys has also been explored [9]. Embedding capacitive sensing into keyboards has been studied by various researchers. It lends itself to detect finger events on and slightly above keys and can be integrated into mass-manufacturing processes. Rekimoto et al. [268] investigated capacitive sensing on a keypad, but not a full keyboard. Habib et al. [142] and Tung et al. [323] proposed to use capacitive sensing embedded into a full physical keyboard to allow touchpad operation on the keyboard surface. Tung et al. [323] developed a classifier to automatically distinguish between text entry and touchpad mode on the keyboard. Shi et al. developed microgestures on capacitive sensing keys [297,298]. Similarly, Zheng et al. [362,363] explored various interaction mappings for finger and hand postures. Sekimoro et al. focused on exploring gestural interactions on the space bar [293]. Extending the idea of LCD-programmable keyboards [159], Block et al. extended the output capabilities of touch-sensitive, capacitive sensing keyboard by using a top-mounted projector [35]. Several commercial products have also augmented physical keyboards with additional, partly interactive, displays (e.g., Apple Touch Bar [2], Logitech G19 [206], Razer Death-Stalker Ultimate [262]).
Maiti et al. [215] explored the use of randomized keyboard layouts on physical keyboards using an OST display. Wang et al. [333] explored the use of an Augmented Reality extension to a desktopbased analytics environment. Specifically, they added a stereoscopic data view using a HoloLens to a traditional 2D desktop environment and interacted with keyboard and mouse across both the HoloLens and the desktop.
Schneider et al. [290] explored the design space of using physical keyboards in VR beyond text entry, see Figure 19. Specifically, they proposed three different input mappings: 1 key to 1 action (standard mode of interaction using keyboards), multiple keys to a single action (e.g., mapping a large virtual button to several physical buttons), as well as mapping a physical key to a coordinate in a two-dimensional input space. Similarly, they proposed three different output mappings: augmenting individual keys (e.g., showing an emoji on a key), augmenting on and around the keyboard (e.g., adding additional user interface elements on top of the keyboard such as virtual sliders) as well as transforming the keyboard geometry itself (e.g., only displaying single buttons, or replacing the keyboard by other visuals). Those ideas were later also considered in the domain of immersive analytics [129].
Mouse-based pointing has been studied in depth outside of AR and VR for pointing on single monitors [56] as well as multi-display environments [7,18,21]. However, it has been found that standard 2D mice do not adapt well to multi-display interaction [331], an issue which is also relevant for AR and VR. Consequently, standard mice have been modified in various ways to add additional degrees of freedom. For example, Villar et al. [328] explored multiple form factors for multi-touch enabled mice. Other researchers have added additional mouse sensors to support yawing [212,244], pressure sensors for discrete selection [59,177] to allow for three instead of two degrees of freedom. Three-dimensional interaction was enabled using the Rockin'Mouse [10] and the VideoMouse [154]. Both works added a dome below the device to facilitate 3D interaction. Steed and Slater [307] proposed to add a dome on top of the mouse rather than below. Further form factors have also been proposed to facilitate pointing based interaction in 3D [111,112]. Recently, researchers also worked on unifying efficient input both in 2D and 3D [246,278].
Standard mice using a scroll wheel can also be efficiently used for 3D object selection when being combined with gaze-tracking in virtual multi-display environments [29]. For example, in the Windows Mixed Reality Toolkit [221], the x and y-movement of the mouse can be mapped to the x and y movements on a proxy shape such as a cylinder (or any object on that cylinder, like a window). The scroll wheel is used for changing the pointer depth (in discrete steps). The x-and ymovements can be limited to the current field of view of the user to allow for acceptable control to display ratios. The user gaze can then be used to change the view on different regions of the proxy shape.

VIRTUAL AGENTS
Virtual agents can be considered as "intelligent" software programs performing tasks on behalf of users' questions or commands. While it can be argued, what "intelligent" really means in this context, a widely accepted characteristic of this "intelligence" is context-aware behaviour [126,329]. This allows an agent to interact with the user and environment through sensing and acting in an independent and dynamic way. The behaviour is typically well-defined and allows to trigger actions based on a set of conditions [240]. The rise of voice assistants (or conversational agents) [157], which interact with users through natural language, has brought media attention and a prevalence in various areas such as home automation, in-car operation, automation of call centers, education and training [239].  [287]. Image courtesy of Gerd Bruder.
In AR and VR, virtual agents often use more than a single modality for input and output. Complementary to voice in-an output, virtual agents in AR and VR can typically react to body gestures or postures or even facial expressions of the users. Due to their graphical representations those agents are embodied in the virtual world. The level of embodiment of a virtual agent has been studied extensively [84,355]. For example it has been shown, that the effect of adding a face was larger than effect of visual realism (both photo-realism and behavioural realism of the avatar) [355]. In VR, the level of visual realism of the virtual agent is typically matched to the visual realism of the environment, see Figure 20. In contrast, in AR, there is often a noticeable difference between the agent representation and the physical scene and those effects are still underexplored [176], see Figure 21. Hantono et al. review the use of virtual agents in AR in educational settings. Norouzi et al. provide review of the convergence between AR and virtual agents [238].
Specifically for AR, Maes et al. [214] introduced a magic mirror AR system, in which humans could interact with a dog through both voice and gestures. Similarly, Cavazza et al. [58] allowed participants to interact with virtual agents in an interactive storytelling environment. MacIntyre et al. [210] used pre-recorded videos of phyiscal actors to let users interact with them using OST HMDs. Anabuki et al. [1] highlight that having virtual agents and users share the same physical environment is the most distinguishing aspect of virtual agents in AR. They introduced Welbo, an animated virtual agent, which is is aware of its physical environment and can avoid standing in the user's way. Barakony et al. [12] presented AR Puppet, a system that explored the context-aware animated agents within AR. The authors investigated aspects like visualization, appearance or behaviors. They also studied AR-specific aspects such as the ability of the agent to avoid physical obstacles or its ability to interact with physical objects. Based on this initial research, the authors explored various applications [13,14]. Chekhlov et al. [62] presented a system based on Simultanteous Localization and Mapping (SLAM) [93], in which a virtual agent had to move in a physical environment. Blum et al. [36] introduced an outdoor AR game which included virtual agents. Kotranza et al. [184,185] used a tangible physical representation of a human that could be touched, along with a virtual visual representation in a medical education context. They called this dual representaiton mixed reality humans and argued that affording touch between a human and a virtual agent enables interpersonal scenarios.

SUMMARY AND OUTLOOK
This chapter served as an overview of a wide variety of interaction techniques MR -covering both device-and prop-based input such as tangible interaction, pen and keyboard input as well as utilizing human effector-based input such as spatial gestures, gaze or speech.
The historical development of the presented techniques was closely coupled to the available sensing capabilities. For example, in order to recognize props such as paddles [170], they had to be large enough in order to let fiducials be recognized by low-resolution cameras. With the advancement of computer vision-based sensing, fiducials could be come smaller, change their appearance to natural looking images or be omitted altogether (e.g., for hand and finger tracking). Further, the combination of more than one modality became possible through increasing computational capabilities of MR systems.
In the future, we expect an ongoing trend of both minimizing the size and price of sensors, as well as the ubiquitous availability of those sensors, in dedicated computing devices, in everyday objects [337], on [284] or even in the human body itself [292]. Hence, MR interaction techniques will play a central role on shaping the future of both pervasive computing [126] as well as augmenting humans with (potentially) super human capabilities (e.g., motor capabilities [172,188], cognitive and perceptual capabilities [285]). Besides technological and interaction challenges along the way, the field of MR interaction will greatly benefit from including both social and ethical implications when designing future interfaces.