[Eng] VR UX 15. Key Interaction Concepts

Interaction fidelity

Interaction fidelity in virtual reality (VR) refers to how closely the actions and tasks within the virtual environment match those in the real world. It's a crucial concept for VR designers to consider because it greatly impacts the user experience and the effectiveness of the VR application. Let's delve deeper into the different types of interaction fidelity:

1. High Interaction Fidelity: This level of fidelity aims to replicate real-world actions and sensations as closely as possible within the virtual environment. High interaction fidelity is particularly valuable for training simulations, where users need to learn and practice tasks that mimic real-world scenarios. For example, in a VR medical training simulation, high interaction fidelity would ensure that actions like surgical procedures or patient examinations feel realistic and provide tactile feedback similar to real-life experiences. High interaction fidelity enhances immersion and can improve learning outcomes by making the training experience more authentic and engaging.
2. Low Interaction Fidelity: In contrast to high fidelity, low interaction fidelity offers experiences that may deviate from physical laws or real-world constraints. This approach allows designers to introduce novel and imaginative interactions that enhance performance and enjoyment in the virtual environment. Low interaction fidelity is often employed in gaming or entertainment applications where realism may not be the primary goal. For example, in a VR game set in a fantasy world, low interaction fidelity might enable players to perform magical gestures or wield supernatural powers that defy the laws of physics. This type of fidelity prioritizes creativity and gameplay experience over strict adherence to real-world mechanics.
3. Magical Interactions: Magical interactions introduce fantastical enhancements or intelligent guidance within the virtual environment. These interactions prioritize user experience and may involve elements of surprise, delight, or awe. Magical interactions blur the line between reality and imagination, offering users a sense of wonder and immersion. In gaming, magical interactions can contribute to the game's narrative, world-building, and player engagement. For example, a VR puzzle game might feature magical tools or artifacts that assist players in solving puzzles or navigating through challenges. By incorporating magical elements, designers can create memorable and captivating experiences that go beyond the constraints of traditional interaction fidelity.

Overall, interaction fidelity is a multidimensional concept that encompasses various aspects such as biomechanical symmetry, input veracity, and control symmetry. Designers must carefully consider the level of interaction fidelity appropriate for their VR application, taking into account factors such as user goals, context, and desired user experience. Whether aiming for high realism, imaginative exploration, or magical immersion, understanding interaction fidelity is essential for creating successful and engaging VR experiences.

 

Reference frames

Reference frames in virtual reality (VR) are essential components that define the spatial context and facilitate interactions within the virtual environment. Let's explore the different types of reference frames commonly used in VR design:

1. Virtual-World Reference Frame: This reference frame aligns with the virtual environment itself. It serves as the primary coordinate system for navigation, mapping, and positioning of objects within the virtual space. Users perceive and interact with virtual objects relative to this reference frame, which helps them orient themselves and move around within the VR environment. For example, in a VR game, the virtual-world reference frame determines how users navigate through virtual landscapes or explore virtual buildings.
2. Real-World Reference Frame: Unlike the virtual-world reference frame, the real-world reference frame is tied to physical space. It allows for stable interactions by aligning virtual objects with corresponding real-world objects or physical movements. For instance, if a user holds a physical controller that represents a virtual tool or object within the VR environment, the real-world reference frame ensures that the virtual object's position and appearance match that of the physical controller. This alignment enhances immersion and allows users to manipulate virtual objects more intuitively.
3. Torso Reference Frame: The torso reference frame is based on the body's spinal axis, providing a frame of reference that moves with the user's torso or upper body movements. It is particularly useful for close interactions and navigation within the VR environment. For example, when a user turns their body or leans forward, virtual objects positioned within the torso reference frame also adjust their orientation or position accordingly. This helps maintain spatial consistency and enhances the sense of presence in VR experiences.
4. Hand Reference Frames: Hand reference frames are centered around the position of the user's hands, typically used in conjunction with hand-held controllers or hand-tracking devices. These reference frames are crucial for accurately representing hand movements and interactions within the virtual environment. For example, virtual interfaces, labels, or objects attached to the user's hands, such as a virtual watch or wrist-mounted interface, provide relevant information or controls that are easily accessible during the VR experience. Hand reference frames enable precise manipulation of virtual objects and enhance the sense of agency and control for users.

 

Speech and gestures

Speech and gestures play integral roles in user interaction within virtual reality (VR) environments, offering intuitive and immersive means of communication with the virtual world. Let's explore the effectiveness and significance of speech and gestures in VR design:

• Simplicity and Quantity: Streamlining commands and gestures simplifies the user experience in VR. By reducing complexity and the number of required inputs, designers can make interactions more intuitive and accessible to users. Limiting the variety and quantity of commands and gestures ensures that users can quickly learn and remember how to interact with the VR environment, enhancing usability and reducing cognitive load.
• Visual Cues and Icons: Visual cues or icons accompanying speech and gestures provide clear communication with the VR system, reinforcing user understanding and engagement. These visual aids serve as prompts or feedback, guiding users on how to perform actions and interpret their effects within the virtual environment. Incorporating visual elements alongside speech and gestures enhances communication effectiveness and user experience, especially for users who may rely more on visual cues.
• Natural User Interactions: Clearly defined gestures and voice commands enable users to interact more naturally with the virtual environment, mimicking real-world interactions. By recognizing and responding to natural gestures and speech patterns, VR systems create a more immersive and intuitive user experience. Natural user interactions promote user engagement and satisfaction, fostering a deeper sense of presence within the virtual world.
• Verification and Feedback: Verification and feedback mechanisms are essential for ensuring accurate recognition of gestures and speech commands, especially in hidden recognition systems where users may not receive immediate visual confirmation of their inputs. Providing feedback through visual, auditory, or haptic cues confirms user actions and reinforces successful interactions. Verification and feedback mechanisms enhance user confidence and reduce frustration, contributing to a more seamless and enjoyable VR experience.

 

• Gestures: Gestures in VR encompass both dynamic actions and static postures, conveying meanings through bodily movements. Dynamic gestures involve motion, often tracked by sensors or controllers, while static postures represent still configurations. Gestures can communicate spatial information by manipulating or indicating virtual objects within the environment. For example, a pushing gesture can move an object away from the user. They can also convey symbolic information, representing concepts or ideas indirectly. For instance, forming a "V" shape with fingers signifies victory or success. Furthermore, gestures may reflect pathic information, revealing thoughts or emotions through subconscious hand movements during interaction. Additionally, affective information can be conveyed through gestures, expressing emotions such as distress or enthusiasm, particularly in human-to-human interaction scenarios. Incorporating gestures into VR design enriches communication and interaction possibilities, enabling users to express themselves and manipulate the virtual environment in more nuanced and expressive ways. Gesture recognition technology plays a crucial role in accurately interpreting and responding to user gestures, enhancing immersion and user engagement in VR experiences.

 

Speech recognition

Speech recognition technology in virtual reality (VR) enables users to interact with the virtual environment using spoken commands, converting speech into text and meaningful actions. Let's explore the categories of speech recognition and strategies used in VR design:

 

Categories of Speech Recognition:

1. Speaker-Independent: This category of speech recognition recognizes a limited set of words from various users without prior training. It is suitable for VR systems with few options or commands. Speaker-independent recognition allows multiple users to interact with the VR environment without the need for individual training sessions. However, it may have limitations in accurately interpreting speech in noisy environments or with variations in speakers' voices.
2. Speaker-Dependent: Speaker-dependent speech recognition recognizes a more extensive vocabulary from one trained user. Users need to undergo a training process where the system learns to recognize their voice patterns and speech characteristics. Speaker-dependent recognition works well for personal VR setups where users have consistent speech patterns and vocabulary. It offers higher accuracy compared to speaker-independent recognition but may not be suitable for shared VR environments.
3. Adaptive Recognition: Adaptive recognition combines the speaker-independent and speaker-dependent approaches, adapting to users' speech patterns and characteristics over time. The system learns from user interactions and continuously refines its recognition capabilities, offering improved accuracy and usability. Adaptive recognition is well-suited for dynamic VR environments where multiple users with varying speech patterns interact with the system.

Strategies for Speech Recognition:

1. Discrete/Isolated: Discrete or isolated speech recognition recognizes single words or short phrases at a time. It is suitable for recognizing distinct commands or actions within the VR environment, such as "save" or "undo." Discrete recognition simplifies the speech recognition process by focusing on individual words, making it easier to implement and ensuring high accuracy for specific commands.
2. Continuous/Connected: Continuous or connected speech recognition recognizes consecutive words or phrases, allowing for more natural and fluid interactions with the VR system. Unlike discrete recognition, continuous recognition processes entire sentences or phrases, making it more complex to implement but enabling users to speak more naturally without pausing between words. Continuous recognition enhances the user experience by mimicking natural speech patterns and allowing for more flexible and conversational interactions.
3. Phonetic: Phonetic speech recognition focuses on recognizing individual speech sounds or phonemes, analyzing the acoustic properties of speech to identify phonetic patterns. While phonetic recognition offers high accuracy and robustness, it is less commonly used in VR due to its complexity and computational demands. Phonetic recognition may be employed in specialized applications where precise phonetic transcription is required, such as language learning or speech therapy.
4. Spontaneous/Conversational: Spontaneous or conversational speech recognition focuses on understanding word context and conversational cues, enabling more natural dialogue between users and the VR system. This approach considers the surrounding context and user intent to interpret speech accurately and provide relevant responses. Spontaneous recognition enhances the user experience by simulating natural conversations and allowing for more fluid and intuitive interactions in VR.

 

Multimodal interaction

Multimodal interaction in virtual reality (VR) integrates multiple sensory inputs and outputs to create richer and more immersive user experiences. By combining different modes of interaction such as voice, gestures, and physical inputs, multimodal interfaces offer users increased flexibility, efficiency, and accessibility.

© Interaction Design Foundation, CC BY-SA 4.0

 

Let's explore the different types of input integration in multimodal interaction:

1. Specialized Integration: This approach involves using the most suitable method for a specific task or interaction. For example, in a VR environment, selecting an object by pointing at it with a controller may be the most efficient method for certain tasks that require precise manipulation or selection.
2. Equivalent Integration: In equivalent integration, users are provided with multiple options to achieve the same outcome. This allows users to choose the method that best suits their preferences or capabilities. For instance, in a VR design program, users may have the option to create objects either by speaking a command or selecting them from a menu.
3. Redundant Integration: Redundant integration involves using two or more inputs simultaneously to ensure that the system correctly understands the user's intent. For example, in VR, users may select an object using a hand gesture while simultaneously verbally confirming their selection with a command like "select."
4. Concurrent Integration: Concurrent integration enables users to issue different commands simultaneously, allowing for multitasking-like interactions in VR. For instance, a user may point to fly in one direction while verbally asking about a distant object, enabling seamless and efficient multitasking within the virtual environment.
5. Complementary Integration: Complementary integration combines different inputs into a single command or action, streamlining interactions and reducing the time required to perform tasks. For example, in a VR game, users may use both voice commands and hand gestures together to quickly delete an object, enhancing the speed and efficiency of the interaction.
6. Transfer Integration: Transfer integration involves shifting from one input method to another seamlessly during an interaction. This approach improves understanding and efficiency by leveraging different input modalities based on the context or user preference. For instance, a user may initiate a task in VR with a voice command and then seamlessly transition to using hand gestures to complete the task, ensuring a smooth and intuitive interaction flow.

Learn More

• Watch this video to learn more about VR interactions.
• Read this Medium article, “A case for gestural interactions in spatial computing”. 
• To learn more about artificiality, read “A Tale of Two Worlds: The Natural World and the Artificial World” by J. T. Trevors & M. H. Saier.
• Chapter 26 of Jason Jerald’s The VR Book: Human-Centered Design for Virtual Reality explores the VR concepts covered in this lesson and more.