OctoXR - VR Interaction Framework

To better understand the tool that we developed, reader needs to understand the concept of hand tracking technology, what preceded it, and why is it a more immersive and realistic way of interaction in VR, at least in our humble opinion.

What is hand tracking in VR?

Hand tracking is slowly but surely becoming an integral feature of recent generations of immersive virtual reality devices. It allows users to interact with all sorts of virtual elements without the need to use controllers. Sensors and cameras on VR headsets track users' hands and then use the data they gather to create a real-time virtual embodiment of them.

Before, hand tracking technology in VR was introduced through the first and second iterations of the Oculus Quest, and interactions with the virtual environment were usually performed with hand-held controllers. The user's hands were visualized in the applications in a limited array of poses based on finger position, assumed from contact with triggers and buttons on these controllers.

Hand tracking can be used for more than just picking up objects or firing weapons at enemies: it also makes interactions with other people feel more real as well as it allows for faster communication between users. With hand tracking enabled on your device, all kinds of things become possible ranging from simple gestures like clapping hands together ("high five") or even communicating in sign language.

In comparison with controllers, hand tracking technology offers three significant advantages:

1. Increased immersion and presence
Research has shown that we have an almost supernatural sense of the positions and shape of our hands when they are shielded or out of sight, and when our hands are removed from our visual worlds, it is a reminder of our disembodiedness. We spend most of our time during various daily tasks moving and twisting our hands, so removing them from the visual scene or not showing the exact movement of our hands in a virtual environment (which is the case when using controllers in VR) has several consequences for our visuomotor behavior.

2. More effective interaction
The main goal of virtual reality is to enable the user to interact with a computer-generated environment in a natural way. An example of such an interaction in its simplest form can be moving the user’s head to experience a wide visual world. But more modern VR experiences usually involve some form of manual interaction like pushing a button or carrying an object. Accurate hand tracking potentially allows for far more precise and better interactions compared to controllers, adding not only to the user’s immersion but even the accuracy of his movements, which seems particularly crucial in the training context.

3. More effective communication
Recent research has shown that individuals have difficulty recognizing many key emotions in body images that have had their hands removed, emphasizing how important information about the shape of the hand is in communication experiences. The value of hand gestures for communication in virtual environments is increasing as real-time face tracking and visualization are technically very difficult due to the extremely high temporal and spatial resolution required to detect and track micro-expressions.

One of the most exciting things about using hands in VR is enabling users to interact with the virtual world more like they do in real life. The core of this is the ability to grab and manipulate objects. We noticed that there is no out-of-the-box solution on the market for quick and easy implementation of tracking and interaction (especially physics) with hands in VR. Without OctoXR, grabbing an object with hands in VR is preceded by weeks or more of development that are now converted to just a few minutes.

Enter OctoXR 

OctoXR is a low-code Unity game engine plugin that allows its users to significantly reduce the development time of their realistic and physics-based hand interactions.

Developing a solution in virtual reality is a time-consuming and complicated process, especially when trying to interact using hands. OctoXR allows the user to facilitate and shorten the process of creating applications within virtual reality using hand tracking.

The plugin offers a range of features such as physical interactions with objects, a ready-made solution for capturing objects from a distance or proximity, pre-built solutions for moving in virtual reality such as teleportation and continuous movement, inverse kinematics, VR user interface, and gesture recognition that allows the user to further personalize desired interactions. Our plugin includes a library of interaction components that are modular, composable, and scalable that you can integrate into your existing architecture or run standalone. OctoXR takes complicated and undocumented parts of virtual reality and offers them as ready-made solutions which allow a user who does not necessarily have a background in development to turn an idea into reality much faster and easier than ever before.

OctoXR features

1. Kinematic grab
The kinematic grab feature lets you create joint-based objects which collide with the environment or kinematic grabbables for a more game-like feel in just a few clicks! Create grab points with a single click and fine-tune their position in the editor. With the OctoXR interactable module, grabbing stuff has never been easier.

2. Hand physics
This feature lets you easily create and edit a physics-enabled hand rig that makes a VR hand interact with other physically simulated objects in the scene. It will also automatically respond to collisions between any desired hand part and other objects. You can use provided scripts and extend them to make your custom behaviors and interactions based on physical or trigger collisions between hands and other objects.

3. Distance grab
Distance grab lets you detect distant grabbables with raycasting and interact with them from afar using the distance grabbing module or grab what's within the range of your hand using the proximity interactor.

4. Bad hand positioning
Hand tracking can sometimes act out, with the OctoXR plugin you can stop your hands when things go wrong, preventing your interactions from messing up. Our bad hand tracking detection system allows your hands to stop in place with visual feedback, so your users can know exactly what's going on.

5. Gesture recognition
Making apps with gesture recognition is made easier than ever. You can create gestures with a single click of a button. Created gestures are saved in the projects Asset’s folder. You can then choose which gestures you wish to load in your scene. After you load gestures, with a simple drag and drop technique you attach functionalities to specific gestures. Gestures are recognized by comparing current hand bone positions with the ones saved with every gesture. If a gesture is recognized, the gesture detector fires a recognized event and executes attached functionalities.

6. Locomotion
Developing a good locomotion technique in VR has always been a big issue. Especially using hand tracking. OctoXR offers two readymade solutions: continuous locomotion (or walking) and teleport locomotion. There are a couple of predefined teleportation modes: pinch teleport and gesture teleport. You can create your own gestures you wish to use for teleport and attach them to gesture teleport. But if you feel you need even more freedom to develop your own type of teleport, you can simply inherit the teleport base class and by overriding, a couple of lines of code have your own brand-new teleport.

7. OctoXR UI
We developed the “oculus home like” UI manipulation that all Oculus developers are used to. Simply attach UIPointer to the hand and insert OctoInputModule prefab in the scene and it just works.

8. Inverse kinematics
The inverse kinematics feature provides support for models using the Unity humanoid rig. We integrated head, arm and finger tracking.

Hand tracking in VR is a game-changer. Many companies are already using it to make their games more immersive, accessible and fun. It's an important part of the future of virtual reality, and OctoXR will help you get there sooner than you think. We believe that by giving people a powerful tool like this, we can make the future of VR so much more exciting and fun. If you have any questions about or feedback on the topic please feel free to reach out.

For more info about OctoXR make sure to check here.