Researchers from Carnegie Mellon University demonstrated a practical full-body tracking system in virtual reality using cameras mounted on the controllers to get a better view of the user’s body.
Although it is possible to achieve full body tracking in virtual reality today, this requires the use of additional hardware that must be attached to your body (eg Vive trackers or IMU trackers). This makes full-body tracking a non-starter for all but VR enthusiasts who are willing to spend the money and time attaching extra gear.
Since standalone VR headsets already come with cameras to track their position in the world and the user’s controllers, in theory it’s also possible to look at the user’s body and use a vision approach. by computer to follow it. Unfortunately, the angle of the headset cameras is too extreme to get a reliable view of the user’s legs, leading Meta to recently conclude that full-body tracking just isn’t viable on a standalone headset. (especially as they get smaller).
But researchers at Carnegie Mellon University are challenging that notion with a prototype standalone VR system that adds cameras to controllers to get a much clearer view of the user’s body, allowing reliable tracking data to be extracted. for legs and torso.
What’s particularly interesting about this approach is that it seems to align with the direction next-gen VR controllers are already heading; Meta’s Project Cambria and Magic Leap 2 both use controllers that ditch a headset-dependent tracking system in favor of calculating their position with their own backwards tracking system.
Using a standard Quest 2 headset as the basis for their prototype system, the researchers added two cameras to the controller that face the user. With the user’s hands in front of them, the cameras can get a much clearer view of the upper and lower body. This view is corrected so that a computer vision system can optimally extract the pose of the user, then combine this data with the known position of the head and hands to create a full-body tracking model.
Of course, the user’s hands will not be still been before them. The researchers say that some limited testing has shown that VR users have their hands outstretched in front of them about 68% of the time. When the hands are not in a good position to capture the body, the system must revert to a CI estimate of the body position. And while their prototype didn’t go that far, the researchers say they believe that with an additional camera angle on the controller, it should be possible to capture leg position even when the arms and controllers of the l user rest by their side.
As for accuracy, the researchers, Karan Ahuja, Vivian Shen, Cathy Mengying Fang, Nathan Riopelle, Andy Kong and Chris Harrison, say millimeter tracking accuracy is probably out of the question for this type of system, but tracking accuracy centimeter is likely on the table, which may be sufficient for many VR use cases. For their particular prototype, the system had an “average 3D joint error of 6.98cm”, although the researchers say this should be “considered the floor of performance, not the ceiling”, given the limited time they spent optimizing the system.
With full body tracking, the legs are finally a viable part of the experience. This is desirable not only to make your avatar more realistic to others, but also to incorporate your lower body into the experience, add to the immersion, and provide another input for players to use in-game.
The researchers not only created a full tracking model for the system, but they also performed prototype experiments to show how tracked legs can add to gameplay. They showed a hockey goalie experience, where players can block the puck with any part of their body; a “body shape matching” experience, where players match the shape of an incoming wall to walk through; and even a “Feet Saber” game, where players cut blocks with their hands and feet.
– – – – –
So, could we see full body tracking from headsets like Magic Leap 2 and Project Cambria? It’s hard to say at this point; although the controllers seem to do their own upside-down tracking, the controllers’ cameras seem to be pointing mostly away from the user.
But maybe a future headset – or just an improved controller – could make it happen.
Wherever these headsets land, this research shows that convenient, low-friction full-body tracking on standalone VR headsets may not be that far off. Combined with the ability to perform highly realistic face tracking, the autonomous headsets of the future will dramatically increase the embodiment felt in virtual reality.