I am currently running an experiment, that will become a series of experiments, in the CAVE lab at UCL. The experiments require full body tracking and a flexible 3D platform that runs in the CAVE cluster. In this post I’m going to discuss the technical approach to running Unity and a Kinect in UCL’s four-walled CAVE-like system. Some topics and technologies I’m going to cover include Unity, MiddleVR, OpenNI and NITE, KinectSDK, and VRPN.
Body Tracking
The key requirement of the body tracking system is that it is practical to use and provides robust data in order to puppeteer an avatar in real-time. We have two professional motion capture systems set up in the CAVE, but I don’t consider them to be practical in this case: asking each participant to wear a motion capture suit, fitted with retro-reflective markers and calibrating the system for person’s body size is a process that takes ~20 minutes; not ideal for an experimental procedure that takes only 10 minutes!
Enter the Kinect. Most people reading this are likely to know a thing or two about this device. It’s a rare instance of a consumer device superseding, for certain uses, technologies at research labs. It doesn’t provide as high-fidelity or low-latency skeletal motion capture data as professional systems can, but this is counterbalanced by its practicality as it is markerless and features instant calibration. For experimental studies into avatar embodiment, it doesn’t provide any new modes of interaction, but its sheer practicality opens up the field. I wouldn’t have attempted to run these experiments had it not been for the Kinect.
So, I positioned a Kinect in our CAVE as shown in the images below. It’s mounted at a 23° declining angle. It took a bit of trial and error and wrangling to fit it into place in order to give a good view of the CAVE space, but it is able to capture the majority of the “usable” space.
On to the software. In the Kinect development community, there are two options for real-time skeletal tracking: NITE, which is a module of the OpenNI framework, and Microsoft’s official KinectSDK. There are pros and cons for both of these solutions that I’m not going to go into now. The crux of the argument is that OpenNI provides a more complete depth camera and NUI framework, while the KinectSDK provides better quality skeletal data. After a lot of experimentation with NITE, I decided that the jitter, particularly due to the angle at which the Kinect must be mounted in the CAVE, was unacceptable. When viewing an avatar from a third-person perspective, and in an experiment studying body ownership, this was quite a limitation. However, I did like the functionality of the OpenNI framework. If only there was a bridge to enable the KinectSDK skeleton tracking to operate as an OpenNI module. Something like this maybe! Using the kinect-mssdk-openni-bridge, Microsoft’s skeleton can be loaded as an OpenNI node. This means you can benefit from the functionality of OpenNI, and the quality of the KienctSDK skeletal tracking.
CAVE Rendering Platform
The VE community has long suffered from lack of easy to use software. That idea itself may be an oxymoron, but, following 30-odd years of pioneering research, the proliferation of “VR-like” devices into the home, I think that the community itself can be considered mature now. Arguably there isn’t a silver bullet when it comes to a VR software platform: systems vary in their setup, and different sensory devices require require particular operating environments. Recently, Unity has garnered a lot of attention in the VE community, and I have increasingly seen it being used to run experiments in non-immersive (i.e. no surrounding display, stereoscopy, or head tracking) systems.
Unity is quite unique in terms of being a free to use and fully-featured 3D game engine (others include UDK and CryEngine). It’s friendly to both designers and developers, importing assets from common 3D pipeline tools is easy, scripting can be done in C#, Javasript or Boo, supports custom libraries, and is able to deploy the same code to most the major platforms. Of course there is no VR rendering library for Unity as it is aimed quite squarely at the games market rather than at the simulation or VR industries. I did some research into how we could run Unity on our Windows cluster that featuring nVidia Quadro cards and CrystalEyes active shutter glasses before coming across a post in Sébastien “VR Geek” Kuntz’s blog. Turns out that for the last year and a half, he had been hard at work on a VR plugin for Unity.
Enter MiddleVR. Sébastien visited UCL following the JVRC 2011 conference, this year held in Nottingham. Following a morning of installation, configuration, and calibration of MiddleVR to the CAVE’s parameters and tracking system, our Unity projects can now be visualised in the head-tracked, stereoscopic, and surrounding 3D. Head and hand tracking is achieved using an Intersense IS-900, which runs on a separate machine from the cluster using VRPN. Here is a video of the system in use:
Putting it All Together
In summary, we are now able to run Unity in the CAVE. Using the OpenNI wrapper for Unity, together with the kinect-mssdk-openni-bridge to enable the KinectSDK skeleton tracking, we now have a very powerful platform for all kinds of experimentation into virtual embodiment, NUIs, and VEs in general. Note that the 23° declining angle of the mounted Kinect must be accounted for within Unity. The easiest way to do this is apply a -23° rotation to the avatar’s pre-calibrated root joint. Finally, here’s a quick video of me in the cave, embodied as an avatar with a tail. Why I’m doing this will have to wait for another post!
