Gizmodo was recently invited to visit CSIRO’s Data61 team to try one of their fabled Microsoft Hololenses.

Woman wearing a

Gizmodo was recently invited to visit CSIRO’s Data61 team to try one of their fabled Microsoft Hololenses. While we were there, CSIRO’s Matt Adcock gave us a rundown of CSIRO’s history with VR, which goes back long before the Hololens, Oculus Rift and HTC Vive ever existed. Not only VR, however, CSIRO has also been playing with augmented reality and its odd lovechild with traditional VR, something they call ‘augmented virtuality’.

What Is Augmented Virtuality?

Augmented virtuality is the name of a third category that joins both augmented and virtual reality. While virtual reality is the replacement of reality with an entirely virtual world and augmented reality is the act of bringing virtual objects into the real world, augmented virtuality refers to bringing real world objects into a virtual space. While the consumer side of VR is seeing a little of this, with technology such as the HTC Vive’s room-scale technology, CSIRO has already developed a number of applications for augmented virtuality.

HTC augmented reality

One was implemented in a partnership with Rio Tinto at their remote working facility. CSIRO built a VR application that controlled one of the rock breaking machines on site, with a virtual model of the machine that operators could view in VR. However operators still had to see the rocks they were breaking, as a large part of the job of operating these machines involves selecting the rocks that need to be crushed, through visual inspection. Real time images of the rocks being processed were incorporated into this program, allowing the machines to be operated remotely from Rio Tinto’s Perth centre.

While this tech has a huge number of uses for a huge number of fields, one of the other applications of augmented virtuality being developed by the CSIRO is a kind of virtual classroom. In one example, students can use virtual reality devices to tour a computer generated representation of the historical site they’re learning about, while their teacher is inserted into the space to offer instruction, or switched out to a live 360 degree video of the classroom.

User Interfaces and Proprioception

Image by North Sullivan Photography / CSIRO

Image by North Sullivan Photography / CSIRO

Proprioception is something that’s of key importance for both VR and AR — its a sixth sense (or so to speak) that every person possesses, their natural ability to locate their body and its movements in space. Proprioception is the reason why you can walk down the street without looking at your feet, or why you can touch a finger to your nose even with your eyes closed (go on, try it). It’s also a big problem to do with presence and interaction in VR — when you aren’t technically occupying a physical space, but a virtual one.

One of the earliest solutions that CSIRO had for this question of proprioception was around long before the Hololens had even been dreamed up: the haptic workbench. This is a workbench with a monitor suspended above it, a reflective screen for the operator to view the monitor’s feed below that, and a haptic input arm below that, for the operator to interact with whatever is displayed on the screen.

This system has been used in a number of different areas already, from analysing drill core data to training medical practitioners in surgeries that are otherwise incredibly hard to practice without an actual human body to work with.

The introduction of the Microsoft Hololens changes the makeup of the haptic workbench, of course. In fact, the only part of the setup that need remain if the Hololens is integrated is the haptic arm itself.

Using VR For Remote Coworking

remote coworking - augmented reality

As evidenced by some of the Hololens’s promotional images, remote co-working is one of the perceived uses of the device. This would allow a knowledgeable expert to guide an unskilled person in a task remotely, without having to be there in person. In the image above a father (presumably) helps his daughter fix a plumbing problem through the Hololens, giving visual directions that appear in her field of view.

CSIRO haven’t let their former lack of Hololens technology stop them from achieving this tech already, however. One co-working system involved having laser projected images guide an unskilled person in carrying out a skilled job, with visual directions much like Dad’s arrow and circle in the image above. Without a type of headset technology, laser projectors were used instead, providing bright and clear directions.

This projection method only really worked on 2D planes however, and didn’t take into account the direction the person on location was viewing the object in question from. Other technologies used simulated 3D projections and even drew upon Microsoft’s earlier technology to help — the Kinect depth-sensing camera.

A Revolution In 3D Scanning

3D scanning

After being somewhat of a failure in the gaming industry, the Kinect has found a surprising lot of use in both development and research. While it’s often used as a budget friendly 3D scanner and motion capture device for indie game developers, CSIRO’s Data61 is working with more precise scanning instruments of their own invention.

Each of the scanners boasted by Data61 had its own specific purpose and use — one was the first micro 3D scanner, able to scan much much smaller objects than traditional 3D scanners, which was then used to catalogue the National Insect Collection. Another is able to easily scan objects in full colour, while one can even scan thermal images in 3D.

The Consumer Age Of VR Is Upon Us

It’s clear that Data61 hasn’t let a lack of technology stop their adventures in VR, but the new consumer technology coming into the market now is only expediting their research. Matt Adcock compares VR hardware to the situation of graphics cards some years back. The processors originally used by the CSIRO were so huge you could almost fit a person inside.

Now, the equivalent in processing power is able to be packed inside the thinnest of smartphones — though these huge advances weren’t made until the technology started to come into the mainstream. Now, the same thing is happening with VR and AR technology — the gaming audience is driving the cost of this tech down dramatically.

In fact, even the development tools for the Hololens reflect this — one of the main supported platforms is the free game development software Unity. And with Microsoft, we’re seeing not only the Hololens, but the entire Windows Holographic ecosystem: a platform designed for ‘holographic computing’ that it hopes to integrate with many more devices.

While Data61 is still very much experimenting with the new technology, it’s clear they’ve got big plans for it. One potential use that Adcock suggested was for CSIRO’s bushfire research unit — with the Hololens, researchers could view a 3D projection of fire and smoke patterns, spread across a desk or boardroom table. With a new way of visualising and analysing 3D data, this technology could even end up saving lives.

With the Hololens being sent out to developers and researchers, the HTC Vive and the Oculus Rift getting a consumer release and the world coming to see the promise of VR in industry, education and life, it’ll be interesting to see what CSIRO and Data61 do with their strong background in this field. It’s a brave new augmented world out there — or at least, it will be soon.