Real-Time Remote Guidance Using the Firefly MV

Remote Tech Support Using Firefly MV

In our global economy, our daily interactions involve people and companies from all over the world.  For companies supporting worldwide customers and colleagues, this can be a challenge.  For example, a technician could be in Bangledesh trying to fix a problem on their production line but the technical support engineer is in Switzerland.  What’s the best way to ensure consistent, quality production, and support worldwide and remotely?  For complex technical problems, phone calls and emails, even web conferencing, are nowhere close to replacing having the technical support engineer onsite.

Troubleshooting with real-time images

XMReality (Linköping, Sweden) has taken the problem of remote support to a new level that virtually places engineers in remote locations to work with on-site technicians. Using low-bandwidth communication protocols developed for military remote support and guidance applications, XMReality’s XMExpert uses helmet-mounted cameras, displays, microphones, and a wearable computer to allow remote experts to see and hear what on-site technicians experience in real time. Conversely, the technician can both see and hear the expert, guided by real-time video images of the expert’s hands as they direct the technician to solving the problem.

“The technician is instructed in the same way as if the expert were standing beside them,” explains Niklas Rengfors, Sales Manager at XMReality. “They can also discuss problems together, and when the technician is unclear about the instructions, the remote expert can grab the tool or show the technician what buttons to push to conduct important tests and repairs.”

No Small Feat

XMExpert is essentially a high-tech helmet for the technician connected to a nearby laptop with Internet connection that communicates over a low-bandwidth protocol to remote experts with their own camera and computer. A camera and microphone on the on-site technician’s helmet captures his or her visual reality and transmits that image to the remote expert. The expert sees the same image on his computer screen as the technician. The XMExpert software compresses sound and picture from the two units, transmitting them digitally via the Internet, satellite, or mobile network and uniting them into a common image that is displayed on a helmet-mounted display (HMD). Through this setup, the technician in the field not only sees his own reality but, superimposed on the view through his helmet-mounted glasses, he also sees the remote expert pointing at objects and demonstrating how to carry out tasks, such as handling tools, spare parts, and conducting tests.

The right camera

XMReality has used FLIR cameras since the first XMExpert system was fielded in 2012, thanks to the FLIR’s expertise in making small, robust cameras capable of high video rates. Weight and performance are critical issues for helmet-mounted displays because heavy cameras will tire the technician, and choppy video can make a person motion sick looking through a helmet-mounted display.

After evaluating a variety of low-cost “web cams,” XMReality settled on FLIR’s 0.3 MP Firefly MV FMVU-03MTC-CS global shutter CMOS camera because of its small size, low weight, and output of 60 frames per second. “We felt that 60 Hz was the optimal solution because these units are worn by people moving around,” says Torbjorn Gustafsson, co-founder of XMReality. “We needed to get optimum picture and also for the engineer in field not to feel sick when moving around with the video goggles in front of his or her eyes.”

Streaming Without Buffering at 128 kbit/s

In operation, the images from the Firefly MV are fed over USB 2.0 to a wearable computer running Windows 7, as well as FLIR’s FlyCapture SDK. The FlyCapture software overlays the image from the remote expert‘s camera onto the image from the technician’s field-of-view camera. The combined image is then fed back to the technician’s helmet display.

“All we need is an Internet connection of some sort, and both the technician and expert can work together as if both were standing in the same room,” Gustafsson says. “Coming from a military application, the emphasis has been laid on running at low broadband connections and work out in remote places. XMExpert is superior to all other ways of communication as the system allows the technician and the expert to interact in a natural way. The system focuses on the realistic real-time experience as picture and sound are always synchronized, even if the broadband connection is inadequate. The system works well even at speeds as low as 128 kbit/s.”

Learn more at XMReality