Yesterday I looked through the eyes of a robot.

It was a NAO robot fitted with a 3D printed set of goggles, so that the robot has two real cameras on its head (the eyes of the NAO robot are not in fact cameras). I was in another room wearing an Oculus Rift headset. The Oculus was hooked up to the NAO’s goggle cameras, so that I could see through those cameras – in stereo vision.

But it was even better than that. The head positioning system of the Oculus headset was also hooked up to the robot, so I could turn my head and – in sync – the robot’s head moved. And I was standing in front of a Microsoft Kinect that was tracking my arm movements. Those movements were being sent to the NAO, so by moving my arms I was also moving the robot’s arms.

All together this made for a pretty compelling immersive experience. I was able look down while moving my (robot) arms and see them pretty much where you would expect to see your own arms. The illusion was further strengthened when Peter placed a mirror in front of the NAO robot, so I could see my (robot) self moving in the mirror. Then it got weird. Peter asked me to open my hand and placed a paper cup into it. I instinctively looked down and was momentarily shocked not to see the cup in my hand. That made me realise how quickly – within a couple of minutes of donning the headset – I was adjusting to the new robot me.

This setup, developed here in the BRL by my colleagues Paul Bremner and Pete Gibbons, is part of a large EPSRC project called Being There. Paul and Peter are investigating the very interesting question of how humans interact with and via teleoperated robots, in shared social spaces. I think teleoperated robot avatars will be hugely important in the near future – more so than fully autonomous robots. But our robot surrogates will not look like a younger buffer Bruce Willis. They will look like robots. How will we interact with these surrogate robots – robots with human intelligences, human personalities – seemingly with human souls? Will they be treated with the same level of respect that would be accorded their humans if they were actually there, in the flesh? Or be despised as voyeuristic; an uninvited webcam on wheels?

Here is a YouTube video of an earlier version of this setup, without the goggles or Kinect:

I didn’t get the feeling of what it is like to be a robot, but it’s a step in that direction.

OAK, which stands for ‘Observation and Access with Kinect’, is a software application for use by people with severe disabilities. It has been developed by RCAST at the University of Tokyo, in collaboration with Microsoft Japan, and is available from assist-i corporation.

“The Kinect camera detects, in three dimensions, where people are and what posture they have. We’ve used that capability to develop a system called Air Switch. This system creates virtual switches in the air. For example, if a person has a cervical injury, and can only move from the neck up, you can put a switch next to their head. The switch can be worked by any part of the body, so it can be used by moving the head, like this.”

“Air Switch works by detecting how far away an object is. The system also has a Motion History capability, which detects color changes. For example, if Motion History detects that you’re moving your mouth, the system can put a switch near your mouth. In fact, I’m moving my mouth quite a lot right now, and this system can detect motion with millimeter precision. So, for people who’ve been thought of as totally immobile until now, this system could open up a lot more possibilities, based on scientific data, such as shoulder movements, for example.”

Another option is called Face Switch, which keeps detecting facial movements even if the neck moves to some extent. It can be set so that the switch activates when you open your mouth, or it can also be controlled by sticking out your tongue, blinking or changing the orientation of your face.

“This system was released in April. It’ll soon be available in North America and Europe, too.”

1.  iRobot Roomba or Create
2. ROS Electric installed on Ubuntu Linux and configured according to the tutorials
3. Some way of connecting the robot to your computer (1, 2, or modify a USB-to-serial connector to interface with the mini-DIN port). For teleoperation, you can use the connected computer, another computer, smart phone, tablet, iPad, or anything with a browser supporting web sockets.

Let’s get into the specifics. With ROS installed, you’ll need a few more packages that aren’t included in the base install. Rosbridge is a solid middleware package that is going to enable communication with ROS through a webpage. It can do a lot more, but this satisfies our goal for now. Let’s install that (from Brown screencast):

> sudo apt-get install ros-electric-brown-remotelab

You’ll also need a package to enable ROS to communicate with the robot. The package, at its most basic functionality, will listen for Twist messages (ROS-speak for motor drive commands), and convert it to the low-level serial commands for the roomba. (Though this tutorial focuses on roomba/create, you can use any robot with an existing ROS package, or write your own!) You have three options, based on your roomba model:

1. iRobot Roomba 500 series or up (uses iRobot OI spec [pdf])
2. iRobot Roomba pre-500 series (uses iRobot SCI spec [pdf])
3. iRobot Create (uses iRobot OI spec)

> mkdir ros_workspace
> cd ros_workspace
> mkdir myStacks
> cd myStacks
> git clone https://github.com/dpiet/robotsindc-ros-pkg

Be sure to rosmake. Check to make sure the directory (or parent directory) is included in the ROS package path:

> echo ROS_PACKAGE_PATH

If it’s not, add it:

> export ROS_PACKAGE_PATH=$ROS_PACKAGE_PATH:/home/david/ros_workspace

There’s one more piece we need in order to control the robot through a webpage: the webpage. Fortunately, there are plenty of rosbridge examples out there and a javascript novice like me was able to hack together a simple page with keyboard inputs and button inputs. Download the page (or you already have it if you cloned the git repo) to your web server directory (if you’re running one) or to you /home and you need only modify the IP address to point to the computer controlling the robot. And that’s it!

Here’s the procedure for putting this altogether at execution:

In a separate terminal for each line, run the following commands:

> roscore
> rosrun roomba_400_series roomba400_light_node (assumes robot is attached to /dev/ttyUSB0)
> rosrun rosbridge rosbridge.py
(Optional) if you don't have a webserver, cd to your /home directory with the teleop.html file and use the magic of python:
> python -m SimpleHTTPServer

1. Grab your iPad (or other device), make sure it’s on the same wifi local network as the robot computer, and load up the page. If you’re running SimpleHTTPServer, type this in: http://[robot ip address here]:8000/teleop.html
2. If all goes according to plan you’ll get a rudimentary screen with buttons for directional control and a log showing connection status messages
3.  Happy roomba driving!

To recap, we connected a roomba to a laptop running: ROS, the open source Robot Operating System from Willow Garage; Rosbridge, a convenient middleware package from Brown University; and a web server (in this case Python’s SimpleHTTPServer). We were then able to load up a webpage with some javascript from a laptop, android smartphone, or iPad, with each able to drive the roomba with basic forward/reverse/left/right commands. Cool.

By using ROS and a few open source packages, we were able to get going with a moving robot via teleoperation with a simple user interface relatively quickly, or at least much quicker than if we were to program everything from scratch. So the benefit has been the ability to stand up a teleoperation robot in short order through code reuse. A criticism is that we used a fairly heavy framework, with a lot of pieces working under the hood, which means a lot of upfront work to read system documentation/tutorials and review examples. But as I argued above, the upfront cost is worth it, because now the robot is based on software modules with a vibrant community of contributors that facilitates code reuse without you having to reinvent the wheel or become a path-planning expert.

To take advantage of the upfront work we put in to learning ROS, we can now easily extend the first part of the tutorial and add a video feed to our web interface so we can now teleoperate remotely. Here’s what we need to add:
//

• The mjpeg_server ROS package (to install, download the package or svn checkout. cd into the directory and run “rosmake –rosdep-install”)
• Microsoft Kinect ROS package using OpenNI drivers (to install, run “sudo apt-get install ros-electric-openni-kinect”)
• A new html file with some additional javascript

The following video demonstrates the first run after integrating all the components on the robot. Not too bad, but there’s definitely room for improvement. Next steps will be to add a roslaunch file to our repo to simplify startup, accelerometer controls for navigation, and further testing to identify sources of lag.

Here’s the procedure for getting the demo running. Be sure to change the robot’s IP address as appropriate. Please note: this demo is currently compatible with ROS Electric, not the latest version, Fuerte.

In a separate terminal for each line, run the following commands:

> roscore
> rosrun roomba_400_series roomba400_light_node (assumes robot is attached to /dev/ttyUSB0)
> rosrun rosbridge rosbridge.py
> rosrun mjpeg_server mjpeg_server _port:=9091
> roslaunch openni_launch openni.launch
(Optional) if you don't have a webserver, cd to your /home directory with the teleop_video.html file and use the magic of python:
> python -m SimpleHTTPServer

Where can we go from here? We’ve essentially built a Turtlebot, replacing the iRobot Create with the roomba (though we’re obviously missing the convenient mounting hardware). So there are plenty of resources for integrating advanced behaviors like mapping, autonomous navigation, or even a manipulator arm. In particular, the Pi Robot project has done some interesting demonstrations with the Turtlebot platform.

Have a roomba gathering dust and a Kinect sitting next to your Xbox? Then you have all the major components to build a moving robot.