UPDATE: New video of a collaborative, cloud-based mapping experiment. Mapping is essential for mobile robots and a cornerstone of many more robotics applications that require a robot to interact with its physical environment. It is widely considered the most difficult perceptual problem in robotics, both from an algorithmic but also from a computational perspective. Mapping essentially requires solving a huge optimization problem over a large amount of images and their extracted features. This requires beefy computers and high-end graphics cards – resulting in power-hungry and expensive robots.

Update

Here is a video of a collaborative mapping experiment using Rapyuta, the open source cloud robotics framework. Instead of just compressing each frame of the stream, we now run a dense visual odometry algorithm to produce keyframes and these keyframes are eventually sent to the cloud for optimization. This technique is a very natural way to compress since keyframes are produced only when the robot has moved a certain distance or angle. The Cloud, in addition to optimization, now also does merging of maps produced by multiple robots. The code is open under the Apache 2.0 license at https://github.com/rapyuta/rapyuta-mapping.

Together with our RoboEarth colleagues Dominique Hunziker, Dhananjay Sathe, Mayank Singh, we have set up an inexpensive, light weight robot so that it can perform full 3D mapping in real-time by offloading heavy computation to the RoboEarth Cloud Engine (nickname: Rapyuta) (see previous Robohub article). In the video below Master student Vlad Usenko shows how robots can sidestep this problem.

How it works
What actually happens here? As you can see from the video and the figure below, the robot is equipped with an RGBD sensor (ASUS Xtion PRO), an ARM-based single board computer (Odriod-U2), and an off-the-shelf wireless dongle. The robot transmits RGB images (QVGA resolution) and depth images (QVGA resolution) at 30 FPS to Rapyuta with a frame by frame compression (JPEG for RGB, and PNG for depth images). The single board computer is only used for compression, communication, and low-level control of the robot – the heavy computation all happens off-board. The optimized map shown at the end of the video is built by manually triggering the optimization process in the cloud after the demo (i.e., not in real time).

The main purpose of this demonstrator is to showcase that today’s wireless technologies allow robots to outsource computationally heavy data processing to the cloud. As shown in the figure below, and as I’ve laid out in an earlier article on Cloud Robotics, bandwidth is a key driver for cloud-based robot services and applications.

Most actual mapping scenarios call for a slightly different split in work-load: You’d want to perform visual odometry locally and only send keyframes to Rapyuta for global optimization. The good news is that we already supports this setup: Rapyuta provides the corresponding software packages to run on ODroid boards along with the necessary cloud computing infrastructure.

What’s next?
This demonstration stops short of actually using the data returned from the cloud: A map is created and available in real time, but not actively used to control the robot. Using the map to control the robot, e.g. to improve exploration, is one logical extension.

Another natural next step would be to use the computational resources in the cloud for localizing in an existing map. Imagine a robot entering a new building and trying to get its bearings. Such localization in an existing map requires performing a search over the entire map – essentially a very large set of images and their features – to find the best match between what the robot is seeing now and a part of the map.

But the cloud can be used to do much more interesting things than improve single robot navigation. Multiple robots can instantaneously share their maps, so applications like multi-robot mapping become much easier. This implies using the cloud as more than a pure computational resource – it becomes a medium to store, share, and collectively improve data. This is interesting for static scenes, but things get even more interesting for dynamic scenes, for example when multiple robots track dynamic objects in the map. With WIRE, some of my RoboEarth colleagues have developed algorithms to do just that. We are currently working on integrating this functionality with Rapyuta and the RoboEarth database and will make it available as a Rapyuta cloud service later this year.

Many of the key advantages of using this Cloud Robotics approach to mapping will only become apparent when maps are brought together with the vast amounts of knowledge required to understand environments. Performing mapping in the cloud not only allows the creation of maps but also the ability to understand them: By bringing computation close to the knowledge required to make sense of all of a robot’s sensor information, Cloud Robotics offers robots a very powerful way to understand the world around them. When working together, cloud computing frameworks like Rapyuta and online knowledge bases like RoboEarth can allow us to make robots that are both smart and inexpensive.

Additional information
For further information, see:

• Technical details and tutorial for this setup (available early July)
• Video of recent live Cloud Robotics Mapping demo at ROSCon 2013
• Cloud Robotics Workshop at IROS 2013 in November, Tokyo
• Homepage of the RoboEarth Cloud Engine (codename “Rapyuta”)
• RoboEarth homepage

Questions?
Feel free to post in the comments below.

Cloud robotics, a shorthand for the idea of leveraging the Internet for robots, offers unprecedented opportunities for robot learning. Apart from using the World Wide Web for faster communication or faster computation, a key opportunity is to allow robots to create and collaboratively update shared knowledge repositories. Hosted in a shared cloud storage infrastructure, such knowledge bases for robots could enable robots to cope with the complexities of human environments and offer a simple, powerful way for life-long robot learning.

As part of RoboEarth my colleagues and I have been working towards this goal by creating something like a “Wikipedia for robots”. By sharing knowledge about objects, maps, and actions in a common framework, RoboEarth allows robots with different hardware and software to exchange – and collectively improve – their knowledge.

The goal of the European-Commission-funded initiative is to develop proof-of-concept demonstrations that show how cloud repositories like RoboEarth’s databases can greatly speed up robot learning and how they may ultimately allow robots to perform well beyond their preprogrammed behaviors.

As one of its first demonstrators, the team has shown that robots with different hardware can use RoboEarth to share maps. Another demonstrator allowed a complex service robot to download hardware-independent object models from the RoboEarth database and use them for perception and manipulation. And a third demonstrator has shown how a service robot can identify, download, and use high-level task descriptions to serve a drink.

Since the start of RoboEarth in late 2009, a number of other groups have begun work on allowing robots to use large, online knowledge repositories. Google announced the formation of a Cloud Robotics team at its Google I/O conference last year. Its main thrust is leveraging existing Google Web services. In a joint talk with Willow Garage Inc., the new Google team also introduced the first pure Java implementation of Willow Garage’s Robot Operating System (ROS). The software, currently in alpha testing, is intended to allow integration of Google’s Android operating system for mobile phones with ROS. Finally, Motoman launched its Web-based remote monitoring service, MyMotoman, in August 2011. It not only enables customers to monitor and update their robots’ status via mobile devices, but also uses historical data to enhance its service offering with predictive maintenance trend data on critical situations. Further initiative are in the works, but, unfortunately, few of them are publicly available (the core RoboEarth software packages are available online).

The above examples mark a wider trend toward harnessing Internet technology for robotics. And this is not surprising: Both the personal computing and mobile phone industries are intimately connected to robotics. Personal computers continue to underpin robot software and act as an important enabler by providing ever-increasing computing power. Mobile phone technology, on the other hand, continues to drive down prices for sensors, including cameras, accelerometers, gyroscopes, GPS sensors, as well as for enabling technologies like wireless networks. In addition, personal computers, mobile phones, and tablets are natural entry points for human-robot interaction for developers as well as B2B clients (think of Telepresence robots) and, increasingly, B2C customers (think of toys like Romotive’s Romo). Both fields have benefitted immensely from Internet connectivity, raising the question if the Internet may, in time, have a similar effect on robotics.

I think there is a likely starting point for such a transformation: The performance of solutions to the field of robotics’ main challenge — robot vision — is tightly coupled with the available resources for computation and data storage. Web services that allow outsourcing computation and data storage to the cloud can offer clear cost benefits for connected robots by replacing expensive local processing capacity and battery power with more powerful cloud services, resulting in cheaper, lighter robots.

With its Goggles mobile phone app, Google has an interesting first version of such an image cloud service. Snap a picture with your smartphone, and you will receive related information from Google product search for objects, translation services for text, or geographic information for landmarks within seconds. Although the current system is aimed at humans and has severe limitations for robotics (e.g., it cannot parse scenes; the returned information is not grounded), it points the way to similarly powerful services for robots.

Vision is not the only robot task that could greatly benefit from web services. Speech-to-text-to-speech services could vastly improve human-robot interaction. Optical character recognition could help robots understand text in all its forms. And any roboticist who has used the turn-by-turn directions offered by a navigation system can appreciate how useful a similar service could be for outdoor or indoor robot navigation.

A transformation like the one the Internet heralded in the personal computing and mobile phone industries, however, is unlikely to result from mere adoption and adaptation of old technology. Just as the killer application for mobile phones was not email, it seems unlikely to me that cloud robotics will be driven by an app store. The app store’s business model, with app sales feeding product sales feeding app sales requires a significant number of users and developers of a common platform for a successful kick-start. This seems difficult to achieve for robotics. However, people are trying and have had some first success.

Another candidate for such a transformation is the use of pooled knowledge to improve robot learning. Kiva Systems, in North Reading, Mass., has pioneered this approach for a homogeneous multirobot system (it is no coincidence that one of Kiva Systems’ co-founders is one of my RoboEarth colleagues) and, as mentioned above, RoboEarth has already made first demonstrations for heterogeneous systems.

As you’d expect, not all knowledge robots can learn is easily exchangeable in a joint knowledge repository. Raw trajectory data or sensor and actuator parameters are often too hardware specific to be exchanged successfully. However, a fair amount of knowledge robots learn can be exchanged: For example, maps, CAD models of objects, and articulation models of doors and drawers have been successfully learnt and shared between different robots. A particularly interesting area for learning are links between shared information, such as where is the fridge (map coordinates), what does it look like (object recognition model) and how do I open it (object articulation model). Another are probabilities, such as given that I see a table, bed, and chair, where would I most likely find a pillow? Robots are well-suited for this type of learning not least because, unlike humans, they are capable of rapid, systematic, and accurate data collection. This capability provides unprecedented opportunities for obtaining consistent and comparable data sets as well as for performing large-scale systematic analysis and data mining.

While the future of Cloud Robotics remains uncertain, my RoboEarth colleagues and I agree that human environments are too nuanced and complicated to be summarized within a limited set of specifications. Any solution to program robots to cope with our daily environments is therefore likely to involve some sort of collective robot learning. Sharing knowledge in a cloud infrastructure like RoboEarth allows robots to perform such learning much faster.

Connecting robots worldwide is a challenging task beyond the scope of any individual initiative. RoboEarth is currently collaborating with related research initiatives in Europe and the United States, as well as with a number of companies through its Industrial Advisory Committee. To join the discussion, feel free to have a look at the RoboEarth website and to get in touch via info@roboearth.org.

Note: A previous version of this article appeared in Robotics Trends.
Credits: Images by Carolina Flores // ETH Zurich // RoboEarth.

(Our CONGRATULATIONS to ALL especially Silicon Valley team OLogic !)

We are very happy to announce the results of the Cloud Robotics Hackathon 2012. We combined all the project evaluations from the judges and chose the top three projects which will be awarded awesome prices.

#3

In third place we find an excellent project that uses a complex robot behaviours to encourage discipline and cleanliness in humans. We really enjoyed the humorous presentation, usage of the cloud and autonomous navigation.

Team No Pain, No Game from Montreal wins the third prices comprises of the following items:

• Ticket to the StartUp Festival
• A GitHub six-month bronze plan

#2

The second place goes to a very impressive project involving autonomous navigation, path planning, communication via the robotic cloud, a crafty game board, and funky smiley faces on a smartphone. The complexity, the attention to detail, and the overall completeness of the projects were most impressive and showed the awesome hacking skills of the team.

Team Oddwerx from Santa Clara wins the second prices comprises of the following items:

• Ticket to the StartUp Festival
• A GitHub one-year bronze plan

#1

The first place goes to the most cloudy of all projects. One that involves shiny lights, felt and lots of colours. This project uses a very interesting communication and collaboration system between robots that helps them fetch messages from the cloud even if they don’t have direct access to the web. This project also tackles the interesting area of education and regards though the use of a game. We really enjoyed the technical complexity, robot communication and human interaction featured in the project

Team LightingBots from Montreal wins the second prices comprises of the following items:

• A Karotz robot
• Ticket to the StartUp Festival
• A GitHub one-year silver plan and a GitHub T-shirt for each team member

More Projects

Unfortunately, we cannot give prices to everyone although we think every team did great and truly impressed us with what they could come up in a single weekend. We invite you to have a look at all their amazing project presentations.

 Keep Hacking Robots, Keep Hacking the Future.

See you Next Year!

More than 60 people showed up to the Hackathon, making us the largest event next to the Montreal Hackerspace which had 100 on opening night. We had 10 teams working over the weekend and have posted demos of working cloud enabled robots from the final session. There were events in 7 locations around the globe, so each location had a presentation and semifinal and the mothership in Montreal will be choosing the final winner today.

The two Santa Clara/Silicon Valley teams voted best and moving through to the finals are: Oddwerx (by OLogic) and HomeCast.

Oddwerx used their own robot(s) to play tic tac toe as your avatar or proxy, so that you and your friend can be located anywhere but control the robot moves via cloud. Hopefully the Oddwerx robot will be on kickstarter soon.

HomeCast used the DFRover and developed a ‘house sitter/home watchdog’ to roam the house while you are away, checking on temperature, humidity, videoing pets/intruders etc. The possibilities are limited only by the range of sensors.

 One of the semifinalists from Montreal is RobotGrrl with her RoboBrrd mesh network. It’s great! http://robotgrrl.com/blog/tag/hackathon/

1. Build robot

2. Connect robot to cloud

3. Make robot do something useful or fun