RBO Team from TU Berlin wins the Amazon Picking Challenge at ICRA 2015. Photo Credit: RBO.

Capturing and processing camera and sensor data and recognizing various shapes to determine a set of robotic actions is conceptually easy. Yet Amazon challenged the industry to do a selecting and picking task robotically and 28 teams from around the world rose to it.

Perception isn’t just about cameras and sensors. Software has to convert the data and infer as to what it “sees”. In the case of the Amazon Picking Challenge held last week at the IEEE International Conference on Robotics & Automation (ICRA), each team’s robot was to pick from a shopping list of consumer items of varying shapes and sizes – from pencils, to toys, tennis balls, cookies and cereal boxes – which were haphazardly stored on shelves, and then place their selected items in a bin. They could use any robot, mobile or not, and any arm and end-of-arm grasping tool or tools to accomplish the task.

It’s tricky for robots using sensors to identify and locate objects that can be confused by plastic packaging within the shelf or storage area. Rodney Brooks, of iRobot, MIT and Rethink Robotics fame, often speaks of an industry-wide aspirational goal regarding perception in robotics: “If we were only able to provide the visual capabilities of a 2-year old child, robots would quickly get a lot better.” That is what this contest is all about.

Software has to first identify the item to pick and then figure out the best way to grab it and move it out of the storage area. Amazon, with its acquisition of Kiva Systems, has mastered bringing goods to the picker/packer and now wants to automate the remaining process of picking the correct goods from the shelves and placing them in the packing box, hence their Amazon Picking Challenge.

The top three winners were the teams from Technical University (TU) – Berlin, with 148 points; MIT in 2nd place with 88 points; and the 3rd place finisher (Oakland U and Dataspeed), which only got 35 points. Teams were scored on how many items were correctly selected, picked and placed.

Many commercial companies with proprietary software for just this type of application (such as Tenessee-based Universal Robotics and their Neocortex Vision System, and Silicon Valley startup Fetch Robotics who were premiering their new Fetch and Freight system at the same ICRA conference) chose not to enter because the terms of the challenge included that the software be open sourced.

Team RBO from TU-Berlin wrote their own vision system software and will soon be working on a paper on the subject. They used a Barrett WAM arm because it was the most flexible device for this task amongst the arms that they had access to in their lab. They used a vacuum cleaner tool augmented with a suction cup and a vacuum cleaner to power the suction. For a base, they decided they needed to go mobile and used an old Nomadic Technologies platform, which they upgraded to fit the needs of the contest. Nomadic no longer exists. It was acquired by 3Com in 2000. TU’s photo at right shows robot’s camera view and random placement of items in cubby holes. Watch a video of their winning run here:

Noriko Takiguchi, a Japanese reporter for RoboNews.net, who was at the contest, observed the TU team and said that their approach was torque-based plus position control of the arm and the mobile base, consequently they had good torque control that that gave them flexibility in choosing where to place the suction cup and how much suction to apply.

Team RBO received a 1st place prize of $20,000 plus travel costs for the equipment and team members.

With the aim of implementing a television service which allows viewers to touch virtual objects, NHK is developing a tactile system which applies stimuli to five points on one finger, making objects feel more real than when using previous systems.

“This device assists people with a visual disability, through what’s called the tactile or kinesthetic sense. It communicates a 3D shape or 2D graph of an object shown on a TV, such as a work of art, as a sensation felt by the hand.”

“The system detects spatial position data for the fingertip in 1/1000 of a second. The finger moves freely if there isn’t anything present, but when it approaches something, it becomes unable to go into the object. By tracing with your finger like this, you can feel that there’s a continuous rough surface there. Also, the jagged feeling of the tooth makes the sensation similar to that of touching an actual object.”

In previous systems, a force was applied to the entire fingertip, and when angular parts of an object were presented, the fingertip tended to deviate from the virtual object, making it hard to understand the object’s shape. But this device makes the corners of an object, in particular, feel natural, by generating a force at five points.

“Now, Domo-kun’s body is one object, while the tooth is another object. The texture can be changed for each object. Here, the tooth alone has been made slightly hard. In that case, if we create a different object using CG, we can do this kind of thing.”

“This is slightly different from a texture that’s actually felt by the skin. For instance, if you hold a pen, you can perceive textures through the pen. This system reproduces that kind of sensation.”

“From now on, we’d like to simplify this device. We want to extend the range of movement of the fingers so that curved surfaces are also recognizable, while keeping the advantages of the current version.”

Professor Takashi Kawai’s lab at Waseda University’s School of Fundamental Science and Engineering is conducting research on a cross-modal perception technology employing multi-sensory integration in which participants perceive tactile sensation from visual stimulation. In a visual-evoked “minute tactile sensation” presentation system prototype, a very simple mechanism is used to enable the phenomenon of tactile sensation perception, despite the lack of any physical contact, simply by viewing a video image.

“One element of the system is the use of a see-through head-mounted display to present a stereoscopic image. Unlike when typically viewing an image on a display, it is important here for the image to be viewed in space. An extremely important aspect in constructing this system is that the participant is able to observe in a natural manner a part of his or her body such as the hands making contact with the image.”

First, a commercially available see-through head-mounted display is worn. An image of an object floating about 30cm in front of the eyes becomes visible. Presently this object is displayed as a ball of fire or ice to represent temperature sensation. The participant places his hands as if touching the viewed object so that the ball of fire or ice is observed resting on the hands. By adjusting the hands so that the object is touching the palms and then by gently shaking the head left and right so that a slow-speed motion is perceived, he can experience a subtle tactile sensation such as a slight breeze or cool or warm sensation.

“Feasible areas of application are entertainment or communication, or as a new form of media for producing visual content to induce tactile illusion. Or for more serious applications, there are various possibilities for consideration including rehabilitation as well as training and education where this type of system can be used effectively.”

With the prototype system, the research group has discovered that the participant’s perceived tactile sensation will change qualitatively, such as a sensation of warmth or cold, according to the content of the video image. The group will continue to conduct research on video content with associated tactile sensation and search for effective utilization of tactile sensation in fields such as education and social welfare.