Research team : Maxence LEVEZIEL, Wissem HAOUAS, Michaël GAUTHIER, Guillaume J. LAURENT, Redwan DAHMOUCHE.

By Redwan Dahmouche

Speed and precision are two major issues in robotics and in Industry of the Future (also known as Industry 4.0). Within this framework, RoMoCo research team of AS2M department at FEMTO-ST Institute has developed MiGriBot, a miniature robot able to perform 720 pick-and-place operations of sub-millimeter objects per minute. The results of this research work have been published in Science Robotics.

These performances are made possible thanks to its architecture, that allows it to grip and manipulate micro-objects barely visible to the naked eye (from 40 micrometers to several hundred micrometers). In fact, where other microrobots have a rigid end-effector, MiGriBot is based on a principle with an articulated end. This articulated end allows to drive a microgripper without any wire or embedded actuator. The second advantage of this robot is that all its degrees of mobility, including the ones from the microgripper on the articulated end, are operated from the base of the robot, making its mobile parts very lightweight. Finally, its robotic structure occupies a surface of only 20x20mm2. This level of compactness is achieved by using Silicon for the rigid elements, a polymer (Polydimethylsiloxane – PDMS) as flexible joints and piezoelectric actuators equipped with position sensors. MiGriBot is therefore lighter, more compact and faster than existing robotic micro-manipulators.

MiGriBot holding a cylindrical ruby with a diameter of 700µm and a thickness of 200µm.

While the fastest industrial pick-and-place robots do not exceed 250 cycles per minute, the combination of all the features of this robot: soft joints, small footprint, integrated gripping, lightweight structure and closed-loop control of the fast actuators allows MiGriBot to reach 720 pick-and-place operations per minute with about one micrometer accuracy. In the blink of an eye, MiGRiBot will have manipulated 5 micro-objects, which means that it will have approached, grabbed, moved and released an object 5 times successively.

This robot will be used to assemble Micro-Electro-Mechanical and Optical Systems (MEMS/MOEMS) used in the electronics industry, where the production throughput is increasingly high. Thanks to its speed and compactness, more than 2000 robots can be placed in 1m2 to perform more than one million operations per second. Increasing work rates will improve the productivity and competitiveness of manufacturers, which may encourage the relocation of production to Europe, America and countries with high labor costs. Applications in watch industry, medical instrumentation, aerospace, and other fields are also possible.

MiGriBot holding another parallel structure showing the capacity of the microrobot to manipulate heavy objects.

This work was funded by the ANR MiniSoRo project (ANR-19-CE10-0004) and by Grand Besançon Métropole.

Research team: Maxence LEVEZIEL, Wissem HAOUAS, Michaël GAUTHIER, Guillaume J. LAURENT, Redwan DAHMOUCHE (associate professor at université de Franche-Comté, project leader and head of the research team: redwan.dahmouche@univ-fcomte.fr).

• PAPER – MiGriBot: A miniature parallel robot with integrated gripping for high-throughput micromanipulation. Maxence Leveziel, Wissem Haouas, Guillaume J. Laurent, Michaël Gauthier, and Redwan Dahmouche. Science Robotics, 7(69).

Yu Sun, Professor of Computer Science and Engineering at the University of South Florida, created and organized the Robotic Grasping and Manipulation Competition. Yu talks about the impact robots will have in domestic environments, the disparity between industry and academia showcased by competitions, and the commercialization of research.

Yu Sun

Yu Sun is a Professor in the Department of Computer Science and Engineering at the University of South Florida (Assistant Professor 2009-2015, Associate Professor 2015-2020, Associate Chair of Graduate Affairs 2018-2020). He was a Visiting Associate Professor at Stanford University from 2016 to 2017, and received his Ph.D. degree in Computer Science from the University of Utah in 2007. Then he had his Postdoctoral training at Mitsubishi Electric Research Laboratories (MERL), Cambridge, MA (2007-2008) and the University of Utah (2008-2009).

He initiated the IEEE RAS Technical Committee on Robotic Hands, Grasping, and Manipulation and served as its first co-Chair. Yu Sun also served on several editorial boards as an Associate Editor and Senior Editor, including IEEE Transactions on Robotics, IEEE Robotics and Automation Letters (RA-L), ICRA, and IROS.

Links

• Download mp3 (49.0 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

MIT researchers have developed a system that enables a robot to learn a new pick-and-place task based on only a handful of human examples. This could allow a human to reprogram a robot to grasp never-before-seen objects, presented in random poses, in about 15 minutes. Courtesy of the researchers

By Adam Zewe | MIT News Office

With e-commerce orders pouring in, a warehouse robot picks mugs off a shelf and places them into boxes for shipping. Everything is humming along, until the warehouse processes a change and the robot must now grasp taller, narrower mugs that are stored upside down.

Reprogramming that robot involves hand-labeling thousands of images that show it how to grasp these new mugs, then training the system all over again.

But a new technique developed by MIT researchers would require only a handful of human demonstrations to reprogram the robot. This machine-learning method enables a robot to pick up and place never-before-seen objects that are in random poses it has never encountered. Within 10 to 15 minutes, the robot would be ready to perform a new pick-and-place task.

The technique uses a neural network specifically designed to reconstruct the shapes of 3D objects. With just a few demonstrations, the system uses what the neural network has learned about 3D geometry to grasp new objects that are similar to those in the demos.

In simulations and using a real robotic arm, the researchers show that their system can effectively manipulate never-before-seen mugs, bowls, and bottles, arranged in random poses, using only 10 demonstrations to teach the robot.

“Our major contribution is the general ability to much more efficiently provide new skills to robots that need to operate in more unstructured environments where there could be a lot of variability. The concept of generalization by construction is a fascinating capability because this problem is typically so much harder,” says Anthony Simeonov, a graduate student in electrical engineering and computer science (EECS) and co-lead author of the paper.

Simeonov wrote the paper with co-lead author Yilun Du, an EECS graduate student; Andrea Tagliasacchi, a staff research scientist at Google Brain; Joshua B. Tenenbaum, the Paul E. Newton Career Development Professor of Cognitive Science and Computation in the Department of Brain and Cognitive Sciences and a member of the Computer Science and Artificial Intelligence Laboratory (CSAIL); Alberto Rodriguez, the Class of 1957 Associate Professor in the Department of Mechanical Engineering; and senior authors Pulkit Agrawal, a professor in CSAIL, and Vincent Sitzmann, an incoming assistant professor in EECS. The research will be presented at the International Conference on Robotics and Automation.

Grasping geometry

A robot may be trained to pick up a specific item, but if that object is lying on its side (perhaps it fell over), the robot sees this as a completely new scenario. This is one reason it is so hard for machine-learning systems to generalize to new object orientations.

To overcome this challenge, the researchers created a new type of neural network model, a Neural Descriptor Field (NDF), that learns the 3D geometry of a class of items. The model computes the geometric representation for a specific item using a 3D point cloud, which is a set of data points or coordinates in three dimensions. The data points can be obtained from a depth camera that provides information on the distance between the object and a viewpoint. While the network was trained in simulation on a large dataset of synthetic 3D shapes, it can be directly applied to objects in the real world.

The team designed the NDF with a property known as equivariance. With this property, if the model is shown an image of an upright mug, and then shown an image of the same mug on its side, it understands that the second mug is the same object, just rotated.

“This equivariance is what allows us to much more effectively handle cases where the object you observe is in some arbitrary orientation,” Simeonov says.

As the NDF learns to reconstruct shapes of similar objects, it also learns to associate related parts of those objects. For instance, it learns that the handles of mugs are similar, even if some mugs are taller or wider than others, or have smaller or longer handles.

“If you wanted to do this with another approach, you’d have to hand-label all the parts. Instead, our approach automatically discovers these parts from the shape reconstruction,” Du says.

The researchers use this trained NDF model to teach a robot a new skill with only a few physical examples. They move the hand of the robot onto the part of an object they want it to grip, like the rim of a bowl or the handle of a mug, and record the locations of the fingertips.

Because the NDF has learned so much about 3D geometry and how to reconstruct shapes, it can infer the structure of a new shape, which enables the system to transfer the demonstrations to new objects in arbitrary poses, Du explains.

Picking a winner

They tested their model in simulations and on a real robotic arm using mugs, bowls, and bottles as objects. Their method had a success rate of 85 percent on pick-and-place tasks with new objects in new orientations, while the best baseline was only able to achieve a success rate of 45 percent. Success means grasping a new object and placing it on a target location, like hanging mugs on a rack.

Many baselines use 2D image information rather than 3D geometry, which makes it more difficult for these methods to integrate equivariance. This is one reason the NDF technique performed so much better.

While the researchers were happy with its performance, their method only works for the particular object category on which it is trained. A robot taught to pick up mugs won’t be able to pick up boxes or headphones, since these objects have geometric features that are too different than what the network was trained on.

“In the future, scaling it up to many categories or completely letting go of the notion of category altogether would be ideal,” Simeonov says.

They also plan to adapt the system for nonrigid objects and, in the longer term, enable the system to perform pick-and-place tasks when the target area changes.

This work is supported, in part, by the Defense Advanced Research Projects Agency, the Singapore Defense Science and Technology Agency, and the National Science Foundation.

• PAPER – Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation Anthony Simeonov, Yilun Du, Andrea Tagliasacchi, Joshua B. Tenenbaum, Alberto Rodriguez, Pulkit Agrawal, and Vincent Sitzmann.

Dr. Carolyn Matl, Research Scientist at Toyota Research Institute, explains why Interactive Perception and soft tactile sensors are critical for manipulating challenging objects such as liquids, grains, and dough. She also dives into “StRETcH” a Soft to Resistive Elastic Tactile Hand, a variable stiffness soft tactile end-effector, presented by her research group.

Carolyn Matl

Carolyn Matl is a research scientist at the Toyota Research Institute, where she works on robotic perception and manipulation with the Mobile Manipulation Team. She received her B.S.E in Electrical Engineering from Princeton University in 2016, and her Ph.D. in Electrical Engineering and Computer Sciences at the University of California, Berkeley in 2021. At Berkeley, she was awarded the NSF Graduate Research Fellowship and was advised by Ruzena Bajcsy. Her dissertation work focused on developing and leveraging non-traditional sensors for robotic manipulation of complicated objects and substances like liquids and doughs.

Carolyn Matl’s Related Research Videos

Links

• StRETcH: a Soft to Resistive Elastic Tactile Hand
• Carolyn Matl’s personal website
• Download mp3 (48.7 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

The robot seen here can’t see the human arm during the entire dressing process, yet it manages to successfully get a jacket sleeve pulled onto the arm. Photo courtesy of MIT CSAIL.

By Steve Nadis | MIT CSAIL

Robots are already adept at certain things, such as lifting objects that are too heavy or cumbersome for people to manage. Another application they’re well suited for is the precision assembly of items like watches that have large numbers of tiny parts — some so small they can barely be seen with the naked eye.

“Much harder are tasks that require situational awareness, involving almost instantaneous adaptations to changing circumstances in the environment,” explains Theodoros Stouraitis, a visiting scientist in the Interactive Robotics Group at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).

“Things become even more complicated when a robot has to interact with a human and work together to safely and successfully complete a task,” adds Shen Li, a PhD candidate in the MIT Department of Aeronautics and Astronautics.

Li and Stouraitis — along with Michael Gienger of the Honda Research Institute Europe, Professor Sethu Vijayakumar of the University of Edinburgh, and Professor Julie A. Shah of MIT, who directs the Interactive Robotics Group — have selected a problem that offers, quite literally, an armful of challenges: designing a robot that can help people get dressed. Last year, Li and Shah and two other MIT researchers completed a project involving robot-assisted dressing without sleeves. In a new work, described in a paper that appears in an April 2022 issue of IEEE Robotics and Automation, Li, Stouraitis, Gienger, Vijayakumar, and Shah explain the headway they’ve made on a more demanding problem — robot-assisted dressing with sleeved clothes. 

The big difference in the latter case is due to “visual occlusion,” Li says. “The robot cannot see the human arm during the entire dressing process.” In particular, it cannot always see the elbow or determine its precise position or bearing. That, in turn, affects the amount of force the robot has to apply to pull the article of clothing — such as a long-sleeve shirt — from the hand to the shoulder.

To deal with obstructed vision in trying to dress a human, an algorithm takes a robot’s measurement of the force applied to a jacket sleeve as input and then estimates the elbow’s position. Image: MIT CSAIL

To deal with the issue of obstructed vision, the team has developed a “state estimation algorithm” that allows them to make reasonably precise educated guesses as to where, at any given moment, the elbow is and how the arm is inclined — whether it is extended straight out or bent at the elbow, pointing upwards, downwards, or sideways — even when it’s completely obscured by clothing. At each instance of time, the algorithm takes the robot’s measurement of the force applied to the cloth as input and then estimates the elbow’s position — not exactly, but placing it within a box or volume that encompasses all possible positions. 

That knowledge, in turn, tells the robot how to move, Stouraitis says. “If the arm is straight, then the robot will follow a straight line; if the arm is bent, the robot will have to curve around the elbow.” Getting a reliable picture is important, he adds. “If the elbow estimation is wrong, the robot could decide on a motion that would create an excessive, and unsafe, force.” 

The algorithm includes a dynamic model that predicts how the arm will move in the future, and each prediction is corrected by a measurement of the force that’s being exerted on the cloth at a particular time. While other researchers have made state estimation predictions of this sort, what distinguishes this new work is that the MIT investigators and their partners can set a clear upper limit on the uncertainty and guarantee that the elbow will be somewhere within a prescribed box.   

The model for predicting arm movements and elbow position and the model for measuring the force applied by the robot both incorporate machine learning techniques. The data used to train the machine learning systems were obtained from people wearing “Xsens” suits with built-sensors that accurately track and record body movements. After the robot was trained, it was able to infer the elbow pose when putting a jacket on a human subject, a man who moved his arm in various ways during the procedure — sometimes in response to the robot’s tugging on the jacket and sometimes engaging in random motions of his own accord.

This work was strictly focused on estimation — determining the location of the elbow and the arm pose as accurately as possible — but Shah’s team has already moved on to the next phase: developing a robot that can continually adjust its movements in response to shifts in the arm and elbow orientation. 

In the future, they plan to address the issue of “personalization” — developing a robot that can account for the idiosyncratic ways in which different people move. In a similar vein, they envision robots versatile enough to work with a diverse range of cloth materials, each of which may respond somewhat differently to pulling.

Although the researchers in this group are definitely interested in robot-assisted dressing, they recognize the technology’s potential for far broader utility. “We didn’t specialize this algorithm in any way to make it work only for robot dressing,” Li notes. “Our algorithm solves the general state estimation problem and could therefore lend itself to many possible applications. The key to it all is having the ability to guess, or anticipate, the unobservable state.” Such an algorithm could, for instance, guide a robot to recognize the intentions of its human partner as it works collaboratively to move blocks around in an orderly manner or set a dinner table. 

Here’s a conceivable scenario for the not-too-distant future: A robot could set the table for dinner and maybe even clear up the blocks your child left on the dining room floor, stacking them neatly in the corner of the room. It could then help you get your dinner jacket on to make yourself more presentable before the meal. It might even carry the platters to the table and serve appropriate portions to the diners. One thing the robot would not do would be to eat up all the food before you and others make it to the table.  Fortunately, that’s one “app” — as in application rather than appetite — that is not on the drawing board.

This research was supported by the U.S. Office of Naval Research, the Alan Turing Institute, and the Honda Research Institute Europe.

• PAPER – Set-based State Estimation with Probabilistic Consistency Guarantee under Epistemic Uncertainty. Shen Li, Theodoros Stouraitis, Michael Gienger, Sethu Vijayakumar, and Julie A. Shah. Published at IEEE Robotics and Automation Letters, 2022. https://doi.org/10.48550/arXiv.2110.09584

Lilly interviews Stephanie Schneider, a PhD candidate at Stanford working on unconventional space robotics. Schneider explains her work on Reachbot, a long-reach crawling and anchoring robot, which repurposes extendable booms for mobile manipulation. They discuss the challenges and exciting elements of robotic prototyping for low-gravity or otherwise unique environments.

Stephanie Schneider

Stephanie Schneider is a Ph.D. candidate in Aeronautics and Astronautics in the Autonomous Systems Laboratory at Stanford, working with Professor Marco Pavone. She received her BS in Mechanical Engineering from Cornell University in 2014. Stephanie’s research interests include real-time spacecraft motion-planning, grasping and manipulation in space, and adaptive control for autonomous robotics.

Links

• Download mp3 (26.6 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

Congratulations to Huy Ha and Shuran Song who have won the CoRL 2021 best system paper award!

Their work, FlingBot: the unreasonable effectiveness of dynamic manipulations for cloth unfolding, was highly praised by the judging committee. “To me, this paper constitutes the most impressive account of both simulated and real-world cloth manipulation to date.”, commented one of the reviewers.

Below, the authors tell us more about their work, the methodology, and what they are planning next.

What is the topic of the research in your paper?

In my most recent publication with my advisor, Professor Shuran Song, we studied the task of cloth unfolding. The goal of the task is to manipulate a cloth from a crumpled initial state to an unfolded state, which is equivalent to maximizing the coverage of the cloth on the workspace.

Could you tell us about the implications of your research and why it is an interesting area for study?

Historically, most robotic manipulation research topics, such as grasp planning, are concerned with rigid objects, which have only 6 degrees of freedom since their geometry does not change. This allows one to apply the typical state estimation – task & motion planning pipeline in robotics. In contrast, deformable objects could bend and stretch in arbitrary directions, leading to infinite degrees of freedom. It’s unclear what the state of the cloth should even be. In addition, deformable objects such as clothes could experience severe self occlusion – given a crumpled piece of cloth, it’s difficult to identify whether it’s a shirt, jacket, or pair of pants. Therefore, cloth unfolding is a typical first step of cloth manipulation pipelines, since it reveals key features of the cloth for downstream perception and manipulation.

Despite the abundance of sophisticated methods for cloth unfolding over the years, they typically only address the easy case (where the cloth already starts off mostly unfolded) or take upwards of a hundred steps for challenging cases. These prior works all use single arm quasi-static actions, such as pick and place, which is slow and limited by the physical reach range of the system.

Could you explain your methodology?

In our daily lives, humans typically use both hands to manipulate cloths, and with as little as a single high velocity fling or two, we can unfold an initially crumpled cloth. Based on this observation, our key idea is simple: Use dual arm dynamic actions for cloth unfolding.

FlingBot is a self-supervised framework for cloth unfolding which uses a pick, stretch, and fling primitive for a dual-arm setup from visual observations. There are three key components to our approach. First is the decision to use a high velocity dynamic action. By relying on cloths’ mass combined with a high-velocity throw to do most of its work, a dynamic flinging policy can unfold cloths much more efficiently than a quasi-static policy. Second is a dual-arm grasp parameterization which makes satisfying collision safety constraints easy. By treating a dual-arm grasp not as two points but as a line with a rotation and length, we can directly constrain the rotation and length of the line to ensure arms do not cross over each other and do not try to grasp too close to each other. Third is our choice of using Spatial Action Maps, which learns translational, rotational, and scale equivariant value maps, and allows for sample efficient learning.

What were your main findings?

We found that dynamic actions have three desirable properties over quasi-static actions for the task of cloth unfolding. First, they are efficient – FlingBot achieves over 80% coverage within 3 actions on novel cloths. Second, they are generalizable – trained on only square cloths, FlingBot also generalizes to T-shirts. Third, they expand the system’s effective reach range – even when FlingBot can’t fully lift or stretch a cloth larger than the system’s physical reach range, it’s able to use high velocity flings to unfold the cloth.

After training and evaluating our model in simulation, we deployed and finetuned our model on a real world dual-arm system, which achieves above 80% coverage for all cloth categories. Meanwhile, the quasi-static pick & place baseline was only able to achieve around 40% coverage.

What further work are you planning in this area?

Although we motivated cloth unfolding as a precursor for downstream modules such as cloth state estimation, unfolding could also benefit from state estimation. For instance, if the system is confident it has identified the shoulders of the shirt in its state estimation, the unfolding policy could directly grasp the shoulders and unfold the shirt in one step. Based on this observation, we are currently working on a cloth unfolding and state estimation approach which can learn in a self-supervised manner in the real world.

About the authors

Huy Ha is a Ph.D. student in Computer Science at Columbia University. He is advised by Professor Shuran Song and is a member of the Columbia Artificial Intelligence and Robotics (CAIR) lab.

Shuran Song is an assistant professor in computer science department at Columbia University, where she directs the Columbia Artificial Intelligence and Robotics (CAIR) Lab. Her research focuses on computer vision and robotics. She’s interested in developing algorithms that enable intelligent systems to learn from their interactions with the physical world, and autonomously acquire the perception and manipulation skills necessary to execute complex tasks and assist people.

Find out more

• Read the paper on arXiv.
• The videos of the real-world experiments and code are available here, as is a video of the authors’ presentation at CoRL.
• Read more about the winning and shortlisted papers for the CoRL awards here.

Researchers at MIT have developed a fully-integrated robotic arm that fuses visual data from a camera and radio frequency (RF) information from an antenna to find and retrieve objects, even when they are buried under a pile and fully out of view. Credits: Courtesy of the researchers

By Adam Zewe | MIT News Office

A busy commuter is ready to walk out the door, only to realize they’ve misplaced their keys and must search through piles of stuff to find them. Rapidly sifting through clutter, they wish they could figure out which pile was hiding the keys.

Researchers at MIT have created a robotic system that can do just that. The system, RFusion, is a robotic arm with a camera and radio frequency (RF) antenna attached to its gripper. It fuses signals from the antenna with visual input from the camera to locate and retrieve an item, even if the item is buried under a pile and completely out of view.

The RFusion prototype the researchers developed relies on RFID tags, which are cheap, battery-less tags that can be stuck to an item and reflect signals sent by an antenna. Because RF signals can travel through most surfaces (like the mound of dirty laundry that may be obscuring the keys), RFusion is able to locate a tagged item within a pile.

Using machine learning, the robotic arm automatically zeroes-in on the object’s exact location, moves the items on top of it, grasps the object, and verifies that it picked up the right thing. The camera, antenna, robotic arm, and AI are fully integrated, so RFusion can work in any environment without requiring a special set up.

In this video still, the robotic arm is looking for keys hidden underneath items. Credits: Courtesy of the researchers

While finding lost keys is helpful, RFusion could have many broader applications in the future, like sorting through piles to fulfill orders in a warehouse, identifying and installing components in an auto manufacturing plant, or helping an elderly individual perform daily tasks in the home, though the current prototype isn’t quite fast enough yet for these uses.

“This idea of being able to find items in a chaotic world is an open problem that we’ve been working on for a few years. Having robots that are able to search for things under a pile is a growing need in industry today. Right now, you can think of this as a Roomba on steroids, but in the near term, this could have a lot of applications in manufacturing and warehouse environments,” said senior author Fadel Adib, associate professor in the Department of Electrical Engineering and Computer Science and director of the Signal Kinetics group in the MIT Media Lab.

Co-authors include research assistant Tara Boroushaki, the lead author; electrical engineering and computer science graduate student Isaac Perper; research associate Mergen Nachin; and Alberto Rodriguez, the Class of 1957 Associate Professor in the Department of Mechanical Engineering. The research will be presented at the Association for Computing Machinery Conference on Embedded Networked Senor Systems next month.

Sending signals

RFusion begins searching for an object using its antenna, which bounces signals off the RFID tag (like sunlight being reflected off a mirror) to identify a spherical area in which the tag is located. It combines that sphere with the camera input, which narrows down the object’s location. For instance, the item can’t be located on an area of a table that is empty.

But once the robot has a general idea of where the item is, it would need to swing its arm widely around the room taking additional measurements to come up with the exact location, which is slow and inefficient.

The researchers used reinforcement learning to train a neural network that can optimize the robot’s trajectory to the object. In reinforcement learning, the algorithm is trained through trial and error with a reward system.

“This is also how our brain learns. We get rewarded from our teachers, from our parents, from a computer game, etc. The same thing happens in reinforcement learning. We let the agent make mistakes or do something right and then we punish or reward the network. This is how the network learns something that is really hard for it to model,” Boroushaki explains.

In the case of RFusion, the optimization algorithm was rewarded when it limited the number of moves it had to make to localize the item and the distance it had to travel to pick it up.

Once the system identifies the exact right spot, the neural network uses combined RF and visual information to predict how the robotic arm should grasp the object, including the angle of the hand and the width of the gripper, and whether it must remove other items first. It also scans the item’s tag one last time to make sure it picked up the right object.

Cutting through clutter

The researchers tested RFusion in several different environments. They buried a keychain in a box full of clutter and hid a remote control under a pile of items on a couch.

But if they fed all the camera data and RF measurements to the reinforcement learning algorithm, it would have overwhelmed the system. So, drawing on the method a GPS uses to consolidate data from satellites, they summarized the RF measurements and limited the visual data to the area right in front of the robot.

Their approach worked well — RFusion had a 96 percent success rate when retrieving objects that were fully hidden under a pile.

“We let the agent make mistakes or do something right and then we punish or reward the network. This is how the network learns something that is really hard for it to model,” co-author Tara Boroushaki, pictured here, explains. Credits: Courtesy of the researchers

“Sometimes, if you only rely on RF measurements, there is going to be an outlier, and if you rely only on vision, there is sometimes going to be a mistake from the camera. But if you combine them, they are going to correct each other. That is what made the system so robust,” Boroushaki says.

In the future, the researchers hope to increase the speed of the system so it can move smoothly, rather than stopping periodically to take measurements. This would enable RFusion to be deployed in a fast-paced manufacturing or warehouse setting.

Beyond its potential industrial uses, a system like this could even be incorporated into future smart homes to assist people with any number of household tasks, Boroushaki says.

“Every year, billions of RFID tags are used to identify objects in today’s complex supply chains, including clothing and lots of other consumer goods. The RFusion approach points the way to autonomous robots that can dig through a pile of mixed items and sort them out using the data stored in the RFID tags, much more efficiently than having to inspect each item individually, especially when the items look similar to a computer vision system,” says Matthew S. Reynolds, CoMotion Presidential Innovation Fellow and associate professor of electrical and computer engineering at the University of Washington, who was not involved in the research. “The RFusion approach is a great step forward for robotics operating in complex supply chains where identifying and ‘picking’ the right item quickly and accurately is the key to getting orders fulfilled on time and keeping demanding customers happy.”

The research is sponsored by the National Science Foundation, a Sloan Research Fellowship, NTT DATA, Toppan, Toppan Forms, and the Abdul Latif Jameel Water and Food Systems Lab.

RE2 Sapien 6M Robot Arm

In this episode, Shihan Lu interviews Travis Schneider, Business Development Manager at RE2 Robotics, focusing on their work on mobile outdoor manipulation. Travis introduces several robotic products and services of RE2, including RE2 Sapien robotic arms and human-centered system integration, and how they can be used to help rather than replace workers. He also shares his thoughts about challenges in human-robot collaboration.

Travis Schneider
Travis Schneider, Business Development Manager at RE2 Robotics (RE2), works to foster commercial B2B partnerships with groups interested in leveraging RE2’s robotic technology for applications in a wide variety of markets, including aviation, construction, energy, and medical. With a background in Mechanical Engineering, Travis has an intimate knowledge of many of the fundamentals associated with robotic systems including electromechanical design, motion control, and associated software.

Links

• RE2 Robotics
• Download mp3 (12.7 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

The Allen Institute for AI (AI2) announced the 3.0 release of its embodied artificial intelligence framework AI2-THOR, which adds active object manipulation to its testing framework. ManipulaTHOR is a first of its kind virtual agent with a highly articulated robot arm equipped with three joints of equal limb length and composed entirely of swivel joints to bring a more human-like approach to object manipulation.

AI2-THOR is the first testing framework to study the problem of object manipulation in more than 100 visually rich, physics-enabled rooms. By enabling the training and evaluation of generalized capabilities in manipulation models, ManipulaTHOR allows for much faster training in more complex environments as compared to current real-world training methods, while also being far safer and more cost-effective.

“Imagine a robot being able to navigate a kitchen, open a refrigerator and pull out a can of soda. This is one of the biggest and yet often overlooked challenges in robotics and AI2-THOR is the first to design a benchmark for the task of moving objects to various locations in virtual rooms, enabling reproducibility and measuring progress,” said Dr. Oren Etzioni, CEO at AI2. “After five years of hard work, we can now begin to train robots to perceive and navigate the world more like we do, making real-world usage models more attainable than ever before.”

Despite being an established research area in robotics, the visual reasoning aspect of object manipulation has consistently been one of the biggest hurdles researchers face. In fact, it’s long been understood that robots struggle to correctly perceive, navigate, act, and communicate with others in the world. AI2-THOR solves this problem with complex simulated testing environments that researchers can use to train robots for eventual activities in the real world.

With the pioneering of embodied AI through AI2-THOR, the landscape has changed for the common good. AI2-THOR enables researchers to efficiently devise solutions that address the object manipulation issue, and also other traditional problems associated with robotics testing.

“In comparison to running an experiment on an actual robot, AI2-THOR is incredibly fast and safe,” said Roozbeh Mottaghi, Research Manager at AI2. “Over the years, AI2-THOR has enabled research on many different tasks such as navigation, instruction following, multi-agent collaboration, performing household tasks, reasoning if an object can be opened or not. This evolution of AI2-THOR allows researchers and scientists to scale the current limits of embodied AI.”

In addition to the 3.0 release, the team is hosting the RoboTHOR Challenge 2021 in conjunction with the Embodied AI Workshop at this year’s Conference on Computer Vision and Pattern Recognition (CVPR). AI2’s challenges cover RoboTHOR object navigation; ALFRED (instruction following robots); and Room Rearrangement.

To read AI2-THOR’s ManipulaTHOR paper: ai2thor.allenai.org/publications

MIT researchers developed a picking robot that combines vision with radio frequency (RF) sensing to find and grasps objects, even if they’re hidden from view. The technology could aid fulfilment in e-commerce warehouses. Credits: Courtesy of the researchers

By Daniel Ackerman | MIT News Office

In recent years, robots have gained artificial vision, touch, and even smell. “Researchers have been giving robots human-like perception,” says MIT Associate Professor Fadel Adib. In a new paper, Adib’s team is pushing the technology a step further. “We’re trying to give robots superhuman perception,” he says.

The researchers have developed a robot that uses radio waves, which can pass through walls, to sense occluded objects. The robot, called RF-Grasp, combines this powerful sensing with more traditional computer vision to locate and grasp items that might otherwise be blocked from view. The advance could one day streamline e-commerce fulfillment in warehouses or help a machine pluck a screwdriver from a jumbled toolkit.

The research will be presented in May at the IEEE International Conference on Robotics and Automation. The paper’s lead author is Tara Boroushaki, a research assistant in the Signal Kinetics Group at the MIT Media Lab. Her MIT co-authors include Adib, who is the director of the Signal Kinetics Group; and Alberto Rodriguez, the Class of 1957 Associate Professor in the Department of Mechanical Engineering. Other co-authors include Junshan Leng, a research engineer at Harvard University, and Ian Clester, a PhD student at Georgia Tech.

As e-commerce continues to grow, warehouse work is still usually the domain of humans, not robots, despite sometimes-dangerous working conditions. That’s in part because robots struggle to locate and grasp objects in such a crowded environment. “Perception and picking are two roadblocks in the industry today,” says Rodriguez. Using optical vision alone, robots can’t perceive the presence of an item packed away in a box or hidden behind another object on the shelf — visible light waves, of course, don’t pass through walls.

But radio waves can.

For decades, radio frequency (RF) identification has been used to track everything from library books to pets. RF identification systems have two main components: a reader and a tag. The tag is a tiny computer chip that gets attached to — or, in the case of pets, implanted in — the item to be tracked. The reader then emits an RF signal, which gets modulated by the tag and reflected back to the reader.

The reflected signal provides information about the location and identity of the tagged item. The technology has gained popularity in retail supply chains — Japan aims to use RF tracking for nearly all retail purchases in a matter of years. The researchers realized this profusion of RF could be a boon for robots, giving them another mode of perception.

“RF is such a different sensing modality than vision,” says Rodriguez. “It would be a mistake not to explore what RF can do.”

RF Grasp uses both a camera and an RF reader to find and grab tagged objects, even when they’re fully blocked from the camera’s view. It consists of a robotic arm attached to a grasping hand. The camera sits on the robot’s wrist. The RF reader stands independent of the robot and relays tracking information to the robot’s control algorithm. So, the robot is constantly collecting both RF tracking data and a visual picture of its surroundings. Integrating these two data streams into the robot’s decision making was one of the biggest challenges the researchers faced.

“The robot has to decide, at each point in time, which of these streams is more important to think about,” says Boroushaki. “It’s not just eye-hand coordination, it’s RF-eye-hand coordination. So, the problem gets very complicated.”

The robot initiates the seek-and-pluck process by pinging the target object’s RF tag for a sense of its whereabouts. “It starts by using RF to focus the attention of vision,” says Adib. “Then you use vision to navigate fine maneuvers.” The sequence is akin to hearing a siren from behind, then turning to look and get a clearer picture of the siren’s source.

With its two complementary senses, RF Grasp zeroes in on the target object. As it gets closer and even starts manipulating the item, vision, which provides much finer detail than RF, dominates the robot’s decision making.

RF Grasp proved its efficiency in a battery of tests. Compared to a similar robot equipped with only a camera, RF Grasp was able to pinpoint and grab its target object with about half as much total movement. Plus, RF Grasp displayed the unique ability to “declutter” its environment — removing packing materials and other obstacles in its way in order to access the target. Rodriguez says this demonstrates RF Grasp’s “unfair advantage” over robots without penetrative RF sensing. “It has this guidance that other systems simply don’t have.”

RF Grasp could one day perform fulfilment in packed e-commerce warehouses. Its RF sensing could even instantly verify an item’s identity without the need to manipulate the item, expose its barcode, then scan it. “RF has the potential to improve some of those limitations in industry, especially in perception and localization,” says Rodriguez.

Adib also envisions potential home applications for the robot, like locating the right Allen wrench to assemble your Ikea chair. “Or you could imagine the robot finding lost items. It’s like a super-Roomba that goes and retrieves my keys, wherever the heck I put them.”

The research is sponsored by the National Science Foundation, NTT DATA, Toppan, Toppan Forms, and the Abdul Latif Jameel Water and Food Systems Lab (J-WAFS).

Nearly all real-world applications of reinforcement learning involve some degree of shift between the training environment and the testing environment. However, prior work has observed that even small shifts in the environment cause most RL algorithms to perform markedly worse. As we aim to scale reinforcement learning algorithms and apply them in the real world, it is increasingly important to learn policies that are robust to changes in the environment.

Robust reinforcement learning maximizes reward on an adversarially-chosen environment.

Broadly, prior approaches to handling distribution shift in RL aim to maximize performance in either the average case or the worst case. The first set of approaches, such as domain randomization, train a policy on a distribution of environments, and optimize the average performance of the policy on these environments. While these methods have been successfully applied to a number of areas (e.g., self-driving cars, robot locomotion and manipulation), their success rests critically on the design of the distribution of environments. Moreover, policies that do well on average are not guaranteed to get high reward on every environment. The policy that gets the highest reward on average might get very low reward on a small fraction of environments. The second set of approaches, typically referred to as robust RL, focus on the worst-case scenarios. The aim is to find a policy that gets high reward on every environment within some set. Robust RL can equivalently be viewed as a two-player game between the policy and an environment adversary. The policy tries to get high reward, while the environment adversary tries to tweak the dynamics and reward function of the environment so that the policy gets lower reward. One important property of the robust approach is that, unlike domain randomization, it is invariant to the ratio of easy and hard tasks. Whereas robust RL always evaluates a policy on the most challenging tasks, domain randomization will predict that the policy is better if it is evaluated on a distribution of environments with more easy tasks.

Prior work has suggested a number of algorithms for solving robust RL problems. Generally, these algorithms all follow the same recipe: take an existing RL algorithm and add some additional machinery on top to make it robust. For example, robust value iteration uses Q-learning as the base RL algorithm, and modifies the Bellman update by solving a convex optimization problem in the inner loop of each Bellman backup. Similarly, Pinto ‘17 uses TRPO as the base RL algorithm and periodically updates the environment based on the behavior of the current policy. These prior approaches are often difficult to implement and, even once implemented correctly, they requiring tuning of many additional hyperparameters. Might there be a simpler approach, an approach that does not require additional hyperparameters and additional lines of code to debug?

To answer this question, we are going to focus on a type of RL algorithm known as maximum entropy RL, or MaxEnt RL for short (Todorov ‘06, Rawlik ‘08, Ziebart ‘10). MaxEnt RL is a slight variant of standard RL that aims to learn a policy that gets high reward while acting as randomly as possible; formally, MaxEnt maximizes the entropy of the policy. Some prior work has observed empirically that MaxEnt RL algorithms appear to be robust to some disturbances the environment. To the best of our knowledge, no prior work has actually proven that MaxEnt RL is robust to environmental disturbances.

In a recent paper, we prove that every MaxEnt RL problem corresponds to maximizing a lower bound on a robust RL problem. Thus, when you run MaxEnt RL, you are implicitly solving a robust RL problem. Our analysis provides a theoretically-justified explanation for the empirical robustness of MaxEnt RL, and proves that MaxEnt RL is itself a robust RL algorithm. In the rest of this post, we’ll provide some intuition into why MaxEnt RL should be robust and what sort of perturbations MaxEnt RL is robust to. We’ll also show some experiments demonstrating the robustness of MaxEnt RL.

Intuition

So, why would we expect MaxEnt RL to be robust to disturbances in the environment? Recall that MaxEnt RL trains policies to not only maximize reward, but to do so while acting as randomly as possible. In essence, the policy itself is injecting as much noise as possible into the environment, so it gets to “practice” recovering from disturbances. Thus, if the change in dynamics appears like just a disturbance in the original environment, our policy has already been trained on such data. Another way of viewing MaxEnt RL is as learning many different ways of solving the task (Kappen ‘05). For example, let’s look at the task shown in videos below: we want the robot to push the white object to the green region. The top two videos show that standard RL always takes the shortest path to the goal, whereas MaxEnt RL takes many different paths to the goal. Now, let’s imagine that we add a new obstacle (red blocks) that wasn’t included during training. As shown in the videos in the bottom row, the policy learned by standard RL almost always collides with the obstacle, rarely reaching the goal. In contrast, the MaxEnt RL policy often chooses routes around the obstacle, continuing to reach the goal for a large fraction of trials.

	Standard RL	MaxEnt RL
Trained and evaluated without the obstacle:
Trained without the obstacle, but evaluated with the obstacle:

Theory

We now formally describe the technical results from the paper. The aim here is not to provide a full proof (see the paper Appendix for that), but instead to build some intuition for what the technical results say. Our main result is that, when you apply MaxEnt RL with some reward function and some dynamics, you are actually maximizing a lower bound on the robust RL objective. To explain this result, we must first define the MaxEnt RL objective: $J_{MaxEnt}(\pi; p, r)$ is the entropy-regularized cumulative return of policy $\pi$ when evaluated using dynamics $p(s’ \mid s, a)$ and reward function $r(s, a)$. While we will train the policy using one dynamics $p$, we will evaluate the policy on a different dynamics, $\tilde{p}(s’ \mid s, a)$, chosen by the adversary. We can now formally state our main result as follows:

$$\min_{\tilde{p} \in \tilde{\mathcal{P}}(\pi)} J_\text{MaxEnt}(\pi; \tilde{p}, r) \ge \exp(J_\text{MaxEnt}(\pi; p, \bar{r}) + \log T.$$

The left-hand-side is the robust RL objective. It says that the adversary gets to choose whichever dynamics function $\tilde{p}(s’ \mid s, a)$ makes our policy perform as poorly as possible, subject to some constraints (as specified by the set $\tilde{\mathcal{P}}$). On the right-hand-side we have the MaxEnt RL objective (note that $\log T$ is a constant, and the function $\exp(\cdots)$ is always increasing). Thus, this objective says that a policy that has a high entropy-regularized reward (right hand-side) is guaranteed to also get high reward when evaluated on an adversarially-chosen dynamics.

The most important part of this equation is the set $\tilde{\mathcal{P}}$ of dynamics that the adversary can choose from. Our analysis describes precisely how this set is constructed and shows that, if we want a policy to be robust to a larger set of disturbances, all we have to do is increase the weight on the entropy term and decrease the weight on the reward term. Intuitively, the adversary must choose dynamics that are “close” to the dynamics on which the policy was trained. For example, in the special case where the dynamics are linear-Gaussian, this set corresponds to all perturbations where the original expected next state and the perturbed expected next state have a Euclidean distance less than $\epsilon$.

More Experiments

Our analysis predicts that MaxEnt RL should be robust to many types of disturbances. The first set of videos in this post showed that MaxEnt RL is robust to static obstacles. MaxEnt RL is also robust to dynamic perturbations introduced in the middle of an episode. To demonstrate this, we took the same robotic pushing task and knocked the puck out of place in the middle of the episode. The videos below show that the policy learned by MaxEnt RL is more robust at handling these perturbations, as predicted by our analysis.

Standard RL	MaxEnt RL

The policy learned by MaxEntRL is robust to dynamic perturbations of the puck (red frames).

Our theoretical results suggest that, even if we optimize the environment perturbations so the agent does as poorly as possible, MaxEnt RL policies will still be robust. To demonstrate this capability, we trained both standard RL and MaxEnt RL on a peg insertion task shown below. During evaluation, we changed the position of the hole to try to make each policy fail. If we only moved the hole position a little bit ($\le$ 1 cm), both policies always solved the task. However, if we moved the hole position up to 2cm, the policy learned by standard RL almost never succeeded in inserting the peg, while the MaxEnt RL policy succeeded in 95% of trials. This experiment validates our theoretical findings that MaxEnt really is robust to (bounded) adversarial disturbances in the environment.

Standard RL	MaxEnt RL	Evaluation on adversarial perturbations

MaxEnt RL is robust to adversarial perturbations of the hole (where the robot
inserts the peg).

Conclusion

In summary, our paper shows that a commonly-used type of RL algorithm, MaxEnt RL, is already solving a robust RL problem. We do not claim that MaxEnt RL will outperform purpose-designed robust RL algorithms. However, the striking simplicity of MaxEnt RL compared with other robust RL algorithms suggests that it may be an appealing alternative to practitioners hoping to equip their RL policies with an ounce of robustness.

Acknowledgements
Thanks to Gokul Swamy, Diba Ghosh, Colin Li, and Sergey Levine for feedback on drafts of this post, and to Chloe Hsu and Daniel Seita for help with the blog.

This post is based on the following paper:

• Maximum Entropy RL (Provably) Solves Some Robust RL Problems.
  Benjamin Eysenbach and Sergey Levine.

Humanoid robot hand. Futuristic cyborg concept.

While modern cameras provide machines with a very well-developed sense of vision, robots still lack such a comprehensive solution for their sense of touch. At ETH Zurich, in the group led by Prof. Raffaello D’Andrea at the Institute for Dynamic Systems and Control, we have developed a tactile sensing principle that allows robots to retrieve rich contact feedback from their interactions with the environment. I recently described our approach in a TEDx talk at the last TEDxZurich. The talk features a tech demo that introduces the novel tactile sensing technology targeting the next generation of soft robotic skins.

Design

The sensing technique is based on a camera that tracks fluorescent particles, which are densely and randomly distributed within a soft, deformable gel. The randomness of the patterns simplifies production of the gel and their density provides strain information at each pixel of the resulting image. In addition, the technique does not make any assumption about the shape of the sensing surface, which can exhibit an arbitrary geometry.

Data Processing

While the images capturing the particles’ motion are intuitive and to some extent visually interpretable, the extraction of accurate physical quantities is challenging. In order to overcome the complexity of modeling the behavior of soft materials in real-time, the information extracted from the images is mapped to the distribution of the applied (shear and pressure) contact forces in a data-driven fashion. Specifically, a neural network is employed, which is trained entirely via accurate finite-element simulations to extract the aforementioned mapping.

Applications

This technology has the potential to impact several application fields, the obvious one being robotic manipulation. In a recent proof-of-concept, we have demonstrated how highly dynamic manipulation tasks can be achieved via the sole use of tactile sensing, as shown in the video below.

In addition, the versatility of this approach makes it suitable for various products beyond the robotics domain. In fact, an artificial sense of touch may find applications in smart prosthetic systems, with the potential to restore tactile sensations to people who have lost limbs.

The environment is constantly changing and the robot has to be able to respond immediately.

Shobhit Yadav, mechatronics engineer smart industries and robotics at TNO

Automatically adapting to the situation
Dennis van der Plas, senior global lead packaging lines at Heineken, says, “We are becoming a high-tech company and attracting more and more technically trained staff. Repetitive tasks – like picking up fallen bottles from the conveyor belt will not provide them job satisfaction.” As part of the SMITZH innovation programme, Heineken and RoboHouse fieldlab, with support from the Netherlands Organisation for Applied Scientific Research (TNO), have developed a solution on the basis of flexible manufacturing: automated handling of unexpected situations.

According to Shobhit Yadav of TNO, flexible manufacturing is one of the most important developments in smart industry. “Today, manufacturing companies mainly produce small series on demand. It means that manufacturers have to be able to make many different products. This can be achieved either with a large number of production lines or with a small number that are flexible enough to adapt.” The Heineken project fell into the second category and involved developing a robot that could recognise different kinds of beer bottles that had fallen over on the conveyor. The robot had to pick them up while the belt was still moving. “The environment is constantly changing and the robot has to be able to respond immediately”, explains Shobhit. “This is a typical example of a flexible production line that automatically adapts to the situation.”

Robotics for a safe and enjoyable working environment
Industrial robots have obviously been around for a while. “The automotive industry deploys robots for welding car parts, whereas our sector uses them for automatically palletising products”, says Dennis. “But with this project we took a different approach. Our starting point was not a question of which robots exist and how they could be used. Instead, we focused on the needs and wishes of the people in the breweries, the operators who control and maintain the machines and how robots could support them in their work.”

The solution, in other words, had to lead not only to process optimisation but also ­– especially – to improved safety and greater job satisfaction. In addition, it would result in Heineken becoming a better employer. It is why Dennis and his colleague Wessel Reurslag, global lead packaging engineering & robotics, asked the operators what they would need to make their work safer and more interesting. One of the use cases that emerged was picking up bottles that had fallen over on the conveyor belt: repetitive but also unsafe as the glass bottles could break.

The lab is the place to meet for anyone involved in robotics.

Wessel Reurslag, global lead packaging engineering & robotics at Heineken

Dennis van der Plas, senior global lead packaging lines at Heineken, has a single minded focus: “Our first priority is to keep our people safe.”

Experimenting without a business case
Heineken initially made contact with RoboHouse field lab through a sponsorship project with X!Delft, an initiative that strengthens corporate innovation and closes the gap between industry and Delft University of Technology. “The lab is the place to meet for anyone involved in robotics”, says Wessel. “It is also linked to SMITZH and thus connected to TNO.”

The parties soon realised that their ambitions overlapped. Heineken was seeking independent advice and both TNO and RoboHouse were looking for an applied research project that focused on flexible manufacturing. “This kind of partnership is very valuable to all involved”, says Shobhit. “SMITZH allows us at TNO to work with current issues in the industry and establish valuable contacts, which makes our research more relevant. In turn, manufacturers have somewhere to go with their questions and problems regarding smart technologies.”

Accessible way to do research and experiment
The guys at Heineken have nothing but praise for the innovative collaboration with TNO and RoboHouse. Dennis says, “The great thing is that all project partners were in it to learn something. At RoboHouse, we had access to the expertise of robotics engineers and state-of-the-art technologies like robotic arms. We supplied the bottle conveyor ourselves and TNO also added knowledge to the mix. It is an accessible way to do research and experiment. This would be a lot more difficult with a business case, which must involve an operational advantage from the start.”

Through close cooperation, we really developed a joint product.

Bas van Mil, mechanical engineer and robot gripper expert at RoboHouse

A joint product by TNO and RoboHouse
TNO and RoboHouse distilled two research goals from the use cases presented by Heineken: enabling real-time robot control and using vision technology to direct the robots with cameras. The main challenge involved devising a solution that could be applied to Heineken’s high-speed packaging lines. TNO worked on the control and movements of the robot, while RoboHouse took on the vision technology aspect. This entailed recognising the fallen bottles, developing the system’s software-based control and building the ‘gripper’ to pick up the bottles.

In many existing robotic systems, […] the robot will do a ‘blind pick.’

Bas van Mil, mechanical engineer and robot gripper expert at RoboHouse

“Communication between the robot and the computer is very important”, explains Bas van Mil, mechanical engineer at RoboHouse. “Our input and TNO’s work were complementary. For example, RoboHouse did not possess Shobhit’s knowledge of control technology, which is indispensable for controlling the robot. Through close cooperation, we really developed a joint product.”

Every millisecond counts
The biggest challenge in detecting and tracking fallen beer bottles is that they never stop moving. As Bas explains, “They do not just move in the direction of the bottle conveyor but can also roll around on the belt itself. In many existing robotic systems, the camera takes a single photo that informs the movements of the robot. The robot will do a ‘blind pick’ with no way of knowing whether anything has changed since the photo was taken. This only works if the environment stays the same – but in this case, it doesn’t.”

The solution involves a system in which the camera and the movements of the robot are constantly connected to each other. Bas says, “Every millisecond counts as the bottle will disappear from view and the robot will still try to pick it up from the spot where it was half a second ago.” A RoboHouse programmer developed the camera software to be as fast and efficient as possible. The field lab even purchased a powerful computer running an advanced AI system especially for the project.

Shobhit Yadav, mechatronics engineer smart industries and robotics at TNO, was crucial for the controlling the robot.

We are receiving feedback as well as requests for the new robotic systems from breweries all over the globe.

Dennis van der Plas, senior global lead packaging lines at Heineken

TNO and RoboHouse then wrote a programme together that determines the speed of the robot from the moment a fallen bottle is detected. This enables the robot to move with the bottle based on its calculated speed. It is what makes this robot so different from existing ones. “The robot responds immediately to changes”, says Shobhit. “In fact, it is 30 per cent faster than the current top speed of Heineken’s bottle conveyors. As a result, it has a wide range of applications and can be used in a variety of environments with different production speeds.”

Smarter thanks to independent partners
Heineken valued not just the successful innovation but also the independent character of TNO and RoboHouse during the development process. Dennis says, “We now have a much better idea of what is technically feasible, what the challenges are and what we can realistically ask of our technical suppliers. Thanks to this project, we can act much more like a smart buyer and make smarter demands of our suppliers. This information is relevant to have, especially as we operate in such an innovative field where we do not just buy parts off the shelf. After all, if I ask too little, I will not get the best out of my project. But if I ask too much, it affects our relationship with the supplier.”

The project has also served as a source of inspiration for Dennis’s colleagues worldwide. “We have been sharing videos and reports from SMITZH on the intranet, building a kind of community within Heineken. We are receiving feedback as well as requests for the new robotic systems from breweries all over the globe.” To meet the demand, Dennis and Wessel want to supply breweries with a ready-to-use version. “We are currently looking for parties that can make the technology available and provide support services.”

Wessel Reurslag, global lead packaging engineering & robotics at Heineken, is already looking for the next step: “A robotic solution that is plug-and-play, flexible and easy to integrate in the brewery.”

Meanwhile, RoboHouse and TNO aim to continue optimising the robot. “This is only the pilot version”, says Bas. “We can still improve its flexibility, for example by installing a different vision module, thereby making the technology even more widely applicable.” Both organisations are therefore looking for use cases in which they can use the same technology to solve other problems. “We are looking at the bigger picture”, explains Shobhit. “This project could serve as a model for similar challenges in other industries.”

Platform for valuable connections
All parties emphasise the benefits of working together and sharing knowledge to achieve a successful and relevant innovation. “There are very few places like SMITZH where manufacturing companies can go with these kinds of questions”, says Shobhit. “SMITZH is quite unique in that sense and is vitally important because it offers a specific platform for appropriate collaborations.” Wessel agrees: “If there is one thing this project has made clear, it is that producers like Heineken, tech companies and knowledge institutions should collaborate more intensively to substantiate these kinds of projects. You won’t find a solution in a PDF or a presentation. There can be no genuine solution until it is made real and put into practice.”

Do you have a similar use case and are you, like Heineken, searching for a practical solution? Or would you first like to learn more about RoboHouse or this specific project? Feel free to contact our lead engineer Bas van Mil, or email SMITZH with questions about its collaboration programme, or contact Shobhit Yadav at TNO more information.

SMITZH and future of work fieldlab RoboHouse
SMITZH is an innovation programme focused on smart manufacturing solutions in West Holland. It brings together supply and demand to stimulate the industrial application of smart manufacturing technologies and help regional companies innovate. Each SMITZH project consists of at least one manufacturing company and a fieldlab. RoboHouse served as the fieldlab in this project.

The post Rapid Robotics for Operator Safety: What a Bottle Picker Can Do appeared first on RoboValley.

The ability for humans to generalize their knowledge and experiences to new situations is remarkable, yet poorly understood. For example, imagine a human driver that has only ever driven around their city in clear weather. Even though they never encountered true diversity in driving conditions, they have acquired the fundamental skill of driving, and can adapt reasonably fast to driving in neighboring cities, in rainy or windy weather, or even driving a different car, without much practice nor additional driver’s lessons. While humans excel at adaptation, building intelligent systems with common-sense knowledge and the ability to quickly adapt to new situations is a long-standing problem in artificial intelligence.

A robot trained to perform a given task in a lab environment may not generalize
to other environments, e.g. an environment with moving disco lights, even
though the task itself remains the same.

In recent years, learning both perception and behavioral policies in an end-to-end framework by deep Reinforcement Learning (RL) has been widely successful, and has achieved impressive results such as superhuman performance on Atari games played directly from screen pixels. Although impressive, it has become commonly understood that such policies fail to generalize to even subtle changes in the environment – changes that humans are easily able to adapt to. For this reason, RL has shown limited success beyond the game or environment in which it was originally trained, which presents a significant challenge in deployment of policies trained by RL in our diverse and unstructured real world.

Generalization by Randomization

In applications of RL, practitioners have sought to improve the generalization ability of policies by introducing randomization into the training environment (e.g. a simulation), also known as domain randomization. By randomizing elements of the training environment that are also expected to vary at test-time, it is possible to learn policies that are invariant to certain factors of variation. For autonomous driving, we may for example want our policy to be robust to changes in lighting, weather, and road conditions, as well as car models, nearby buildings, different city layouts, and so forth. While the randomization quickly evolves into an elaborate engineering challenge as more and more factors of variation are considered, the learning problem itself also becomes harder, greatly decreasing the sample efficiency of learning algorithms. It is therefore natural to ask: rather than learning a policy robust to all conceivable environmental changes, can we instead adapt a pre-trained policy to the new environment through interaction?

Left: training in a fixed environment. Right: training with
domain randomization.

Policy Adaptation

A naïve way to adapt a policy to new environments is by fine-tuning parameters using a reward signal. In real-world deployments, however, obtaining a reward signal often requires human feedback or careful engineering, neither of which are scalable solutions.

In recent work from our lab, we show that it is possible to adapt a pre-trained policy to unseen environments, without any reward signal or human supervision. A key insight is that, in the context of many deployments of RL, the fundamental goal of the task remains the same, even though there may be a mismatch in both visuals and underlying dynamics compared to the training environment, e.g. a simulation. When training a policy in simulation and deploying it in the real world (sim2real), there are often differences in dynamics due to imperfections in the simulation, and visual inputs captured by a camera are likely to differ from renderings of the simulation. Hence, the source of these errors often lie in an imperfect world understanding rather than misspecification of the task itself, and an agent’s interactions with a new environment can therefore provide us with valuable information about the disparity between its world understanding and reality.

Illustration of our framework for adaptation. Left: training before
deployment. The RL objective is optimized together with a self-supervised
objective. Right: adaptation during deployment. We optimize only the
self-supervised objective, using observations collected through interaction
with the environment.

To take advantage of this information we turn to the literature of self-supervised learning. We propose PAD, a general framework for adaptation of policies during deployment, by using self-supervision as a proxy for the absent reward signal. A given policy network $\pi$ parameterized by a collection of parameters $\theta$ is split sequentially into an encoder $\pi_{e}$ and a policy head $\pi_{a}$ such that $a_{t} = \pi(s_{t}; \theta) = \pi_{a} (\pi_{e}(s_{t}; \theta_{e}) ;\theta_{a})$ for a state $s_{t}$ and action $a_{t}$ at time $t$. We then let $\pi_{s}$ be a self-supervised task head and similarly let $\pi_{s}$ share the encoder $\pi_{e}$ with the policy head. During training, we optimize a self-supervised objective jointly together with the RL task, where the two tasks share part of a neural network. During deployment, we can no longer assume access to a reward signal and are unable to optimize the RL objective. However, we can still continue to optimize the self-supervised objective using observations collected through interaction with the new environment. At every step in the new environment, we update the policy through self-supervision, using only the most recently collected observation:

$$s_t \sim p(s_t | a_{t-1}, s_{t-1}) \\
\theta_{e}(t) = \theta_{e}(t-1) – \nabla_{\theta_{e}}L(s_{t}; \theta_{s}(t-1), \theta_{e}(t-1))$$

where L is a self-supervised objective. Assuming that gradients of the self-supervised objective are sufficiently correlated with those of the RL objective, any adaptation in the self-supervised task may also influence and correct errors in the perception and decision-making of the policy.

In practice, we use an inverse dynamics model $a_{t} = \pi_{s}( \pi_e(s_{t}), \pi_e(s_{t+1}))$, predicting the action taken in between two consecutive observations. Because an inverse dynamics model connects observations directly to actions, the policy can be adjusted for disparities both in visuals and dynamics (e.g. lighting conditions or friction) between training and test environments, solely through interaction with the new environment.

Adapting policies to the real world

We demonstrate the effectiveness of self-supervised policy adaptation (PAD) by training policies for robotic manipulation tasks in simulation and adapting them to the real world during deployment on a physical robot, taking observations directly from an uncalibrated camera. We evaluate generalization to a real robot environment that resembles the simulation, as well as two more challenging settings: a table cloth with increased friction, and continuously moving disco lights. In the demonstration below, we consider a Soft Actor-Critic (SAC) agent trained with an Inverse Dynamics Model (IDM), with and without the PAD adaptation mechanism.

Transferring a policy from simulation to the real world. SAC+IDM is a
Soft Actor-Critic (SAC) policy trained with an Inverse Dynamics Model (IDM),
and SAC+IDM (PAD) is the same policy but with the addition of policy
adaptation during deployment on the robot.

PAD adapts to changes in both visuals and dynamics, and nearly recovers the original success rate of the simulated environment. Policy adaptation is especially effective when the test environment differs from the training environment in multiple ways, e.g. where both visuals and physical properties such as object dimensionality and friction differ. Because it is often difficult to formally specify the elements that vary between a simulation and the real world, policy adaptation may be a promising alternative to domain randomization techniques in such settings.

Benchmarking generalization

Simulations provide a good platform for more comprehensive evaluation of RL algorithms. Together with PAD, we release DMControl Generalization Benchmark, a new benchmark for generalization in RL based on the DeepMind Control Suite, a popular benchmark for continuous control from images. In the DMControl Generalization Benchmark, agents are trained in a fixed environment and deployed in new environments with e.g. randomized colors or continuously changing video backgrounds. We consider an SAC agent trained with an IDM, with and without adaptation, and compare to CURL, a contrastive method discussed in a previous post. We compare the generalization ability of methods in the visualization below, and generally find that PAD can adapt even in non-stationary environments, a challenging problem setting where non-adaptive methods tend to fail. While CURL is found to generalize no better than the non-adaptive SAC trained with an IDM, agents can still benefit from the training signal that CURL provides during the training phase. Algorithms that learn both during training and deployment, and from multiple training signals, may therefore be preferred.

Generalization to an environment with video background. CURL is a
contrastive method, SAC+IDM is a Soft Actor-Critic (SAC) policy trained
with an Inverse Dynamics Model (IDM), and SAC+IDM (PAD) is the same
policy but with the addition of policy adaptation during deployment.

Summary

Previous work addresses the problem of generalization in RL by randomization, which requires anticipation of environmental changes and is known to not scale well. We formulate an alternative problem setting in vision-based RL: can we instead adapt a pre-trained policy to unseen environments, without any rewards or human feedback? We find that adapting policies through a self-supervised objective – solely from interactions in the new environment – is a promising alternative to domain randomization when the target environment is truly unknown. In the future, we ultimately envision agents that continuously learn and adapt to their surroundings, and are capable of learning both from explicit human feedback and through unsupervised interaction with the environment.

This post is based on the following paper:

• Self-Supervised Policy Adaptation during Deployment
  Nicklas Hansen, Rishabh Jangir, Yu Sun, Guillem Alenyá, Pieter Abbeel, Alexei A. Efros, Lerrel Pinto, Xiaolong Wang
  Ninth International Conference on Learning Representations (ICLR), 2021
  arXiv, Project Website, Code

The MIT CSAIL team’s LaserFactory system can manufacture functional, custom-made devices and robots, without human intervention, potentially enabling rapid prototyping of items like wearables, robots, and printed electronics. Credits: Photo courtesy of the researchers.

By Rachel Gordon | MIT CSAIL

From Star Trek’s replicators to Richie Rich’s wishing machine, popular culture has a long history of parading flashy machines that can instantly output any item to a user’s delight. 

While 3D printers have now made it possible to produce a range of objects that include product models, jewelry, and novelty toys, we still lack the ability to fabricate more complex devices that are essentially ready-to-go right out of the printer. 

A group from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) recently developed a new system to print functional, custom-made devices and robots, without human intervention. Their single system uses a three-ingredient recipe that lets users create structural geometry, print traces, and assemble electronic components like sensors and actuators. 

“LaserFactory” has two parts that work in harmony: a software toolkit that allows users to design custom devices, and a hardware platform that fabricates them. 

CSAIL PhD student Martin Nisser says that this type of “one-stop shop” could be beneficial for product developers, makers, researchers, and educators looking to rapidly prototype things like wearables, robots, and printed electronics. 

“Making fabrication inexpensive, fast, and accessible to a layman remains a challenge,” says Nisser, lead author on a paper about LaserFactory that will appear in the ACM Conference on Human Factors in Computing Systems in May. “By leveraging widely available manufacturing platforms like 3D printers and laser cutters, LaserFactory is the first system that integrates these capabilities and automates the full pipeline for making functional devices in one system.” 

Inside LaserFactory

Let’s say a user has aspirations to create their own drone. They’d first design their device by placing components on it from a parts library, and then draw on circuit traces, which are the copper or aluminum lines on a printed circuit board that allow electricity to flow between electronic components. They’d then finalize the drone’s geometry in the 2D editor. In this case, they’d use propellers and batteries on the canvas, wire them up to make electrical connections, and draw the perimeter to define the quadcopter’s shape. 

The user can then preview their design before the software translates their custom blueprint into machine instructions. The commands are embedded into a single fabrication file for LaserFactory to make the device in one go, aided by the standard laser cutter software. On the hardware side, an add-on that prints circuit traces and assembles components is clipped onto the laser cutter. 

Similar to a chef, LaserFactory automatically cuts the geometry, dispenses silver for circuit traces, picks and places components, and finally cures the silver to make the traces conductive, securing the components in place to complete fabrication. 

The device is then fully functional, and in the case of the drone, it can immediately take off to begin a task — a feature that could in theory be used for diverse jobs such as delivery or search-and-rescue operations.

LaserFactory (in background) can fabricate fully functional drones like this quadcopter. Credits: Photo courtesy of the researchers.

As a future avenue, the team hopes to increase the quality and resolution of the circuit traces, which would allow for denser and more complex electronics. 

As well as fine-tuning the current system, the researchers hope to build on this technology by exploring how to create a fuller range of 3D geometries, potentially through integrating traditional 3D printing into the process. 

“Beyond engineering, we’re also thinking about how this kind of one-stop shop for fabrication devices could be optimally integrated into today’s existing supply chains for manufacturing, and what challenges we may need to solve to allow for that to happen,” says Nisser. “In the future, people shouldn’t be expected to have an engineering degree to build robots, any more than they should have a computer science degree to install software.” 

This research is based upon work supported by the National Science Foundation. The work was also supported by a Microsoft Research Faculty Fellowship and The Royal Swedish Academy of Sciences.

In this episode, our interviewer Lauren Klein speaks with Kim Baraka about his PhD research to enable robots to engage in social interactions, including interactions with children with Autism Spectrum Disorder. Baraka discusses how robots can plan their actions across multiple modalities when interacting with humans, and how models from psychology can inform this process. He also tells us about his passion for dance, and how dance may serve as a testbed for embodied intelligence within Human-Robot Interaction.

Kim Baraka

Kim Baraka is a postdoctoral researcher in the Socially Intelligent Machines Lab at the University of Texas at Austin, and an upcoming Assistant Professor in the Department of Computer Science at Vrije Universiteit Amsterdam, where he will be part of the Social Artificial Intelligence Group. Baraka recently graduated with a dual PhD in Robotics from Carnegie Mellon University (CMU) in Pittsburgh, USA, and the Instituto Superior Técnico (IST) in Lisbon, Portugal. At CMU, Baraka was part of the Robotics Institute and was advised by Prof. Manuela Veloso. At IST, he was part of the Group on AI for People and Society (GAIPS), and was advised by Prof. Francisco Melo.

Dr. Baraka’s research focuses on computational methods that inform artificial intelligence within Human-Robot Interaction. He develops approaches for knowledge transfer between humans and robots in order to support mutual and beneficial relationships between the robot and human. Specifically, he has conducted research in assistive interactions where the robot or human helps their partner to achieve a goal, and in teaching interactions. Baraka is also a contemporary dancer, with an interest in leveraging lessons from dance to inform advances in robotics, or vice versa.

• Download mp3 (13.7 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

PS. If you enjoy listening to experts in robotics and asking them questions, we recommend that you check out Talking Robotics. They have a virtual seminar on Dec 11 where they will be discussing how to conduct remote research for Human-Robot Interaction; something that is very relevant to researchers working from home due to COVID-19.

In this episode, Abate interviews Josh Lessing, co-founder and CEO of Root AI. At Root AI they are developing a system that tracks data on the farm and autonomously harvests crops using delicate grippers and computer vision. Lessing talks about the path they took to build a product with good market fit and how they brought a venture capital backed startup to market.

Josh Lessing

Josh is one of the world’s leading minds on developing robotics and AI systems for the food industry, previously serving as the Director of R&D at Soft Robotics Inc. His current venture, Root AI, is integrating advanced robotics, vision systems and machine perception to automate agriculture. Josh was a Postdoctoral Fellow in Materials Science & Robotics at Harvard University, having earned his Ph.D. studying Biophysics & Physical Chemistry at the Massachusetts Institute of Technology and received an Sc.B. in Chemistry from Brown University.

Links

• Download mp3 (16.0 MB)
• Listen to this episode on Spotify
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

In this episode, Lilly Clark interviews Jeannette Bohg, Assistant Professor at Stanford, about her work in interactive perception and robot learning for grasping and manipulation tasks. Bohg discusses how robots and humans are different, the challenge of high dimensional data, and unsolved problems including continuous learning and decentralized manipulation.

Jeannette Bohg is an Assistant Professor of Computer Science at Stanford University. She was a group leader at MPI until September 2017 and remains affiliated as a guest researcher. Her research focuses on perception for autonomous robotic manipulation and grasping. She is specifically interested in developing methods that are goal-directed, real-time and multi-modal such that they can provide meaningful feedback for execution and learning.

Before joining the Autonomous Motion lab in January 2012, Jeannette Bohg was a PhD student at the Computer Vision and Active Perception lab (CVAP) at KTH in Stockholm. Her thesis on Multi-modal scene understanding for Robotic Grasping was performed under the supervision of Prof. Danica Kragic. She studied at Chalmers in Gothenburg and at the Technical University in Dresden where she received her Masters in Art and Technology and her Diploma in Computer Science, respectively.

Links

• Download mp3 (44.0 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Shihan Lu interviews Dr. Heather Culbertson, Assistant Professor in the Computer Science Department at the University of Southern California, about her work in haptics. Dr. Culbertson discusses the data-driven realistic texture modeling and rendering, haptic technologies in the social touch, the combination of haptics and robots, expectations and obstacles of haptics in the next 5 years.

Heather Culbertson

Heather Culbertson is a Gabilan Assistant Professor of Computer Science at the University of Southern California. Her research focuses on the design and control of haptic devices and rendering systems, human-robot interaction, and virtual reality. Particularly she is interested in creating haptic interactions that are natural and realistically mimic the touch sensations experienced during interactions with the physical world. Previously, she was a research scientist in the Department of Mechanical Engineering at Stanford University where she worked in the Collaborative Haptics and Robotics in Medicine (CHARM) Lab. She received her PhD in the Department of Mechanical Engineering and Applied Mechanics (MEAM) at the University of Pennsylvania in 2015 working in the Haptics Group, part of the General Robotics, Automation, Sensing and Perception (GRASP) Laboratory.

Links

• Download mp3 (14.8 MB)
• Culbertson’s research website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Lilly interviews Juxi Leitner, a Postdoctoral Research Fellow at the Queensland University of Technology and Co-Founder/CEO of LYRO Robotics. LYRO spun out of the 2017 win of the Amazon Robotics Challenge by Team ACRV. Here Juxi discusses deep learning, computer vision, intent in grasping and manipulation, and bridging the gap between abstract and low-level understandings of the world. He also discusses why robotics is really an integration field, the Amazon and other robotics challenges, and what’s important to consider when spinning an idea into a company.

Juxi Leitner

Juxi Leitner is co-founder of LYRO Robotics, a deep-tech startup based in Brisbane, Australia, creating robotic picking and packing solutions. LYRO is a spin-out of the Australian Centre of Excellence for Robotic Vision (ACRV), where Juxi is the research lead for the manipulation research stream (previously Vision and Action project). His research focus is on integrating Robotics, Computer Vision and Machine Learning/Artificial Intelligence (AI) for robust grasping and manipulation in real-world scenarios. In 2017 his team won the Amazon Robotics Challenge. Juxi is active in the local Brisbane deep-tech ecosystem and started Brisbane.AI and the Brisbane robotics interest group.

Links

• Download mp3 (17.3MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

“The set-updemonstrates various techniques that provide a cheaper solution, increased robustness and increased safety for operators”, says Søren Peter Johansen Technology Manager at DTI.

“For example, there is a force-torque sensor in the robot which is used to manoeuvre things carefully into place, and also to increase the confidence of the operators. The force-torque sensor will sense and prevent collisions with people before the built-in safety features of the robot stop the robot arm. Thus, we can also increase safety and confidence by working with collaborative robots”, he says.

Increased effectiveness
Danfoss in Gråsten has tested the robot in connection with the company’s production of frequency converters.

“A frequency converter contains, amongst other things, capacitors assembled in groups of two or four. We have automated this process in that a robot picks up a capacitor and puts it into a fixture. For the capacitor to be correctly placed in the fixture, the electrodes must point in the right direction. Therefore, a camera sees how the electrodes are oriented in the gripper, and orients correctly before it is placed in the fixture”, explains Søren Peter Johansen.

“Then we pick a nut that has to be put on and the robot picks up the socket handle and screws the nut tight. And meanwhile the operator can do something else and thus increases efficiency”, he elaborates.

Three reasons to automate
Peter Lund Andersen, Senior Manufacturing Specialist at Danfoss, says that Danfoss is automating for several reasons:

“It is primarily about work and environmental considerations – better ergonomics and less heavy lifting. We believe that with robots, that can do the work uniformly each time, we can increase the quality of our products while maintaining safety and reducing costs”, he says, adding “The cooperation with DTI gives us the opportunity to come out and try some new things. Some things that aren’t quite mature yet and can’t just be taken off a shelf. That way we are at the forefront with everything.”

If you want to watch more videos, you can explore ROBOTT-NET’s pilot projects on our YouTube-channel.

You can learn more about the urban pest control project here:

This type of prevention is designed to help technicians and companies have better efficiency and control and a faster response, when it comes to controlling rodent pests.

“The project uses a mobile autonomous robotic platform with a robot arm to introduce a camera into the trap. It captures an image that is uploaded to the cloud”.

“The project is in collaboration with Robotnik, which is responsible for the assembly of the robot; and Hispavista, which is in charge of the cloud part”, Marco Lorenzo adds.

Aritz Zabaleta, a Systems Technician at Hispavista Labs explains that the application consists of two components:

“The first manages the entire fleet of robots that communicate with the server in the cloud and it processes the information collected by them. The second component is the one that allows customers to access processed images”.

If you want to watch more fascinating robotics videos, you can explore ROBOTT-NET’s pilot projects on our YouTube-channel.

In this episode, Ron Vanderkley interviews Mark Pivac, Chief Technical Officer and co-founder of FBR (formerly Fastbrick Robotics) about the world’s first end-to-end autonomous bricklaying robot, ‘Hadrian X’. Three years after his first interview, we catch up with Pivac to see how FBR has expanded its operation and chat about their latest commercial prototype, ‘Hadrian X’, as well as the future of the robotic construction industry.

Mark Pivac is the primary inventor of FBR’s automated bricklaying technology. He is an aeronautical and mechanical engineer with over 25 years’ experience working on the development of high technology equipment ranging from lightweight aircraft to heavy off-road equipment. He has 20 years’ experience working with pro/engineer 3D CAD software as well as high-level mathematics, including matrix mathematics, robot transformations and vector mathematics for machine motion. Mark has also worked extensively with design, commissioning and fault finding on servo controlled motion systems achieving very high dynamic performance.

Links

• FBR Website
• Marc Pivac Profile Page
• Hadrian X in action (Video)
• Download mp3 (13.3 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, join our interviewer Andrew Vaziri at Promat 2019, the largest expo for manufacturing and supply chain professionals in North and South America. Andrew interviews a handful of companies which provide warehouse fulfillment robots that can autonomous pick and place items. Our guests explain how advances in AI have made autonomous picking possible. They also talk about the unique technologies they use to stand out in a crowded field of competing products.

The guests in the order they appear are:
Vince Martinelli, Head of Product and Marketing, RightHand Robotics
Jim Liefer, CEO, Kindred
Pete Blair, VP of Marketing, Berkshire Grey
Sean Davis, Technical Product Manager, Osaro
Erik Nieves, CEO, Plus One Robotics

See video of each robot picking items on our YouTube page:

Links

• Download mp3 (11.7 MB)
• Righthand Robotics Website
• Kindred Website
• Berkshire Grey’s Website
• Osaro’s Website
• PlusOne Robotics’ Website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this interview, Audrow Nash interviews Kristoffer Richardsson, Michael Zillich, and Paulo Alvito.

Kristoffer Richardsson, Developer at Bitcraze in Sweden, speaks about small open-source unmanned aerial vehicle and different ways of localizing them.

Michael Zillich, CTO of Blue Danube Robotics in Austria, discusses a robotics platform for picking up toys, called Kenny. Zillich talks about Kenny’s design, including sensors and its robotic arm. He discusses how children can show the robot where to put specific toys, about how this large European project was coordinated, and about possibly commercializing this platform.

Paulo Alvito, CEO and CTO and Co-founder of IDMind in Portugal, discusses remote inspection robot for sewers. Alvito discusses what it means to be remotely autonomous, the design of the robot, and how these robots communicate data.

Links

• Download mp3 (17.3 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Marwa ElDiwiny interview Peer Fisher, a Professor of Physical Chemistry at the University of Stuttgart and the Director of the Micro Nano and Molecular Systems Lab at the Max Planck Institute for Intelligent Systems.  Fischer discusses micro robots that has been designed to move inside of environments similar to the human body called, “micro swimmers.”  He talks about how they are fabricated, powered, and how they can move with light or “nano propellers.”  Fischer also discusses simulating human tissue and the future of micro and nano robots, including how they could be a replacement for certain surgeries.

Peer Fischer

Peer Fischer is a Professor of Physical Chemistry at the University of Stuttgart and he heads the independent Micro Nano and Molecular Systems Lab at the Max Planck Institute for Intelligent Systems in Stuttgart.

He received a BSc. degree in Physics from Imperial College London and a Ph.D. from the University of Cambridge. He was a NATO (DAAD) Postdoctoral Fellow at Cornell University, before joining the Rowland Institute at Harvard. At Harvard he held a Rowland Fellowship and directed an interdisciplinary research lab for five years. In 2009 he received an Attract Award from the Fraunhofer Society which led him to set up a photonics lab at the Fraunhofer Institute for Physical Measurement Techniques in Freiburg. In 2011 he moved his lab to the Max Planck Institute for Intelligent Systems in Stuttgart, and since 2013 he is a Professor at the University of Stuttgart.

Links

• Micro, Nano, & Molecular Systems Lab
• Micro Swimmers
• Download mp3 (13.7 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

The pilot projects

The three companies’ accomplished voucher work varies a lot and sets the basis to enlarge the array of industrial challenges tackled in the pilot work of ROBOTT-NET.

Linak’s pilot project will address some of the barriers that are still hindering the introduction of robots in High-Mix-Low-Volume (HMLV) production. This pilot will focus on the development of a modular assembly robot solution called MARS, that can be introduced in a wide-range of industries. You can check out Linak’s voucher project here:

Irabia’s pilot project UrbanPestPatrol aims to use robots for pest management in industrial facilities. The purpose of the project is to minimize the risk of a pest infestation and its resulting cost. You can learn about Irabia’s voucher project here:

https://www.youtube.com/watch?v=1zmHDOa1ErU

Lastly, Nissan will be looking into automated kitting for complex and diverse applications. This pilot will build a robotic test cell that enables Nissan Motor Manufacturting UK to develop automated kitting capability for its part families. Find out more about Nissan’s voucher project here:

ROBOTT-NET pilots help companies develop their voucher work through proof-of-concept level and accelerate it towards commercialisation. A pilot project will be a medium-scale research prototype installation, lasting for up to 18 months developing the robot technology and business case explored in the voucher stage and applying it to an industrial application demonstration at an end user’s site.

If you want to learn more about the pilot projects, you can check out ROBOTT-NETs website.

In this episode, Abate interviews Andrew Stein from Anki. At Anki they developed an engaging robot called Cozmo which packs sophisticated robotic software inside a lifelike, palm sized, robot. Cozmo recognizes people and objects around him and plays games with them. Cozmo is unique in that a large amount of development has been implemented to make his animations and behavior feel natural, in addition to focusing on classical robotic elements such as computer vision and object manipulation.

Andrew Neil Stein

Andrew Stein is the Head of Robotics & AI at Anki, where he began working on the Cozmo project more than four years ago as the team’s first member. He has contributed to several core systems of the product, including vision, cube manipulation, animation streaming, localization, high-level behaviors, and low-level actions. He earned his Ph.D. from the Robotics Institute at Carnegie Mellon University, and his Bachelor’s and Master’s degrees in Electrical and Computer Engineering from the Georgia Institute of Technology.

Links

• Anki’s Website
• Cozmo’s Website
• Andrew Stein LinkedIn
• Download mp3 (19.1 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Jack Rasiel interviews Vijay Kumar, Professor and Dean of Engineering at the University of Pennsylvania.  Kumar discusses the guiding ideas behind his research on micro unmanned aerial vehicles, gives his thoughts on the future of robotics in the lab and field, and speaks about setting realistic expectations for robotics technology.

Vijay Kumar

Vijay Kumar is the Nemirovsky Family Dean of Penn Engineering with appointments in the Departments of Mechanical Engineering and Applied Mechanics, Computer and Information Science, and Electrical and Systems Engineering at the University of Pennsylvania.

Dr. Kumar received his Bachelor of Technology degree from the Indian Institute of Technology, Kanpur and his Ph.D. from The Ohio State University in 1987. He has been on the Faculty in the Department of Mechanical Engineering and Applied Mechanics with a secondary appointment in the Department of Computer and Information Science at the University of Pennsylvania since 1987. In his time at the university, Dr. Kumar has held numerous positions including director of the GRASP Laboratory, Chairman of the Department of Mechanical Engineering and Applied Mechanics, and Deputy Dean for Education in the School of Engineering and Applied Science. From 2012 to 2013, he served as the assistant director of robotics and cyber physical systems at the White House Office of Science and Technology Policy.

Links

• Vijay Kumar Lab
• Download (11.8 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

A new robot under development can send information on the stiffness, look and feel of a patient to a doctor located kilometres away. Image credit: Accrea

A robotic doctor that can be controlled hundreds of kilometres away by a human counterpart is gearing up for action. Getting a check-up from a robot may sound like something from a sci-fi film, but scientists are closing in on this real-life scenario and have already tested a prototype.

‘The robot at the remote site has different force, humidity and temperature sensors, all capturing information that a doctor would get when they are directly palpating (physically examining) a patient,’ explains Professor Angelika Peer, a robotics researcher at the University of the West of England, UK.

Prof. Peer is also the project coordinator of the EU-funded ReMeDi project, which is developing the robotic doctor to allow medical professionals to examine patients over huge distances.

Through a specially designed surface mounted on a robotic arm, stiffness data of the patient’s abdomen is displayed to the human, allowing the doctor to feel what the remote robot feels. This is made possible thanks to a tool called a haptic device, which has a soft surface reminiscent of skin that can recreate the sense of touch through force and changing its shape.

During the examination, the doctor sits at a desk facing three screens, one showing the doctor’s hand on the faraway patient and a second for teleconferencing with the patient, which will remain an essential part of the exchange.

The third screen displays a special capability of the robot doctor – ultrasonography. This is a medical technique that sends sound pulses into a patient’s body to create a window into the patient. It reveals areas of different densities in the body and is often used to examine pregnant women.

Ultrasonography is also important for flagging injuries or disease in organs such as the heart, liver, kidneys or spleen and can find indications for some types of cancer, too.

‘The system allows a doctor from a remote location to do a first assessment of a patient and make a decision about what should be done, whether to transfer them to hospital or undergo certain treatments,’ said Prof. Peer.

The robot currently resides in a hospital in Poland but scientists have shown the prototype at medical conferences around the world. And they have already been approached by doctors from Australia and Canada where it can take several hours to transfer rural patients to a doctor’s office or hospital.

With the help of a robot, a doctor can talk to a patient, manoeuvre robotic arms, feel what the robot senses and get ultrasounds. Image credit: ReMeDi

‘This is to support an initial diagnosis. The human is still in the loop, but this allows them to perform an examination remotely,’ said Prof. Peer.

Telemedicine

The ReMeDi project could speed up a medical exam and save time for patients and clinics. Another EU-funded project – United4Health (U4H) – looks at a different technology that could be used to remotely diagnose or treat people.

‘We need to transform how we deliver health and care,’ said Professor George Crooks, director of the Scottish Centre for Telehealth & Telecare, UK, which provides services via telephone, web and digital television and coordinates U4H.

This approach is crucial as Europe faces an ageing population and a rise in long-term health conditions like diabetes and heart disease. Telemedicine empowers these types of patients to take steps to help themselves at home, while staying in touch with medical experts via technology. Previous studies showed those with heart failure can be successfully treated this way.

These patients were given equipment to monitor their vital signs and send data back to a hospital. A trial in the UK comparing this self-care group to the standard-care group showed a reduction in mortality, hospital admissions and bed days, says Prof. Crooks.

A similar result was shown in the demonstration sites of the U4H project which tested the telemedicine approach in 14 regions for patients with heart failure, diabetes and chronic obstructive pulmonary disease (COPD). For diabetic patients in Scotland, they kept in touch with the hospital using text messages. For COPD, some patients used video consultations.

Prof. Crooks stresses that it is not all about the electronics – what matters is the service wraparound that makes the technology acceptable and easy to use for patients and clinical teams.

‘It can take two or three hours out of your day to go along to a 15 minute medical appointment and then to be told to keep taking your medication. What we do is, by using technology, patients monitor their own parameters, such as blood sugar in the case of diabetes, how they are feeling, diet and so on, and then they upload these results,’ said Prof. Crooks.

‘It doesn’t mean you never go to see a doctor, but whereas you might have gone seven or eight times a year, you may go just once or twice.’

Crucially, previous research has shown these patients fare better and the approach is safe.

‘There can be an economic benefit, but really this is about saving capacity. It frees up healthcare professionals to see the more complex cases,’ said Prof. Crooks.

It also empowers patients to take more responsibility for their health and results in fewer unplanned visits to the emergency room.

‘Patient satisfaction rates were well over 90 %,’ said Prof. Crooks.

In this episode, Audrow Nash interviews Sergey Levine, assistant professor at UC Berkeley, about deep learning on robotics. Levine explains what deep learning is and he discusses the challenges of using deep learning in robotics. Lastly, Levine speaks about his collaboration with Google and some of the surprising behavior that emerged from his deep learning approach (how the system grasps soft objects).

In addition to the main interview, Audrow interviewed Levine about his professional path. They spoke about what questions motivate him, why his PhD experience was different to what he had expected, the value of self-directed learning,  work-life balance, and what he wishes he’d known in graduate school.

A video of Levine’s work in collaboration with Google.

https://www.youtube.com/watch?v=cXaic_k80uM&feature=youtu.be

Sergey Levine

Sergey Levine is an assistant professor at UC Berkeley. His research focuses on robotics and machine learning. In his PhD thesis, he developed a novel guided policy search algorithm for learning complex neural network control policies, which was later applied to enable a range of robotic tasks, including end-to-end training of policies for perception and control. He has also developed algorithms for learning from demonstration, inverse reinforcement learning, efficient training of stochastic neural networks, computer vision, and data-driven character animation.

Links

• Download mp3 (16.0 MB)
• Download extra mp3 (7.1 MB)
• Sergey Levine’s website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

By Eran Klein, University of Washington and Katherine Pratt, University of Washington

In the 1995 film “Batman Forever,” the Riddler used 3-D television to secretly access viewers’ most personal thoughts in his hunt for Batman’s true identity. By 2011, the metrics company Nielsen had acquired Neurofocus and had created a “consumer neuroscience” division that uses integrated conscious and unconscious data to track customer decision-making habits. What was once a nefarious scheme in a Hollywood blockbuster seems poised to become a reality.

Recent announcements by Elon Musk and Facebook about brain-computer interface (BCI) technology are just the latest headlines in an ongoing science-fiction-becomes-reality story.

BCIs use brain signals to control objects in the outside world. They’re a potentially world-changing innovation – imagine being paralyzed but able to “reach” for something with a prosthetic arm just by thinking about it. But the revolutionary technology also raises concerns. Here at the University of Washington’s Center for Sensorimotor Neural Engineering (CSNE) we and our colleagues are researching BCI technology – and a crucial part of that includes working on issues such as neuroethics and neural security. Ethicists and engineers are working together to understand and quantify risks and develop ways to protect the public now.

Picking up on P300 signals

All BCI technology relies on being able to collect information from a brain that a device can then use or act on in some way. There are numerous places from which signals can be recorded, as well as infinite ways the data can be analyzed, so there are many possibilities for how a BCI can be used.

Some BCI researchers zero in on one particular kind of regularly occurring brain signal that alerts us to important changes in our environment. Neuroscientists call these signals “event-related potentials.” In the lab, they help us identify a reaction to a stimulus.

In particular, we capitalize on one of these specific signals, called the P300. It’s a positive peak of electricity that occurs toward the back of the head about 300 milliseconds after the stimulus is shown. The P300 alerts the rest of your brain to an “oddball” that stands out from the rest of what’s around you.

For example, you don’t stop and stare at each person’s face when you’re searching for your friend at the park. Instead, if we were recording your brain signals as you scanned the crowd, there would be a detectable P300 response when you saw someone who could be your friend. The P300 carries an unconscious message alerting you to something important that deserves attention. These signals are part of a still unknown brain pathway that aids in detection and focusing attention.

Reading your mind using P300s

P300s reliably occur any time you notice something rare or disjointed, like when you find the shirt you were looking for in your closet or your car in a parking lot. Researchers can use the P300 in an experimental setting to determine what is important or relevant to you. That’s led to the creation of devices like spellers that allow paralyzed individuals to type using their thoughts, one character at a time.

It also can be used to determine what you know, in what’s called a “guilty knowledge test.” In the lab, subjects are asked to choose an item to “steal” or hide, and are then shown many images repeatedly of both unrelated and related items. For instance, subjects choose between a watch and a necklace, and are then shown typical items from a jewelry box; a P300 appears when the subject is presented with the image of the item he took.

Everyone’s P300 is unique. In order to know what they’re looking for, researchers need “training” data. These are previously obtained brain signal recordings that researchers are confident contain P300s; they’re then used to calibrate the system. Since the test measures an unconscious neural signal that you don’t even know you have, can you fool it? Maybe, if you know that you’re being probed and what the stimuli are.

Techniques like these are still considered unreliable and unproven, and thus U.S. courts have resisted admitting P300 data as evidence.

Imagine that instead of using a P300 signal to solve the mystery of a “stolen” item in the lab, someone used this technology to extract information about what month you were born or which bank you use – without your telling them. Our research group has collected data suggesting this is possible. Just using an individual’s brain activity – specifically, their P300 response – we could determine a subject’s preferences for things like favorite coffee brand or favorite sports.

But we could do it only when subject-specific training data were available. What if we could figure out someone’s preferences without previous knowledge of their brain signal patterns? Without the need for training, users could simply put on a device and go, skipping the step of loading a personal training profile or spending time in calibration. Research on trained and untrained devices is the subject of continuing experiments at the University of Washington and elsewhere.

It’s when the technology is able to “read” someone’s mind who isn’t actively cooperating that ethical issues become particularly pressing. After all, we willingly trade bits of our privacy all the time – when we open our mouths to have conversations or use GPS devices that allow companies to collect data about us. But in these cases we consent to sharing what’s in our minds. The difference with next-generation P300 technology under development is that the protection consent gives us may get bypassed altogether.

What if it’s possible to decode what you’re thinking or planning without you even knowing? Will you feel violated? Will you feel a loss of control? Privacy implications may be wide-ranging. Maybe advertisers could know your preferred brands and send you personalized ads – which may be convenient or creepy. Or maybe malicious entities could determine where you bank and your account’s PIN – which would be alarming.

With great power comes great responsibility

The potential ability to determine individuals’ preferences and personal information using their own brain signals has spawned a number of difficult but pressing questions: Should we be able to keep our neural signals private? That is, should neural security be a human right? How do we adequately protect and store all the neural data being recorded for research, and soon for leisure? How do consumers know if any protective or anonymization measures are being made with their neural data? As of now, neural data collected for commercial uses are not subject to the same legal protections covering biomedical research or health care. Should neural data be treated differently?

These are the kinds of conundrums that are best addressed by neural engineers and ethicists working together. Putting ethicists in labs alongside engineers – as we have done at the CSNE – is one way to ensure that privacy and security risks of neurotechnology, as well as other ethically important issues, are an active part of the research process instead of an afterthought. For instance, Tim Brown, an ethicist at the CSNE, is “housed” within a neural engineering research lab, allowing him to have daily conversations with researchers about ethical concerns. He’s also easily able to interact with – and, in fact, interview – research subjects about their ethical concerns about brain research.

There are important ethical and legal lessons to be drawn about technology and privacy from other areas, such as genetics and neuromarketing. But there seems to be something important and different about reading neural data. They’re more intimately connected to the mind and who we take ourselves to be. As such, ethical issues raised by BCI demand special attention.

Working on ethics while tech’s in its infancy

As we wrestle with how to address these privacy and security issues, there are two features of current P300 technology that will buy us time.

First, most commercial devices available use dry electrodes, which rely solely on skin contact to conduct electrical signals. This technology is prone to a low signal-to-noise ratio, meaning that we can extract only relatively basic forms of information from users. The brain signals we record are known to be highly variable (even for the same person) due to things like electrode movement and the constantly changing nature of brain signals themselves. Second, electrodes are not always in ideal locations to record.

All together, this inherent lack of reliability means that BCI devices are not nearly as ubiquitous today as they may be in the future. As electrode hardware and signal processing continue to improve, it will be easier to continuously use devices like these, and make it easier to extract personal information from an unknowing individual as well. The safest advice would be to not use these devices at all.

The goal should be that the ethical standards and the technology will mature together to ensure future BCI users are confident their privacy is being protected as they use these kinds of devices. It’s a rare opportunity for scientists, engineers, ethicists and eventually regulators to work together to create even better products than were originally dreamed of in science fiction.

D’Arpino Perez discusses her work teaching Optimus to pick up a bottle. Image: Jason Dorfman, MIT CSAIL

Most robots are programmed using one of two methods: learning from demonstration, in which they watch a task being done and then replicate it, or via motion-planning techniques like optimization or sampling, which require a programmer to explicitly specify a task’s goals and constraints.

Both methods have drawbacks. Robots that learn from demonstration can’t easily transfer one skill they’ve learned to another situation and remain accurate. On the other hand, motion planning systems that use sampling or optimization can adapt to these changes, but are time-consuming, since they usually have to be hand-coded by expert programmers.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have recently developed a system that aims to bridge the two techniques: C-LEARN, which allows non-coders to teach robots a wider range of tasks simply by providing some information about how objects are typically manipulated and then showing the robot a single demo of the task. Importantly, this enables users to teach robots skills that can be automatically transferred to other robots with different “kinematics” (ways of moving) – a key time- and cost-saving measure for companies that want a range of robots to perform similar actions.

“By combining the intuitiveness of learning from demonstration with the precision of motion-planning algorithms, this approach can help robots do new types of tasks that they haven’t been able to learn before, like multi-step assembly using both of their arms,” says Claudia Pérez-D’Arpino, a PhD student who wrote a paper on C-LEARN with MIT professor Julie Shah.

The team tested the system on Optimus, a new two-armed robot designed for bomb disposal that they programmed to perform tasks like opening doors, transporting objects and extracting objects from containers.

(The robot is made of components from multiple manufacturers, including a manipulation system by RE2, a Husky unmanned ground vehicle by Clearpath, 3-finger grippers by Robotiq and the multisense sensor from Carnegie Robotics that is used in the Atlas robot.)

In simulations they showed that Optimus’ learned skills could be seamlessly transferred to Atlas, CSAIL’s six-foot-tall, 400-pound humanoid robot.

The paper was recently accepted to the IEEE International Conference on Robotics and Automation (ICRA), which takes place May 29-June 3 in Singapore.

How it works

With C-LEARN the user first gives the robot a knowledge base of information on how to reach and grasp different objects that have different constraints. (The C in C-LEARN stands for “constraints.”) The operator then uses a 3D interface to show the robot a single demonstration of the specific task, which is represented by a sequence of relevant moments known as “keyframes.” By matching these keyframes with the knowledge base, the robot can automatically suggest motion plans for the operator to approve or edit as needed.

“This approach is actually very similar to how humans learn in terms of seeing how something’s done and connecting it to what we already know about the world,” says Pérez-D’Arpino. “We can’t magically learn from a single demonstration, so we take new information and match it to previous knowledge about our environment.”

One challenge was that existing constraints that could be learned from demonstrations weren’t accurate enough to enable robots to precisely manipulate objects. To overcome that, the researchers developed constraints inspired by computer-aided design (CAD) programs that can tell the robot if its hands should be parallel or perpendicular to the objects it is interacting with.

The team also showed that the robot performed significantly better in collaboration with humans. While the robot successfully executed tasks 87.5 percent of the time on its own, it did so 100 percent of the time when it had a operator that could correct minor sensory inaccuracies.

“Having a knowledge base is fairly common, but what’s not common is integrating it with learning from demonstration,” says Dmitry Berenson, an assistant professor of computer science at the University of Michigan who was not involved in the research. “That’s very helpful, because if you are dealing with the same objects over and over again, you don’t want to then have to start from scratch to teach the robot every new task.”

The system is part of a larger wave of research focused on making LfD approaches more adaptive. If you’re a robot that has learned to take an object out of a tube from a demonstration, you might not be able to do it if there’s an obstacle in the way that requires you to move your arm differently. However, C-LEARN can do this, because it does not learn one specific way to perform the action.

“It’s good for the field that we’re moving away from directly imitating motion, towards actually trying to infer the principles behind the motion,” says Berenson. “By using these learned constraints in a motion planner, we can make systems that are far more flexible than those which just try to mimic what’s being demonstrated”

Shah says that advanced LfD methods could prove important in time-sensitive scenarios like bomb disposal and disaster response, where robots are currently tele-operated at the level of individual joint movements.

“Something as simple as picking up a box could take 20-30 minutes, which is significant for an emergency situation,” says Pérez-D’Arpino.

C-LEARN can’t yet handle certain advanced tasks, like avoiding collisions or planning for different step sequences for a given task. But the team is hopeful that incorporating more insights from human learning will give robots an even wider range of physical capabilities.

“Traditional programming of robots in real-world scenarios is difficult, tedious and requires a lot of domain knowledge,” says Shah. “It would be much more effective if we could train them more like how we train people: by giving them some basic knowledge and a single demonstration. This is an exciting step towards teaching robots to perform complex multi-arm and multi-step tasks necessary for assembly manufacturing and ship or aircraft maintenance.”

Click here to access the full paper.

What do you get when you put together wood and rope? Well according to Plymouth University’s Professor Guido Bugmann: a low-cost, open source, 2 meter tall robot! All buildable for under £2000. The Cheap Arm Project (CHAP) began as an MSc project aimed at developing an affordable mobile robot arm system that could be used by wheelchair users to access daily objects at inaccessible heights or weights (the extreme case being 2 litre bottle).

Now the project has turned open source in the hope of continuing development and gathering more feedback. With the entire hardware and software details available online, with detailed instructions, the team hopes that community members will attempt to construct their own CHAP robot.

While building an entire robot may conventionally seem a complex and expensive undertaking, CHAP stands out with its use of 3D printed parts and wood. In fact, the “under £2000” price tag of required parts is primarily made up of the use six Dynamixel motors, each costing up to £200 and used by the project due to having numerous available during development. With this in mind, future work will look at the use of lower cost motors.

Rather than a robot ‘brain’, CHAP is powered by a human one. To accomplish this, CHAP’s Raspberry Pi was used to host a local web page to enable tele-operation from any connected device, all accessible through a WiFi hotspot also hosted by the Pi. This was an important design choice as it removed the need for platform specific apps and associated costs, simultaneously simplifying deployment and testing.

This was done deliberately to keep the design simple, and to allow users the freedom to interact with their world. Early user-tests discovered that some level of intelligence would be needed to simplify controls. With currently three joints, and three axis available; some steps are needed to automate object interaction (pull requests welcome!).

The next step for CHAP will be refinement of hardware and software; however the very first will be building a community! With full design documents available it is hoped that with support these can be improved and simplified, while simultaneously improving the effectiveness of the software.

Link to the github site can be found here: https://mobile-chap.github.io/Web/

Figure 15: Adding 40m overshot to both ends of each survey line

UgCS is easy-to-use software for planning and flying UAV drone-survey missions. It supports almost any UAV platform, providing convenient tools for areal and linear surveys and enabling direct drone control. What’s more, UgCS enables professional land survey mission planning using photogrammetry techniques.

How to plan photogrammetry mission with UgCS

Standard land surveying photogrammetry mission planning with UgCS can be divided into following steps :

1. Obtain input data
2. Plan mission
3. Deploy ground control points
4. Fly mission
5. Image geotagging
6. Data processing
7. Map import to UgCS (optional)

https://www.youtube.com/watch?v=SalCQoG6WUw&feature=youtu.be

Step one: Obtain input data

Firstly, to reach the desired result, input settings have to be defined:

• Required GSD (ground sampling distance – size of single pixel on ground),
• Survey area boundaries,
• Required forward and side overlap.

GSD and area boundaries are usually defined by the customer’s requirements for output material parameters, for example by scale and resolution of digital map. Overlap should be chosen according to specific conditions of surveying area and requirements of data processing software.

Each data processing software (e.g., Pix4D, Agisoft Photoscan, Dronedeploy, Acute 3d) has specific requirements for side and forward overlaps for different surfaces. To choose correct values, please refer to documentation of chosen software. In general, 75% forward and 60% side overlap will be a good choice. Overlapping should be increased for areas with small amount of visual cues, for example for deserts or forests.

Often, aerial photogrammetry beginners are excited about the option to produce digital maps with extremely high resolution (1-2cm/pixel), and to use very small GSD for mission planning. This is very bad practice. Small GSD will result in longer flight time, hundreds of photos for each acre, tens of hours of processing and heavy output files. GSD should be set according to the output requirements of the digital map.

Other limitations can occur. For example, GSD of 10cm/pixel is required, but designed to use a Sony A6000 camera. Based on mentioned GSD and camera’s parameters, the flight altitude would be set to 510 meters. In most countries, maximum allowed altitude of UAV’s (without special permission) is limited to 120m/400ft AGL (above ground). Taking into account the maximum allowed altitude, the maximum possible GSD in this case could be no more than 2.3cm.

Step two: Plan your mission

Mission planning consists of two stages:

• Initial planning,
• Route optimisation.

-Initial planning:

The first step is to set surveying area using UgCS Photogrammetry tool. Area can be set using visual cues on underlying map or using exact coordinates of edges. The result – survey area is marked with yellow boundaries (Figure 1).

Figure 1: Setting the survey area

The next step is to set GSD and overlapping for the camera in Photogrammetry tool’s settings window (Figure 2).

Figure 2: Setting camera’s Ground Sampling Distance and overlapping

To take photos in Photogrammetry tool’s setting window, define the control action of the camera (Figure 3). Set camera by distance triggering action with default values.

Figure 3: Setting camera’s control action

At this point, initial route planning is completed. UgCS will automatically calculate photogrammetry route (see Figure 4).

Figure 4: Calculated photogrammetry survey route before optimisation

-Route optimisation

To optimise the route, it’s calculated parameters should be known: altitude, estimated flight time, number of shots, etc.

Part of the route’s calculated information can be found in the elevation profile window. To access the elevation profile window (if it is not visible on screen) click the parameters icon on the route card (lower-right corner, see Figure 5), and from the drop-down menu select show elevation:

Figure 5: Accessing elevation window from Route cards Parameters settings

The elevation profile window will present an estimated route length, duration, waypoint count and min/max altitude data:

Figure 6: Route values in elevation profile window

To get other calculated values, open route log by clicking on route status indicator: the green check-mark (upper-right corner, see Figure 7) of the route card:

Figure 7: Route card and status indicator, Route log

Using route parameters, it can be optimised to be more efficient and safe.

-Survey line direction

By default, UgCS will trace survey lines from south to north. But, in most cases, it will be more optimal to fly parallel to the longest boundary line of the survey area. To change survey line direction, edit direction angle field in the photogrammetry tool. In the example, by changing angle to 135 degrees, the number of passes is reduced from five (Figure 4) to four (Figure 8) and route length is 1km instead of 1.3km.

Figure 8: Changed survey line angle to be parallel to longest boundary

-Altitude type

UgCS Photogrammetry tool has the option to define how to trace the route according to altitude, with constant altitude above ground (AGL) or above mean sea level (AMSL). Please refer to your data processing software requirements as to which altitude tracking method it recommend.

In the UgCS team’s experience, the choice of altitude type depends on desired result. For orthophotomap (standard aerial land survey output format) it is better to choose AGL to ensure constant GSD for the entire map. If the aim is to produce DEM or 3D reconstruction, use AMSL so the data processing software has more data to correctly determine ground elevation by photos in order to provide more qualitative output.

Figure 9: Elevation profile with constant altitude above mean sea level (AMSL)

In this case, UgCS will calculate flight altitude based on the lowest point of the survey area.

If AGL is selected in photogrammetry tool’s settings, UgCS will calculate the altitude for each waypoint. But in this case, terrain following will be rough if no “additional waypoints” are added (see Figure 10).

Figure 10: Elevation profile with AGL without additional waypoints

Therefore, if AGL is used, add some “additional waypoints” flags and UgCS will calculate a flight plan with elevation profile accordingly (see Figure 11).

Figure 11: Elevation profile with AGL with additional waypoints

-Speed

In general, if flight speed is increased it will minimise flight time. But high speed in combination with large camera exposure can result in blurred images. In most cases 10m/s is the best choice.

-Camera control method

UgCS supports 3 camera control methods (actions):

1. Make a shot (trigger camera) in waypoint,
2. Make shot every N seconds,
3. Make shot every N meters.

Not all autopilots support all 3 camera control options. For example (quite old) DJI A2 does support all three options, but newer (starting from Phantom 3 and up to M600) cameras support only triggering in waypoints and by time. DJI promised to implement triggering by distance, but it’s not available yet.

Here are some benefits and drawbacks for all three methods:

Table 1: Benefits and Drawback for camera triggering methods

In conclusion:

• Trigger in waypoints should be preferred when possible
• Trigger by time should be used only if no other method is possible
• Trigger by distance should be used when triggering in waypoints is not possible to use

To select triggering method in UgCS Photogrammetry tool accordingly, use one of three available icons:

• Set camera mode
• Set camera by time
• Set camera by distance

-Glibal control

Drones, e.g., DJI Phantom 3, Phantom 4, Inspire, M100 or M600 with integrated gimbal, have the option to control camera position as part of an automatic route plan.

It is advisable to set camera to nadir position in the first waypoint, and in horizontal position before landing to prevent lenses from potential damage.

To set camera position, select the waypoint preceding the photogrammetry area and click set camera attitude/zoom (Figure 12) and enter “90” in the “Tilt” field (Figure 13).

Figure 12: Setting camera attitude

Figure 13: Setting camera position

As described previously, this waypoint should be a Stop&Turn type, otherwise the drone could skip this action.

To set camera to horizontal position, select last waypoint of survey route and click set camera attitude/zoom and enter “0” in the “Tilt” field.

-Turn types

Most autopilots or multirotor drones support different turn types in waypoints. Most popular DJI drones have three turn-types:

• Stop and Turn: drone flies to the fixed point accurately, stays at that fixed point and then flies to next fixed point.
• Bank Turn: the drone would fly with constant speed from one point to another without stopping.
• Adaptive Bank Turn: It is almost the same performance like Bank Turn mode (Figure 13), but the real flight routine will be more accurately than Bank Turn.

It is advisable not to use Bank Turn for photogrammetry missions. Drone interprets Bank Turns as “recommendation destination waypoint”. The drone will fly towards this direction but will almost never pass through the waypoint. Because drone will not pass the waypoint, no action will be executed, meaning the camera will not be triggered, etc.

Adaptive Bank Turn should be used with caution because a drone can miss waypoints and, again, no camera triggering will be initiated.

Figure 14: Illustration of typical DJI drone trajectories for Bank Turn and Adaptive Bank Turn types

Sometimes, adaptive bank turn type has to be used in order to have shorter flight time compared to stop and turn. When using adaptive bank turns, it is recommended to use overshot (see below) for the photogrammetry area.

-Overshot

Initially overshot was implemented for fixed wing (airplane) drones in order to have enough space for manoeuvring a U-turn.

Overshot can be set in photogrammetry tool to add an extra segment to both ends of each survey line.

Figure 15: Adding 40m overshot to both ends of each survey line

In the example (Figure 15) can be seen that UgCS added 40m additional segments to both ends of each survey line (comparing to Figure 8).

Adding overshot is useful for copter-UAVs in two situations:

1. When Adaptive Bank Turns are used (or similar method for non-DJI drones), adding overshot will increase the chance that drone will precisely enter survey line and camera control action will be triggered. UgCS Team recommends to specify overshot that is approximately equal to distance between the parallel survey lines.
2. When Stop and Turn type is in use in combination with action to trigger camera in waypoints, there is a possibility that before making the shot, drone will start rotation to next waypoint – it can result in having photos with wrong orientation or blurred. To avoid that, shorter overshot has to be set, for example 5m. Don’t specify too short value (< 3m) because some drones could ignore waypoints, that are too close.

Figure 16: Example of blurred image taken by drone in rotation to next waypoint

-Takeoff point

It is important to check the takeoff area at site before flying any mission! To better explain best practice on how to set takeoff point, first discuss an example of how it should not be done. Supposing that the takeoff point in our example mission (Figure 17) would be from the point marked with the airplane-icon, and drone pilot would upload the route on the ground with set automatic mission for automatic take-off.

Figure 17: Take-off point example

Most drones in automatic takeoff mode would climb to low altitude about 3-10meters and then fly straight towards the first waypoint. Other drones would fly towards first waypoint straight from ground. Looking closely at the example map (Figure 17), some trees between the takeoff point and the first waypoint can be noticed. In this example, the drone more likely will not reach a safe altitude and will hit the trees.

Not only the surroundings can affect takeoff planning. Drone manufacturers can change drones elevation behavior in drone firmware, therefore after firmware updates it is recommended that you check drones automatic takeoff mode.

Also, a very important consideration is that most small UAVs use relative altitude for mission planing. Altitude counted relatively according to first waypoint is a second reason why an actual takeoff point should be near the first waypoint, and on the same terrain level.

UgCS Team recommends placing the first waypoint as close as possible to actual takeoff point and specifying a safe takeoff altitude (≈30m in most situations will be above any trees, see Figure 18). This is the only method that warrants safe takeoff for any mission. It also protects from any weird drone behaviour and unpredictable firmware updates, etc.

Figure 18: Route with safe take-off

-Entry point to the survey grid

In the previous example, (see Figure 18), it can be noticed, that after adding the takeoff point, the route’s survey grid entry point was changed. This is because if additional waypoint is added subsequently to the photogrammetry area, UgCS will plan to fly the survey grid starting from nearest corner to the previous waypoint.

To change the entry point to survey grid, set additional waypoint close to the desired starting corner (see Figure 19).

Figure 19: Changing survey grid entry point by adding additional waypoint

-Landing point

If no landing point will be added outside the photogrammetry area after the survey mission, the drone will fly and hover in the last waypoint. There are two options for landing:

1. Take manual control over the drone and fly to landing point manually,
2. Activate the Return Home command in UgCS or from Remote Controller (RC).

In situations when the radio link with the drone is lost, for example if the survey area is large or there are problems with the remote controller, depending on the drone and it’s settings, one of these actions can occur:

• Drone will return to home location automatically if lost radio link with ground station,
• Drone will fly to last waypoint of survey area and hover as long as battery capacity will enable that, then: drone will perform emergency landing, or it will try to fly to home location.

The recommendation is to add an explicit landing points to the route in order to avoid relying on unpredictable drone behavior or settings.

If the drone doesn’t support automatic landing, or the pilot prefers to land manually, place the route’s last waypoint over the planned landing point with an altitude for comfortable manual drone descending and landing above any obstacles in the surrounding area. In general 30m is best choice.

-Action execution

Photogrammetry tool has a magic parameter “Action Execution” with three possible values:

• Every point
• At start
• Forward passes

This parameter defines how and where camera actions specified for photogrammetry tool will be executed.

The most useful option for photogrammetry/survey missions is to set forward passes, the drone will make photos only on survey lines, but will not make excess photos on perpendicular lines.

-Complex survey areas

UgCS enables photogrammetry/survey mission planning for irregular areas, having functionality to combine any number of photogrammetry areas in one route, avoiding splitting the area in separate routes.

For example, if a mission has to be planned for two fields connected in a T-shape, and if these two fields are marked as one photogrammetry area, the whole route will not be optimal, regardless any direction of survey lines.

Figure 20: Complex survey area before optimisation

If the survey area is marked as two photogrammetry areas within one route, survey lines for each area can be optimised individually (see Figure 21).

Figure 21: Optimised survey flight passes for each part of a complex photogrammetry area

Step three: deploy ground control points

Ground control points are mandatory if the survey output map has to be precisely aligned to coordinates on Earth.

There are lots of discussions about the necessity of ground control points in cases when a drone is equipped with Real Time Kinematics (RTK) GPS receivers with centimeter-level accuracy. This is useful, but the drone coordinates are not in themselves sufficient because, for precise map aligning, image center coordinates are necessary.

Data processing softwares like Agisoft Photoscan, Dronedeplay, Pix4d, Icarus OneButton and others will produce very accurate maps using geotagged images, but the real precision of the map will not be known without ground control points.

Conclusion: ground control points have to be used to create survey-grade result. For a map with approximate precision, it is sufficient to rely just on RTK GPS and the capabilities of data processing software.

Step four: fly your mission

For carefully planned missions, flying it is the most straightforward step. Mission execution differs according to the type of UAV and equipment used, therefore it will not be described in detail in this article (please refer to equipment’s and UgCS documentation).

Important issues before flying:

• In most countries there are strict regulations for UAV usage. Always comply with the regulations! Usually these rules can be found on web-site of local aviation authority.
• In some countries special permission for any kind of aerial photo/video shooting is needed. Please check local regulations.
• In most cases missions are planned before arriving to flying location (e.g., in office, at home) using satellite imaginary from Google maps, Bing, etc. Before flying always check actual circumstances at the location. There could be a need to adjust take-off/landing points, for example, to avoid tall obstacles (e.g., trees, masts, power lines) in your survey area.

Step five: image geotagging

Image geotagging is optional if ground control points were used, but almost any data processing software will require less time to process geotagged images.

Some of the latest and professional drones with integrated cameras can geotag images automatically during flight. In other cases images can be geotagged in UgCS after flight.

Very important: UgCS uses the telemetry log from drone, that is received via radio channel, to extract the drone’s altitude for any given moment (when pictures were taken). To geotag pictures using UgCS, assure robust telemetry reception during flight.

For detailed information how to geotag images using UgCS refer to UgCS User Manual.

Step six: data processing

For data processing, use third party software or services available on the market.

From UgCS Team experience, the most powerful and flexible software is Agisoft Photoscan (http://www.agisoft.com/), but sometimes too much user input is required to get necessary results. The most uncomplicated solution for users is online service Dronedeploy (https://www.dronedeploy.com/). All other software packages and services will fit somewhere between these two in terms of complexity and power.

Step seven (optional): import created map to UgCS

Should the need arise for the mission to be repeated in the future, UgCS enables importing the GeoTiff file as a map layer and using it for mission planning. More detailed instructions can be found in UgCS User Manual. See the result of an imported map created using UgCS photogrammetry tool imported as GeoTiff file in Figure 22.

Figure 22: Imported GeoTiff map as layer. The map is output of a Photogrammetry survey mission panned with UgCS

Visit the UgCS homepage

Download this tutorial as a PDF

If you liked this tutorial, you may also enjoy these:

• Programming for robotics: Introduction to ROS
• Python programming your NAO robot
• Robotics, maths, python: A fledgling computer scientist’s guide to inverse kinematics
• Adjust how you learn and quickly pick up robotics programming
• 3-D printing hydraulically-powered robots, no assembly required

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

The feedback system enables human operators to correct the robot’s choice in real-time – Jason Dorfman, MIT CSAIL

For robots to do what we want, they need to understand us. Too often, this means having to meet them halfway: teaching them the intricacies of human language, for example, or giving them explicit commands for very specific tasks. But what if we could develop robots that were a more natural extension of us and that could actually do whatever we are thinking?

A team from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and Boston University is working on this problem, creating a feedback system that lets people correct robot mistakes instantly with nothing more than their brains.

Using data from an electroencephalography (EEG) monitor that records brain activity, the system can detect if a person notices an error as a robot performs an object-sorting task. The team’s novel machine-learning algorithms enable the system to classify brain waves in the space of 10 to 30 milliseconds.

While the system currently handles relatively simple binary-choice activities, the paper’s senior author says that the work suggests that we could one day control robots in much more intuitive ways.

“Imagine being able to instantaneously tell a robot to do a certain action, without needing to type a command, push a button or even say a word,” says CSAIL director Daniela Rus. “A streamlined approach like that would improve our abilities to supervise factory robots, driverless cars and other technologies we haven’t even invented yet.”

In the current study the team used a humanoid robot named “Baxter” from Rethink Robotics, the company led by former CSAIL director and iRobot co-founder Rodney Brooks.

The paper presenting the work was written by BU PhD candidate Andres F. Salazar-Gomez, CSAIL PhD candidate Joseph DelPreto, and CSAIL research scientist Stephanie Gil under the supervision of Rus and BU professor Frank H. Guenther.  The paper was recently accepted to the IEEE International Conference on Robotics and Automation (ICRA) taking place in Singapore this May.

Past work in EEG-controlled robotics has required training humans to “think” in a prescribed way that computers can recognize. For example, an operator might have to look at one of two bright light displays, each of which corresponds to a different task for the robot to execute.

The downside to this method is that the training process and the act of modulating one’s thoughts can be taxing, particularly for people who supervise tasks in navigation or construction that require intense concentration.

Rus’ team wanted to make the experience more natural. To do that, they focused on brain signals called “error-related potentials” (ErrPs), which are generated whenever our brains notice a mistake. As the robot indicates which choice it plans to make, the system uses ErrPs to determine if the human agrees with the decision.

The team uses EEG brain signals to detect if the person notices a mistake – Jason Dorfman, MIT CSAIL

“As you watch the robot, all you have to do is mentally agree or disagree with what it is doing,” says Rus. “You don’t have to train yourself to think in a certain way –  the machine adapts to you, and not the other way around.”

ErrP signals are extremely faint, which means that the system has to be fine-tuned enough to both classify the signal and incorporate it into the feedback loop for the human operator.

In addition to monitoring the initial ErrPs, the team also sought to detect “secondary errors” that occur when the system doesn’t notice the human’s original correction.

“If the robot’s not sure about its decision, it can trigger a human response to get a more accurate answer,” says Gil. “These signals can dramatically improve accuracy, creating a continuous dialogue between human and robot in communicating their choices.”

The feedback system enables human operators to correct the robot’s choice in real-time – Jason Dorfman, MIT CSAIL2

While the system cannot yet recognize secondary errors in real time, Gil expects the model to be able to improve to upwards of 90 percent accuracy once it can.

In addition, since ErrP signals have been shown to be proportional to how egregious the robot’s mistake is, the team believes that future systems could extend to more complex multiple-choice tasks.

Salazar-Gomez notes that the system could even be useful for people who can’t communicate verbally: a task like spelling could be accomplished via a series of several discrete binary choices, which he likens to an advanced form of the blinking that allowed stroke victim Jean-Dominique Bauby to write his memoir “The Diving Bell and the Butterfly.”

“This work brings us closer to developing effective tools for brain-controlled robots and prostheses,” says Wolfram Burgard, a professor of computer science at the University of Freiburg who was not involved in the research. “Given how difficult it can be to translate human language into a meaningful signal for robots, work in this area could have a truly profound impact on the future of human-robot collaboration.”

Read the paper here.

The project was funded in part by Boeing and the National Science Foundation.

If you enjoyed this article from CSAIL, you might also be interested in:

• MIT CSAIL’s 6-foot-tall NASA humanoid robot has landed
• Researchers add a splash of human intuition to planning algorithms
• Wearable AI that can detect the tone of a conversation
• Ingestible robots, glasses-free 3-D, and computers that explain themselves
• Design, simulate and build a custom drone

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

The NAIST OpenHand M2S was developed by a team of students as part of the school’s annual CICP project (read the blog post about it here), in which students can propose and organize their own research projects. Based on the Yale OpenHand M2, the NAIST OpenHand M2S was developed for textile manipulation, sensitive grasping as well as pinpoint pushing with high loads. All of the parts (except motors, sensors etc.) can be downloaded from here and 3D printed.

For grasping and manipulation, the hand is equipped with two 3D force sensors that act as fingertips. Using these, the hand can gently grasp textiles and let them glide through its fingers, pull them taut when tension is required, and even recognize different materials. By rubbing its fingertips together, a force signal is generated which allows the hand to sense if it successfully grasped a textile, and which kind of material it is.

Lastly, the rigid finger of the hand allows it to push with high loads and tuck into small openings. This can be used for tasks like making a bed, but also during the manufacturing of a car or airplane seat, which is part of the team’s research. Future versions of the hand will be able to pick up thin objects from a table easily without moving the robot.

You can read the conference paper here. Download the 3D files here, to use the gripper in your own research.

If you liked this article, you may also enjoy these:

• Robotics, maths, python: A fledgling computer scientist’s guide to inverse kinematics
• Python programming your NAO robot
• Soft robotic gripper can pick up and identify wide array of objects
• Sticky business: Five adhesives tested for 3D printing
• 3-D printing hydraulically-powered robots, no assembly required

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this episode, Abate De Mey interviews two speakers from the Agricultural track of the RoboUniverse 2016 conference in San Diego: Dan Harburg of Soft Robotics Inc. and Matthew Borzage of BioTac. Borzage and Harburg discuss their distinct approaches to advancing gripping technology in Agriculture. Borzage stresses the importance of tactile sensing while Harburg pushes for low cost, soft grippers with no on-board sensors.

Dan Harburg
Dan Harburg is the Director of Business Development at Soft Robotics Inc., a venture-backed startup company developing a soft end-of-arm tool for the food industry. Harburg was previously an entrepreneur-in-residence at North Bridge Venture Partners, working with scientists at Harvard, MIT, and the University of Illinois to commercialize technologies at the intersection of materials, medicine, and advanced manufacturing.

Matthew Borzage
Matthew Borzage is a co-founder and the Head of Business Development for SynTouch. Borzage has a background in biomedical and medical device engineering and sales. As Head of Business Development, Matt is interested in how tactile sensing can be used in robotics and prosthesis. 

Links

• Soft Robotics Inc.
• Bakery Automation with a Soft Touch
• SynTouch Inc.
• Biomimetic Design of BioTac
• Download Episode (16.2 MB)

In this episode, Ron Vanderkley spoke to Dr Moritz Tenorth, head of software development at Magazino, a Munich-based startup developing mobile pick-and-place robots for item-specific logistics. They discussed his work on the Toru robot and what it means to the warehouse industry today and in the future.

Dr Moritz Tenorth

Dr. Moritz Tenorth

After obtaining his PhD in robotics from TU Munich Dr Moritz Tenorth spent several years as a post-doc and freelance robotics consultant, with stays at the CMU Robotics Institute and the ATR labs in Japan. His research focuses on knowledge representation methods that can help autonomous robots make smarter decisions, and on methods for task coordination in robotics.

 Links

• Magazino (website)
• Toru Cube – Pick by robot (Youtube)
• Pick-by-Robot | technical features of TORU Cube (YouTube)
• Magazino TORU Cube – DHL Logistik-Center (YouTube)
• Download mp3 (10.1 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

The Arm

The design I used is based on industrial pallet-packing robots, and at its core has three degrees of freedom, or ‘axes’ on which it can move. Think left to right, in and out and up and down; it basically means that the arm can move in three different ways. The GIFs below show the different types of movement: the entire arm can swivel on its base, the “main arm” translates the elbow in and out from the centre, and the “actuator” drives the forearm, which in turn translates the end-effector up and down. A 3-DOF arm like this has the handy property of having a unique solution for each possible position of its end-effector, which made the IK task approachable to someone like me, who’s never done this type of thing before.

Swing

Main

Actuator

Like many well-studied problems, IK has generic solutions, but these are numerically-based and computationally complex. I wanted a simple and efficient solution, so I decided to approach the control of my arm analytically, using geometric principals and logic. The result of this work was a graphical Python application, which connected to the arm via serial and allowed the user to move the goal position of the end-effector by clicking and dragging, or by sending commands from another application running on the same computer (this IK application made up a key part of my project, but that’s a topic for another time).

Here’s a quick look at what I ended up with:

Heads up: The code in this post is written in Python with Numpy, and there’s maths. I won’t pretend that any of the ideas presented here are optimal (or good, for that matter), but hopefully you’ll be able to scrounge something useful!

Let’s go already

To start with, let’s consider the “swing” action of the arm, as it rotates on its base. Since the main arm, elbow and forearm are all constrained to a single plane (look at the GIFs above: only the swing rotation affects the left-right position of the wrist), all we really have to do here is line up that plane – a line from our top-down perspective – with the goal point, ignoring the goal’s vertical position. If this sounds easy, you’re right: it’s plain old trigonometry. Unfortunately for me, a quirk of my design left the arm’s operation plane just off-centre of the axis of rotation, which made the solution much less obvious.

Using the naive trigonometric approach here results in pointing off to the left of the actual goal (right if the shoulder sits on the other side), and the gap is big enough that this is a real problem. I took a shot at solving this geometrically, but quickly decided it wasn’t worth my time figuring it out. In the end, I resorted to an extremely simple numerical technique called a binary search. I’ve used this in the past for other tough geometric problems, like finding an intersection between a Bezier curve and a circle.

In a binary search (also known as the “Bisection Method”), you essentially come up with arbitrary upper and lower bounds for your variable – in this case, theta – and then examine the midpoint to find whether it’s larger or smaller than your target. If it’s larger, then the midpoint becomes your new upper bound; if it’s smaller, the midpoint is your new lower bound. Repeat until you’ve bounded your variable with sufficient accuracy. A binary search is conceptually simple and computationally cheap, and converges fairly quickly: you halve your search space each iteration. I found 3-5 iterations got me within 0.3 degrees of the true value, well within the limits of what my servo motors could practically target.

On to the solution! What we’re actually searching for is the point at which the vector representing our arm is lined up with the vector from the arm’s shoulder joint to the goal: where the angle between these two vectors is zero. Normally, when taking the angle between two vectors (call them A and B) you get the absolute angle out, with no directionality. However, with this search method it’s important to have a signed distinction between “too high” and “too low”. To get around this, I wrote a simple function to find a signed angle between two vectors, with negative showing that A points counter-clockwise to B, and positive showing that A points clockwise from B:

This works by taking a perpendicular vector from B (pointing clockwise), and dot-ing it with A to see how much they line up (using the dot product for
vector projection).

If A points clockwise of B then np.dot(A, perp) will be positive, and vice-versa.

Note that the normalize() function simply sets a vector’s length to 1, while keeping it pointed in the same direction. Mathematically, it’s just doing: A=|A|^-1 (np.clip() just keeps the dot product between 1 and -1).

With that, we can now perform our binary search and find a decent approximation to the target angle. We need a reasonable starting point, so let’s just use the angle from the origin to the goal (the naive approach I mentioned above). Numpy provides a very useful function for finding the angle np.arctan2(y, x) which resolves the arc-tangent of in the correct quadrant.

Now, it’s important to check which side the shoulder sits on: this lets us decide whether to use start_theta as an upper or lower bound. This is as simple as testing the signed angle between the shoulder offset vector (from the origin) and a vector pointing straight forwards. At the same time, we can also establish the other bound by taking the absolute value of this angle and adding or subtracting it from start_theta, since this angle to the shoulder is always going to be greater than the error of start_theta.

With our initial bounds established, the actual search loop is pretty straightforward. Note the use of an iteration counter: it’s only there in case something goes wrong (e.g. if we input an unsolvable configuration), since in normal operation the loop exits once the desired accuracy is reached. We iteratively update self.swing, which will hold the new target swing angle at the end of the search.

Note that the self.shoulder(angle) function just generates the top-down (x, z) position of the shoulder joint for a given swing angle. Besides the swing angle, there’s one other important result from this stage of IK: the radial distance, from the shoulder joint to the end-effector. The calculation is simple enough:

goal_radial = np.linalg.norm(self.goaltd – self.shoulder(self.swing))

Round in circles

We know which way to point: now, let’s consider the movement of the arm in its ‘plane of operation’, which covers the goal end-effector position and the line extending vertically (+y) from the shoulder joint. The main arm and forearm both move across this plane. The trick here is that we’ll still be using a 2D coordinate system, basically a local frame of reference, where our y value remains the same but the x value is the radial distance of the point we’re interested in. For the goal, we computed this radial distance above in the top-down IK, so we can go ahead and construct goalpl – the ‘planar’ goal – which lies at: goalpl=(goal_radial/goal_y).

Side-on, the main arm and forearm form an upright triangle between the shoulder joint and goalpl with the elbow joint sitting at the peak. It’s important to note that the arm’s sections are different lengths, with the forearm being a bit longer. Now, in this plane we’re solving given the end effector’s position, and the origin is static. That just leaves the position of the elbow joint to solve for, which we can do geometrically, since it’s at a fixed distance from both the origin and effector. If we draw two circles, one fixed about the origin with radius = main arm length, and one centred on the end-effector with radius = forearm length, then the elbow must be at one of the two possible intersections between these circles:

Let’s look at a simple Python class which deals with this intersection calculation. There’s a few possible cases for circle-circle intersection, most of which invalidate our solver: if the circles do not contact, either by being separated or in the case that one contains another, then there is no valid solution; if the circles are coincident (impossible for this design, since the radii differ) then there are infinite possible solutions; however, when the circle borders intersect, we can find two distinct intersection points.

For any case we’re interested in, it’s the top intersection that matters. This means we can just pick the one with the largest y value, and call it a day:

And – that’s it! We have the swing angle, and a valid position for the elbow joint to match the shoulder and goal end-effector positions. That’s all we need to define a valid pose for the arm. If you need to, you can reconstruct the positions in 3D space:

Wrapping up

If you’ve made it this far, I hope you’ve found this interesting and/or enlightening. Want to see how it all fits together? I’ll be uploading the full IK application to GitHub here, along with all the design files for the robotic arm. If you’re feeling ambitious, you should be able to 3D-print and build your own to play around with

Other notes:

• I didn’t design the arm completely from scratch: check out the original “LiteArm” design on Thingiverse.
• The IK application uses PyGame, a no-frills SDL wrapper for Python, to open and draw to the graphical window. I’ll probably switch to a meatier library for further development, but this did the job of basic “drawn-to-screen” stuff nicely.

Written by Alistair Wick for the University of Bristol Engineering Maths blog. The original post, and future follow-ups, are available on Alistair’s website.

If you liked this article, you may also enjoy these:

• TrotBot tackles rough terrain
• Sticky business: Five adhesives tested for 3D printing
• Meet RoboGabby: She’s on a mission to teach more girls about robotics
• Adjust how you learn and quickly pick up robotics programming
• 3-D printing hydraulically-powered robots, no assembly required

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this episode, Abate De Mey interviews Edward Neff, founder of SMAC Corporation. Mr. Neff discusses how breakthroughs in his company have allowed them to develop linear actuators compact enough to be used to actuate robotic fingers. Companies like Apple and Samsung push for the development of robotic fingers to perform lifelike tests on their phones.

The robotic fingers developed using SMAC’s linear servo motors simulate a button press on a screen by employing a strong downwards force, as well as small sideways forces tangential to the screen. This mimics a realistic way that people interact with their touchscreens. The robotic fingers can also gently bump into surfaces, pull, and push objects. Other applications for these fingers include electronic circuit board assembly, and screwing bottle caps onto bottles.

Edward Neff

Edward Neff grew up in Birmingham, Michigan (USA) and is a graduate of the University of Michigan. He has a career history of managing automation companies in Belgium and Japan. Mr. Neff is the founder of SMAC Corporation in Carlsbad, California. SMAC is a leading manufacturer of precision programmable electric actuators. The company is planning a 2018 IPO. Mr Neff has over 40 patents in robotics and serves as a Director in BioTech and Medical Device companies.

Links

• SMAC Website
• Robotic Finger using a SmartPhone (YouTube)
• Robotic Finger at the Tokyo Robot Show (YouTube)
• Robotic Finger VS Human (YouTube)
• Smart Assembly Video (YouTube)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

RoDyMan will be an innovative mobile platform with a torso, two lightweight arms and a sensorized head that will execute complex manipulation movements similar to the ones humans carry out on a daily basis. The tasks will be carried out in the presence of humans, hence requiring human-robot cooperation.

Siciliano and his team are working an a very specific goal for this project and the inspiration came from their home city of Naples: a platform for performing all the phases of making a pizza. They plan to present this application by May 2018, and to find out more why this date was chosen, we invite you to watch Siciliano’s video lecture in its entirety.

Download the full presentation here. 

Bruno Siciliano is a Professor of Control and Robotics, and Director of the PRISMA Lab in the Department of Electrical Engineering and Information Technology at the University of Naples. His research interests include: identification and adaptive control, impedance and force control, visual tracking and servoing, redundant and cooperative manipulators, lightweight flexible arms, aerial robots, human-centered and service robotics.

He has co-authored 11 books, 70 journal papers, 200 conference papers and book chapters; his book Robotics: Modelling, Planning and Control is one of the most widely adopted textbooks worldwide. He has delivered more than 100 invited lectures and seminars at institutions worldwide.

Siciliano B. IJARS Video Series:  Human Motion Mapping to a Robot Arm with Redundancy Resolution – The RoDyMan Project [online video]. International Journal of Advanced Robotic Systems, 2015, 12:V5. DOI: 10.5772/61194

• Manufacturing next-generation robotic manipulation
• 2015 sees mobile manipulators coming to market
• Grasping objects in a way that is suitable for manipulation
•  Aerial Manipulation: Lecture by Bruno Siciliano
• Team RBO from Berlin wins Amazon Picking Challenge convincingly
• Shadow Robot’s Dexterous Hand can do more than cook up a storm
• Teaching a robot to ‘cook’ by showing it YouTube videos of cooking shows

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this episode, Audrow Nash speaks to several robotics companies at the company showcase at RoboBusiness 2014, which took place in Boston, Massachusetts.

The following is an ordered list of the people interviewed.

1. Norm Domholt (1m 40s), CEO of NPC Robotics,  talks about their modular remote operate vehicle Black Dog.
2. Annemarie Laudanski (6m 20s), Lead Field Applications Engineer at Xsens, demos their inertial motion capture system.
3. Carl Vause (9m 14s), CEO of Soft Robotics, speaks about the capability of the soft robotic manipulators, which can be used to pick fruit, among other applications.
4. Konrad Fagertun (13m 56s), COO of NLinks, tells about their new robot which drills holes in ceilings – a laborious and difficult task for humans.
5. Asim Ikram (15m 51s), CEO of Neo Robotics, discusses the market for robots in the Gulf Cooperation Council region.
6. Harri Valpola (21m 40s), Head of Technology at Zen Robotics, talks about the companies waste sorting robotic system.
7. Dick Zhang (26m 15s), CEO of Identified Technology, speaks about their self-piloting aerial drone systems for automated energy, mining, and construction site monitoring, surveying, and mapping.

Links:

• Download mp3 (18.8 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Transcript

Norm Domholt (1m 40s), CEO of NPC Robotics:

Audrow Nash: Hi, can you introduce yourself and your company?

Norm Domholt: Yes, thank you. My name is Norm Domholt and I’m president of NPC Robotics. Today, at the show, we’re featuring and debuting our latest invention. We’ve been working on it for three years and it’s called the Black Dog.

Audrow Nash: Now, you have the Black Dog here. Can you describe it?

Norm Domholt: Black Dog is a remote control mobile platform. It’s designed modularly, so that it can be repaired in the field without probing with any sort of instruments and with the use of one particular tool – a quarter inch hex wrench. All the parts can be taken apart and replaced in minutes.

Audrow Nash: Can you tell me a bit more about how it’s modular?

Norm Domholt: We realized, through our experience with Battle Box, that the teams that seemed to win were the teams that designed their robots to be repaired quickly and without the need to take apart wiring.

It’s something that can be plugged in and replaced quickly. So, we decided that there should be around eight components in here that can be replaced with the use of this one tool and they should all be modular. There are no internal wires that you have to worry about, so everything unplugs and plugs together very quickly.

Audrow Nash:  So it’s a trunk-treaded robot that has two motors driving everything and it’s three feet long, a foot tall and two feet wide?

Norm Domholt: Yes, 42 inches long, 22 inches wide and 12 inches tall. It weighs 200 pounds and it goes four miles an hour. It’s got a lot of torque. Each motor is two-and-a-half horsepower – that’s a 24volt lithium battery.

Audrow Nash: Can you give me some examples of how it could be fitted with different components in the field?

Norm Domholt: Yes. Let’s say a SWAT team buys our robot power base; there are many different applications and mission-specific needs that they could have throughout a day. For example, they go to a house with something suspicious going on inside. They can put a telescoping camera, called RibbonLift, on the robot, within – say – ten seconds. They can then maneuver it to the building from a safe distance. If they hear shots fired and see there’s no need for the camera, they can pull the robot back within seconds, take their camera off and put a 300 pound bullet-proof shield on instead. Then, a SWAT person could operate this with a thumb throttle, while crouching behind the shield, and approach the house from a safe position. He can direct it to come back and hurl a teargas canister through the window. It could be adapted to any particular use very quickly.

Audrow Nash: Very good, and can you tell me at what stage of production your company is?

Norm Domholt: We’re debuting this today. We were hoping it would be a little further along and we could operate it. Some of the electronics are not completed yet; the mechanics are almost done and the modularity is complete. The technology that’s going in is already in place and, in another two months, we’ll be testing at facilities – breaking down doors, going through hallways, up and down stairs. We’ll be testing to see how far – wirelessly – this thing can go into a building.

Audrow Nash: Who would be interested in this product?

Norm Domholt: The same people who would be interested in a multi-purpose tool. Police can use it, the home-owner… Who would use a power-base? SWAT, the oil industry, hazardous waste…

Audrow Nash: So, lots of applications?

Norm Domholt: A lot of applications, yes.

Audrow Nash: Thank you.

Norm Domholt: You’re welcome, thank you.

Annemarie Laudanski (6m 20s), Lead Field Applications Engineer at Xsens:

Audrow Nash: Hi, can you introduce yourself and tell me about your company?

Annemarie Laudanski: Hi, my name is Annemarie Laudanski and I work for Xsens, we do animation promotion capture.

Audrow Nash: Can you tell me about what you have set up at this booth and describe it?

Annemarie Laudanski: Today we’re trying out our inertial motion capture system. It’s a full body- measure-capture system built of 17 inertial sensors. In each of these, there are 3-D accelerometers, gyroscopes and magnetometers. They’re used to measure the movement of arms and legs and send information wirelessly to the computer, where it’s translated into a bio-chemical model. So, we can measure exactly what someone is doing and portray that on our software.

Audrow Nash: At the booth you have a dancer?

Annemarie Laudanski: We do, yes. We hired Ashley. She’s the cheerleader for the Boston Celtics and she’s with us during the conference to showcase the use of our system.

Audrow Nash: You mentioned you were using exonorometers, gyroscopes and Megnatometers. So, it doesn’t require any external visual system to track them? You’re just taking the raw data from all of these sensors, running that through, and then feeding it to a model, to develop a person?

Annemarie Laudanski: That’s right. The exonorometers measure how fast each limb segment is moving, the gyroscopes measure the rate of turn of the limb and then we use the megnatometers to correct the drift in the other sensors. A fusion of these three sets of sensor data is sent wirelessly to the computer where it’s combined with our calibration procedure to build a human model. The calibration takes an average reading from the sensors, while the subject is standing still for five-seconds. This is a known position for taking sensor data, that then informs each of the trackers, so that the model is built correctly.

Audrow Nash: I see. As the dancer moves, it’s happening in real time.

Annemarie Laudanski: Yes, all of it is done in real time so all of the sensors are sending data at 30 bites per second to the computer and everything is shown with a latency of, at most, ten milliseconds, which is mostly due to the Microsoft operating system we are working with.

Audrow Nash: What are some applications?

Annemarie Laudanski: The applications are really wide for this type of technology. They go from entertainment purposes, where people are using it for movie and game design, to movement science, where it’s used for sport, rehabilitation or ergonomics and also human-robot interaction. So, you can even have someone in the system and driving a robot, or building up a database of human movements that can be used to program one and allow it to move like a human would.

Audrow Nash: Thank you.

Annemarie Laudanski: Thank you.

Carl Vause (9m 14s), CEO of Soft Robotics:

Audrow Nash: Hi, can you introduce yourself and the company?

Carl Vause: I’m Carl Vause and this is Soft Robotics Incorporated. We’re a Boston-based company that got spun out as a research group at Harvard about a year ago. We’re taking soft robotics technology and making it real, producing commercial products to solve new problems across a variety of markets.

Audrow Nash: Can you tell me about what you’ve displayed at the booth?

Carl Vause: At the booth today we have two soft hands: A very large hand for handling pumpkins, shoe bags and objects of very weighty geometry, and a small-handed, or short, robotic arm. What we wanted to show here is that, currently, there are a lot of problems with robotic manipulation. Manipulators don’t do well with soft, curvy, easily damaged objects and they don’t do well in unstructured gripping scenarios.

What we have solved using soft actuators is the ability to pick up anything with this arm, from a prickly cactus, which you would never pick up with your bare hand, to a racket ball or a cupcake. We want to show that, with almost no control system, building a very clever, soft actuator can solve a problem that has been vexing robotics: How to develop a gripper that is as adaptable and conformal as a human hand. People are spending millions of dollars to solve this problem using traditional hard robotics.

Audrow Nash: Can you describe the hand and the smaller robotic arm?

Carl Vause: One is a four-finger hand. It’s based on proprietary actuator geometry. We use vacuum to curl the fingers out of the way and they pressurize to create a grasp. We have a three-finger gripper as well. It actually has fingernails, so it can access a low profile object like a circuit board.

Audrow Nash: Can you tell me some potential applications of these soft hands? I know there are a lot.

Carl Vause: A lot, but how do you interface a robot with something that has an unknown shape? Or when the position is unknown? Or something easily damaged? Agricultural processing is interesting. Picking lettuces heads, for example. There’s a danger of introducing E. coli into the food chain, so washing and cleaning lettuce. Robotics companies and the US Food and Drug Administration want to automate that task. But robots today can’t handle lettuce, because their heads vary in size, shape and weight, and they are delicate objects. So, one of the things that we really wanted to do with our soft robotic technology is unlock that market for automation, and automate data handling.

They can even work in wet conditions; there are no electrical connections here. You can sterilize them and wash them down with bleach at the end of the day. It really does bring automation into the agricultural process. It makes it a reality.

There’s also a light-weight version for working side-by-side with a human collaborator. In a lab, for example where you need to be careful what you touch. A manipulator can pick up a range of lab instruments, or tools, and hand them safely to you. It has the credentials of collaboration.

Audrow Nash: The ones that you have on display are on open feedback loops. As in, they don’t sense what they’re picking up, they just actuate. You’ve talked a little bit about embedding sensors into the fingers in the hands. Can you tell me more about that?

Carl Vause: We can easily embed sensors to give an object grasp detection. We can embed it in our proprietary manufacturing process, in simple off-the-shelf sensors and feed them back into something very basic. If you want it very simple, take an Arduino and provide passive grasp detection. That’s very simple.

Audrow Nash: Thank you.

Carl Vause: Alright.

Konrad Fagertun (13m 56s), COO of NLinks:

Audrow Nash: Hi, can you introduce yourself and tell me about your company?

Konrad Fagertun: Yes, my name is Konrad Fagertun. I’m a co-founder of NLink Robotics. We’ve come all the way from Norway to Boston, to showcase the world’s first mobile drilling robot. We’re excited to be here at Robobusiness, showing what we’ve been working on for the last two years.

Audrow Nash: Can you describe the robot to me?

Konrad Fagertun: Yes. The challenge in the construction industry is hard labor – manual work – and the hardest of all is drilling holes in concrete ceilings. So, we made a mobile drilling robot. It comprises a mobile lift – that can drive around – and, on top of that, is a robotic arm holding a rotary hammer drill. It thrusts that up to the ceiling and can drill holes wherever you want. It uses lasers and cameras to position itself, so it can drill to a millimeter’s precision. This can be done hundreds of thousands of times, say for a large shopping mall or an office building. It’s repetitive work.

The robot is a little bit bigger than a man, and can reach several feet high, so you don’t have to get up on scaffolding or stairs to do the work; the operator can stand on the ground, safely at a distance. But, if he wants to get close, no problem, the robot is certified to work with humans, so it doesn’t matter if it crashes into your arm, it won’t hurt you and it’ll stop. It’s useful for electrical contractors, plumbers, mechanical engineers…

Audrow Nash: Thank you.

Konrad Fagertun: Thanks.

Asim Ikram (15m 51s), CEO of Neo Robotics:

Audrow Nash: Can you introduce yourself and your company?

Asim Ikram: Yes, my name is Asim Ikram and I’m the CEO of Neo Robotics in Dubai.

Audrow Nash: What do you guys do?

Asim Ikram: We’re a new start up in Dubai, not limited to robotics. In that region, they are not exposed to robots at the moment. We are starting at the lower level, to introduce smart apps and ICT, Information Communication Technologies. Then, we want to switch to robots, because the customers are the same.

Audrow Nash: So, basically you’re acting as a liaison between companies in the US and outside for bringing robots into Dubai?

Asim Ikram: Into the GCC region – the Gulf Cooperating Countries. There are six of them in the Middle East:  Saudi Arabia, Bahrain, Oman, Kuwait, Emirates and Qatar. They have a strong focus on technology, they’re teaching robotics and megatronics and have labs at almost every university. The industry will develop but, at the moment, there’s no pull from for robots.

Audrow Nash: What are some of the more popular technologies that are being brought to Dubai?

Asim Ikram: Not so much industrial robotics, but there are new production lines coming up that have industrial robots in the blueprint. Those lines should start operating in six months. The financing and planning is already done. Healthcare and education, that’s much further ahead. On the education side, there’s big competition between the private schools that are teaching things like systems programming. Now they’re also trying to get robots into the classrooms and teach with them. There’s a lot of movement there.

Audrow Nash: What robotics opportunities do you think there are in Dubai for engineers and entrepreneurs from the rest of the world?

Asim Ikram: There’s a lot of work in real-world drone applications. Already, highway police are using drones. They’ve also started making some national identity card deliveries with drones. Real applications of technology are moving fast. On the security side, I’ve seen a lot of civil defense robots for bomb disposal.

Audrow Nash: How did you decide to take this opportunity and form a company?

Asim Ikram: I’ve been working in this market for some time and I’ve been seeing a lot of pull in different areas. Healthcare and education are moving very fast and the industrial side will come up in the next one or two years. There are a few robotics service companies, but not many locally so I found a gap. It’s a good time to start something.

Harri Valpola(21m 40s), Head of Technology at Zen Robotics:

Audrow Nash: Can you introduce yourself and your company?

Harri Valpola: Yes, hello. I’m Harri Holopainen. I’m one of the founders of Zen Robotics Company in Helsinki, Finland. Currently, our main application is robots that have learnt to pick up waste. A little bit like WALL-E…

Audrow Nash: It looks a little different than WALL-E. Can you describe the robotic system you have?

Harri Valpola: Whereas WALL-E is cute, a small Hollywood robot, our robot is more like an large industrial scale model. It doesn’t look like a human; it’s not cute, but it works very efficiently. It’s powerful. It can lift and throw objects that weigh up to 20 kilos – 40 pounds – like they were nothing.

Audrow Nash: It looks like a conveyor belt and has robotic arms that move around and pick objects off of the conveyor belt and throw them to sort them. Am I correct?

Harri Valpola: Yes. It works in a similar way to human hands. All around the waist is a conveyor belt. The robot recognizes the objects and sorts them, exactly as a human would do, but faster. It’s also stronger and it doesn’t break.

Audrow Nash: What kind of sensors are you using to identify the material? What waste are you sorting now?

Harri Valpola: We are currently focused on construction and demolition waste. The sensors we are using are tailored for that particular waste. The most important material we’re sorting is  wood, followed by mineral-based parts and metal. We’re using sensors that can discriminate between these. They include metal detectors and spectrometers.

Audrow Nash: Where have you implemented the system and what sort of feedback are you getting from customers?

Harri Valpola: Currently, we have three customers in Europe; two in Finland and one in Holland. We’ve been developing this system for two years and now it’s working and customers are happy. We’re ready to scale up, now we have the product, so that other customers can start making use of this solution.

Audrow Nash: What’s the benefit of sorting the waste?

Harri Valpola: In Europe, at least, there are many companies that are floating in waste. Population densities are so high, there’s no place to put it. The only real option is to sort the waste. There are many valuable raw materials there. Sorting stones is not important but there are valuable metals, like copper and aluminum, and also wood. That’s particularly useful in Holland because they don’t have forests, and it can be recycled and used as a construction material.

Audrow Nash: Thank you.

Harri Valpola: Thank you.

Dick Zhang (26m 15s), CEO of Identified Technology:

Audrow Nash: Interviewer:  Can you introduce yourself and tell me about your company?

Dick Zhang: My name is Dick Zhang from Identified Technologies. We build and scale drones for commercial data capture. We work with construction and energy industry clients and help deliver data and insights. We get this information either by deploying robots on the ground, or by utilizing commercial planes and helicopters. It’s very expensive, it’s very dangerous and it’s very slow.

We deliver insights by collecting massive scale amounts of data on what has happened in the past, what’s going on now and what’s going to happen in the future, through predictive capabilities.

We collect the data via a three-part system: an autonomous drone, a dock station, that swaps batteries and uploads data, and a client dashboard that can be used to interact with and control the system from the cloud, from anywhere in real time.

We’ve been working with really exciting partners in energy and construction for the last six months. I run the business, look for exposure and develop partner and client relationships.

Audrow Nash: Can you tell me a bit about the demonstration – the drone, the quadcopter – you have on your booth?

Dick Zhang: The drone itself, everything inside, in terms of hardware, is really off-the-shelf components. Our dock station, that swaps batteries and outputs data, that’s all proprietary mechanical design and hardware. The software system is also proprietary. So a lot goes into flight control – safety checks, process checks, maintenance, things like that.

Audrow Nash: Swapping the batteries, is that automated?

Dick Zhang: Yes, that is automated. On the docking station, there’s a carrousel with six batteries that are always charged. It inserts another battery when the drone needs to fly.

To tell you a story, what inspired all of this was a talk with one of our clients. They run 5,000 job sites around the world. They bought an Internet tool but had no idea how they were going to train 1,500 people to use it. So I automated a system and made it self-operating and self-sufficient. We were able to employ 1,500 really rapidly across the country and across the world.

Audrow Nash: Is it totally self-sustaining?

Dick Zhang: Yes, right now all you need is a standard power plug. Future generations will evolve generators, solar panels and wind fans. But, apart from the source of power, everything happens on its own. It flies itself, captures data and swaps batteries.

Audrow Nash: What kind of navigation do you use on the drones?

Dick Zhang: Navigation in drone technology has become really sophisticated over the last couple of years. We have a combination of GPS and inertial sensors on board, so that, three or four hundred times every second, the drone is saying to itself: “I need to be in position XYZ. Right now I’m not there, but I’m close. Let me get there.” So, four hundred times every second, it’s making these corrections and adjustments so that it can fly along the path that the computer programmed.

Audrow Nash: Other than the IMU and the GPS, what other sensors do you use? Do you have a vision system for landing on the platform that swaps the batteries, and some other sensors?

Dick Zhang: Yes. Now we are working with cameras. We’ve done some early tests with our partners on infrared and gas effect. We do a lot of work in energy. As part of this start up growth, we’re going through a crafty process of crawl, walk and run. It’s the vision that enables the most capabilities, to help our clients right now, with the resources that we have. Now, to touch on how we land, that’s a huge problem. So, we built a proprietary vision system based on ultrasound. There’s a beacon and a dock and a transceiver, where you listen and triangulate.

Audrow Nash: Okay, so you use GPS to get it close enough…

Dick Zhang: Exactly.

Audrow Nash: Then the ultrasound takes over by localizing the sound somehow?

Dick Zhang: Yes, exactly.

Audrow Nash: Okay, thank you.

Dick Zhang: That was awesome. Thank you.

All audio interviews are transcribed and edited for clarity with great care, however, we cannot assume responsibility for their accuracy.

Every April, National Robotics Week fuels a heightened awareness around robotics, its impact on society, and its growing importance in a wide variety of fields and applications. Robotics, however, never seems to achieve its hyped potential from its beginnings in industrial applications, when the benefits of fast, precise, repetitive manipulation in manufacturing were a significant driver for adoption of early robotics solutions. While robot arms in manufacturing debuted the benefits of robotics technology to industry, the robot arms were put in cages and they largely stayed in those environments.

Today, in contrast, while robot arm sales for manufacturing and other industrial uses are at record highs, new robotic solutions outside of manufacturing are largely dominated by mobile robots: drone planes both large and small, UAV’s, telepresence robots, delivery robots (think Aethon and Savioke), and, of course, mobile vacuum cleaners—today’s all-time robot sales leader.

Manipulation is still a core part of manufacturing, and several employment and demographic issues are developing on a global scale—including the escalation of labor rates, double digit chronic turnover, generational culture changes, and a smaller eligible workforce due to societal aging demographics—that are driving the need for new innovations for manufacturing companies, which I think will have interesting implications for robotics. These innovations are focused on three main areas: low cost, end-effector capability, and dexterity and agile assembly.

Low Cost

The cost of robot arms has dropped significantly over the last decade, as much as 50 percent or more for some applications. Over the last few weeks, two new industrial robot arms were introduced that represent the current bar in the lower cost robot arms race: the UR3 from Universal Robotics at US$23,000, and Sawyer from Rethink Robotics at US$29,000. The story around lower cost robot arms for manufacturing, however, is not as straightforward as lower price equals more uses. Focusing just on price distracts from the practical realities of manufacturing robots. Yes, for every drop in price, there will be some cost-benefit analysis that will justify more use of robots, but this is marginally incremental. Our interaction with manufacturers confirms that the cost of the robot for most current uses is not a limiting factor. The bigger considerations are project and integration costs related to system design, programming, and installation, which account for as much as 80 percent or more of the total cost to deploy.

So when I say low cost, I am talking about a new target—industrial grade robot arms at US$5,000 or less. For manufacturers, this is the price point that would open up a significant broad new set of applications. Our team at SRI has been developing low cost dexterous hands over the last few years. We understand in painstaking detail how quickly the component costs can add up for any dexterous robot. We’ve demonstrated a 12 degree-of-freedom dexterous hand with a bill-of-materials around $500 at volume, and now have our sights on developing an arm that could come in at the $5,000 target price point. However, getting to that solution is hard, given the large parts list and the performance demands on transmissions and motors.

End-effectors

The functionality of a robot arm is determined by its end-effector. An interesting aspect of commercial robot arm companies is that, given this fact, very few of them actually offer their own end-effector products. This makes sense when you consider the wide range of end-effectors that are necessary to handle all of the possible use cases for arms.

In order to extend the capabilities of the current robot arms, there has been a recent surge in new companies developing more capable end-effectors. SRI is contributing to this surge through its spinout, Grabit.  Grabit uses electroadhesion as a novel approach to grasping items—including difficult items such as potato chip bags, textiles, and fresh fruit and vegetables—enabling new capabilities in a cost-effective way while requiring very little power. The thin form factor of Grabit grippers can also increase storage efficiency by stacking items more closely together, and now with EA integrated into conveyor systems, packages can be transported between levels in a mechanically simpler way. Other companies include Robotiq, which has been extending its line of grippers for industrial applications over the last few years, as well as Soft Robotics, Inc., and Empire Robotics, both emerging from different bases of work on soft robotic from Harvard and iRobot.

We have yet to see a fully dexterous robot hand emerge at a price point that can serve a mass market, but I believe this is within reach. We should expect to see more innovation around robot end-effectors.

Dexterity and Agile Assembly

Manufacturers are targeting a next set of robot capabilities related to dexterous assembly and other tasks that are now only possible with the touch of the human hand. A good example of this is the each-pick task, i.e., the ability to reach into a bin of unstructured parts, identify one item, grasp one item, remove one item, and then reposition it for assembly into the part. While today there is no general purpose solution to this problem, there is considerable R&D in universities, startup companies, and in industry to come up with this solution.

For each pick, so far, limited success has been achieved from approaches that use multiple robots to perform sorting in stages that ultimately present parts in a uniform way to the assembly station. It is likely that the eventual solution for each-pick will somehow combine physical dexterity and sensing from new end-effectors with perception and planning.

The Near Future

I’ve described the current R&D target for robotics being driven by the manufacturing market: low-cost, high-performance robots, with end-effectors capable of dexterous manipulation and agile assembly. Considerable investment is being made to create robust solutions, justified by real business drivers affecting the manufacturing industry, which I think will be available within five years or so.

This is opportune timing, based on another big trend I see emerging. As an independent R&D organization, the SRI Robotics Program is working at the leading edge of robotics across emerging service robotics markets like agriculture, elder care, and consumer home products. Whereas robot arm solutions in manufacturing 40 years ago were largely caged inside that industry, this new set of solutions targeted by the manufacturing industry will provide a set of tools that will accelerate deployment of robot arm solutions for these service robot applications and more. For instance, the agricultural industry can look to robotics solutions to help alleviate farmers’ challenges in finding the labor they’ve traditionally used to harvest their crops. There are potential applications in the service industry for performing manual tasks in restaurants such as cleaning and even food preparation. And in the healthcare field, the emergence of low cost robotics can also bring about solutions for therapy, rehabilitation, and personal care applications. It will be exciting to see if it plays out this way.

At SRI, we’re looking forward to enabling a new generation of low cost dexterous manipulation solutions that will drive adoption beyond mobility and beyond manufacturing. Also, this year, we’re once again partnering with Xconomy to host Robo Madness West. Join us along with some of the nation’s leaders in robotics technology and business to take a deep look at the real progress that robots are making today.

This post originally appeared on SRI International. 

Watch Rolf Pfeifer’s farewell lecture at the University of Zurich, broadcasting live on Robohub Friday May 23, 2014 (18:00–19:30 CEST/16:00–17:30 UTC).

One of the most prominent figures in the “embodied intelligence” approach to AI, and the intellectual father of the Roboy humanoid, Rolf Pfeifer and his Artificial Intelligence Laboratory (AI Lab) at the University of Zurich have been a force of influence on the fields of robotics and AI for almost 30 years. He is the co-author of the seminal AI text How the Body Shapes the Way We Think, and also initiated the ShanghAI Lectures, a lecture series that brings together people from various cultural and linguistic backgrounds who share the common vision of building robots in order to better understand intelligence.

On the occasion of his retirement from the University of Zurich, Rolf will deliver a farewell lecture in which he reviews his research from the past three decades and discusses its implications for the future with other renowned researchers in AI who have influenced his work. Roboy, the AI Lab’s tendon-driven humanoid robot, will join the event and welcome the audience.

After leaving the University of Zurich, Pfeifer moves on to other projects. He has plans to pursue technology transfer in Switzerland, Japan, and China, in particular a robot lounge in Shanghai where robots (among others) care for the well-being of the guests. He will also be a visiting professor in Japan (Osaka, Chiba) and China (Shanghai). His main goals will be to bring the results from basic research into practice, and to communicate them to a broad audience.

Rolf Pfeifer’s farewell lecture took place on Friday May 23, 18:00–19:30 CEST at the University of Zurich main building.

The talk begins after the opening remarks, at the 05:20 mark.

The talk begins after the opening remarks, at the 05:20 mark.

Tweets about “#RolfGoodbAI”

To manipulate objects, robots are often required to estimate their position and orientation in space. The robot will behave differently if it’s grasping a glass that is standing up, or one that has been tipped over. On the other hand, it shouldn’t make a difference if the robot is gripping two different glasses with similar poses. The challenge is to have robots learn how to grasp new objects, based on previous experience.

To this end, the latest paper in Autonomous Robots proposes a Sparse Pose Manifolds (SPM) method. As shown in the figure above, different objects viewed from the same perspective should share identical poses. All the objects facing right are in the same “pose-bucket”, which is different from the bucket for objects facing left, or forward. For each pose, the robot knows how to behave to guide the gripper to grasp the object. To grip an unknown object, the robot estimates what “bucket” the object falls into.

The videos below shows how this method can efficiently guide a robotic gripper to grasp an unknown object and illustrates the performance of the pose estimation module.

… Grabit, a 2012 SRI International spin-off, secures $3 million in Series A funding from ABB Technology Ventures, Nike and Formation 8, a tech VC in Silicon Valley.

… Grabit’s products include grippers and fixtures, case and box handling grippers, and smart conveyors for material handling.

… At the core is Grabit’s patented electroadhesion-based material-handing technique – a grasping method that can successfully handle a wide variety of objects, shapes, sizes and materials.

Small and medium shops and factories (SMEs) are an untapped marketplace for robotics but direly in need of automation to remain competitive in this global economy. Two new start up companies: Rethink Robotics and Universal Robots have entered that marketplace. Both companies have U.S. sales in the hundreds of units; Universal has a head start internationally and has sold about 3,000 to-date, but Rethink is way ahead in the US. Both have similar 60-100/mo manufacturing run rates – so the future looks bright for selling flexible, lightweight, low-cost robots that are easily programmed, safe for humans to work alongside, don’t require a caged or roped off area, and perform at affordable metrics.

The importance of this new robotic capability and its effect on jobs was the subject of a CBS News piece on October 5. A few of the quotes from that newscast:

• U.S. productivity and efficiency are three times greater than China.
• Baxter’s effective cost is $3 per hour.
• 3-5 million new jobs will happen this decade due to the deployment of Baxter and other co-robots.

Boston-based Rethink Robotics – the maker of the low-cost two-armed Baxter robot shown above on the left – has already begun to sell to small and medium-sized U.S. businesses (SMEs) as well as schools and colleges. They recently announced beginning to sell to European academic institutions.

Danish-based Universal Robots has started to benefit from sales from their new distributor network in the U.S. supplementing their growing European, Asian and International distributor networks and robot sales.

Each company has provided a video profiling how their client companies are using and benefitting from the use of their respective robots. Although these videos are public relations puff pieces, they are also an information resource describing the needs smaller companies have and the benefits offered by robotics.

UNIVERSAL ROBOTS

Of the two companies, for the present time, Universal Robots UR robots are better able to handle industrial tasks than Baxter robots. Universal’s UR5 and UR10 robots can handle 5 and 10 kilos respectively and have sufficient torque to pull open doors and operate hand tools. They also run faster.

Universal Robots got lots of publicity last month when it became known that a UR robot was working at a VW diesel engine production plant in Germany. It was the first publicly-known instance of a robot working alongside a human in an auto factory. In that example, the robot carefully picks up delicate glow plugs and places them into hard-to-reach drill holes alongside a human who then insulates the cylinder head. The robot acts as an assistant and enables the worker to carry out their activities in an upright, healthy posture, unlike previous methods. This week MIT Technology Review showed another Universal robot, this time at a BMW factory in South Carolina, helping workers perform door assembly by doing the strenuous task of applying door sealant thereby freeing human workers to do the less physical tasks.

In the following video, a 72-person short-run metal products maker displays two applications for their new UR robot: CNC machine tending and loading and unloading a metal bender.

[In a recent case study, machine-loaded costs for a part came to $0.97/part in the U.S. and $0.76/part in China. When a robot was used for machine tending and loading, the total cost shifted to $0.73/part in the U.S. versus $0.76/part in China (including offshore costs like duties and transportation fees). In the study, using the robot boosted efficiency. The machine utilization under manual loading was 82%, when you account for shift changes and breaks. When robots did the tending and loading, the efficiency figure approached 99%. Source: PlasticsToday and Fanuc Robotics.]

RETHINK ROBOTICS

This robot is being developed to automate the management of laboratory animal colonies used by pharmaceutical companies and research institutions, primarily those that raise from 10,000 to 30,000 mice or rats. It’s currently under development by Nikkyo Technos and Yaskawa Electric.

“The biggest problem in raising animals is that diseases can spread from people to the animals. If that happens, all the animals have to be killed and replaced with new ones. So, infection by people must be prevented. By managing animals using robots in an enclosed space, it’s basically possible to eliminate the spread of diseases from animals to people or from people to animals.”

This six-axis, vertical, multi-jointed robot can mimic the motions of a human. It can change cage sheets, top up the food, and change the water. Taking out cages, changing sheets, and topping up food are each done with separate tools, which the robot picks up in turn. In this model, the amount of food remaining isn’t taken into account. But in the next model, a camera will be used to see how much food is left, so the robot can add the right amount.

“Mice, especially, are nervous animals, so the robot handles the cages gently. These tasks account for about 80% of the work involved with lab animals. So, our aim is to automate the hard, dirty, and dangerous task of dealing with so much dust and droppings.”

This robot can also work in coordination with a robot that carries cages from the rack to the workbench, and a monitoring system for the animal facility. In this way, it can automate all aspects of animal raising, from surveillance to care.

“The animals can be monitored with cameras 24/7. So, people can check their own cages from the monitoring station. The animals’ body temperature can also be managed. It takes about two hours for the cages to come back from the lab, but data can be viewed directly from a PC in the monitoring room. So, people can see the cages they want right away, wherever they are.”

“Doing this work with robots makes it much faster, so lots of cages can be handled in a short time. Also, using cameras to monitor food and water is safer and more reliable than having people do it. We’d like to complete the system this year, and next year, we’d like to produce several sets, so we can make at least a provisional start.”

The Mitsubishi Research Institute, Inc. (MRI) invites the robotics community to upgrade and to develop a technical catalog of robotic technologies to move and investigate inside nuclear reactor buildings, namely 1) a flight technology to enable access to the top floor of the nuclear reactor building and 2) a technology to move around and conduct investigations under water in a flooded reactor building.

MRI is entrusted by the Agency for Natural Resources and Energy to develop machines and equipment for decommissioning the TEPCO Fukushima Dai-ichi nuclear power plant, which was damaged during the East Japan Earthquake on March 11, 2011.

On December 21, 2011, the Government-TEPCO Council adopted a “Mid-to-Long-Term Roadmap towards the Decommissioning of Fukushima Dai-ichi Nuclear Power Units 1-4”. The roadmap presents a three phase approach to decommission the plant over 40 years.

Japanese decommissioning robots

Sakura

Quince

Quince is a rescue robot designed for CBRNE disasters – situations involving Chemical, Biological, Radiological, Nuclear, and Explosive hazards. It was developed to perform on-site surveys on behalf of humans, such as fire-fighters. Hypothetical disasters include accidents at plants handling toxic substances, leaks of hazardous chemicals from chemical plants, explosions, and acts of terror like the notorious subway sarin incident. Risks are particularly high in enclosed spaces (underground and inside buildings), and expectations are high for robots as they will protect officers from secondary disasters.

Quince surveyed the inside of the Fukushima Dai-ichi nuclear power plant. With some modifications, it was able to photograph the details of the buildings, create radiation dose maps, and sample radioactive materials floating in the air. It has been helping to reduce the radiation dose experienced by on-site workers and to shorten their work period.

Toshiba tetrapod

In November 2012, Toshiba Corporation unveiled a new tetrapod robot that can carry out investigative and recovery work in locations that are too risky for people to enter, such as the Fukushima power plant.

MEISTeR

In December 2012, Mitsubishi Heavy Industries, Ltd. (MHI) unveiled a prototype of the “MEISTeR (Maintenance Equipment Integrated System of Telecontrol Robot),” a two-armed robot to assist recovery work after disasters or severe accidents by performing light-duty tasks in areas inaccessible to humans. By changing its arms’ attachment tools, the robot can perform various tasks such as carrying objects, drilling and opening/closing valves. Going forward, MHI will continue to improve and explore possible applications in crisis management.

Key dates:
Application deadline :15 March
Development phase : 4–12 April
Finalist down-selection phase : Beginning May
Competition event : Mid-October

As mentioned on the ESA website:

Robots will receive their instructions wirelessly. To make the competition more challenging, the robot operators will not have a direct view of the playing field but have to rely on video from four cameras in the playing area and from sensors on their robot, if they choose to equip their robots with them.

The playground is a scaled-down model of the International Space Station, starting at ESA’s Automated Transfer Vehicle (ATV) and finishing at the European laboratory, Columbus.

In under five minutes, the robots must unload cargo of different sizes from ATV, carry it to Columbus and stow it safely. Like in any game, there will be bonus points  and penalties.

The route to Columbus includes scaling a ramp and all robots must end the game in one piece to qualify. ‘Astronauts’ floating around the game area must be avoided and teams must cope with losing video signal for up to 15 seconds.

This is a sketch of the competition area layout:

The competition rules are available on pdf (Volare space robotics competition details – PDF ) and you can see a presentation of the concept by ESA astronaut Luca Parmitano:

Volare robotics challenge link – esa.int