The Jungle Hawk Owl team. Photo: Sally Chapman/MIT

In the event of a natural disaster that disrupts phone and Internet systems over a wide area, autonomous aircraft could potentially hover over affected regions, carrying communications payloads that provide temporary telecommunications coverage to those in need.

However, such unpiloted aerial vehicles, or UAVs, are often expensive to operate, and can only remain in the air for a day or two, as is the case with most autonomous surveillance aircraft operated by the U.S. Air Force. Providing adequate and persistent coverage would require a relay of multiple aircraft, landing and refueling around the clock, with operational costs of thousands of dollars per hour, per vehicle.

Now a team of MIT engineers has come up with a much less expensive UAV design that can hover for longer durations to provide wide-ranging communications support. The researchers designed, built, and tested a UAV resembling a thin glider with a 24-foot wingspan. The vehicle can carry 10 to 20 pounds of communications equipment while flying at an altitude of 15,000 feet. Weighing in at just under 150 pounds, the vehicle is powered by a 5-horsepower gasoline engine and can keep itself aloft for more than five days — longer than any gasoline-powered autonomous aircraft has remained in flight, the researchers say.

The team is presenting its results this week at the American Institute of Aeronautics and Astronautics Conference in Denver, Colorado. The team was led by R. John Hansman, the T. Wilson Professor of Aeronautics and Astronautics; and Warren Hoburg, the Boeing Assistant Professor of Aeronautics and Astronautics. Hansman and Hoburg are co-instructors for MIT’s Beaver Works project, a student research collaboration between MIT and the MIT Lincoln Laboratory.

A solar no-go

Hansman and Hoburg worked with MIT students to design a long-duration UAV as part of a Beaver Works capstone project — typically a two- or three-semester course that allows MIT students to design a vehicle that meets certain mission specifications, and to build and test their design.

In the spring of 2016, the U.S. Air Force approached the Beaver Works collaboration with an idea for designing a long-duration UAV powered by solar energy. The thought at the time was that an aircraft, fueled by the sun, could potentially remain in flight indefinitely. Others, including Google, have experimented with this concept,  designing solar-powered, high-altitude aircraft to deliver continuous internet access to rural and remote parts of Africa.

But when the team looked into the idea and analyzed the problem from multiple engineering angles, they found that solar power — at least for long-duration emergency response — was not the way to go.

“[A solar vehicle] would work fine in the summer season, but in winter, particularly if you’re far from the equator, nights are longer, and there’s not as much sunlight  during the day. So you have to carry more batteries, which adds weight and makes the plane bigger,” Hansman says. “For the mission of disaster relief, this could only respond to disasters that occur in summer, at low latitude. That just doesn’t work.”

The researchers came to their conclusions after modeling the problem using GPkit, a software tool developed by Hoburg that allows engineers to determine the optimal design decisions or dimensions for a vehicle, given certain constraints or mission requirements.

This method is not unique among initial aircraft design tools, but unlike these tools, which take into account only several main constraints, Hoburg’s method allowed the team to consider around 200 constraints and physical models simultaneously, and to fit them all together to create an optimal aircraft design.

“This gives you all the information you need to draw up the airplane,” Hansman says. “It also says that for every one of these hundreds of parameters, if you changed one of them, how much would that influence the plane’s performance? If you change the engine a bit, it will make a big difference. And if you change wingspan, will it show an effect?”

Framing for takeoff

After determining, through their software estimations, that a solar-powered UAV would not be feasible, at least for long-duration use in any part of the world, the team performed the same modeling for a gasoline-powered aircraft. They came up with a design that was predicted to stay in flight for more than five days, at altitudes of 15,000 feet, in up to 94th-percentile winds, at any latitude.

In the fall of 2016, the team built a prototype UAV, following the dimensions determined by students using Hoburg’s software tool. To keep the vehicle lightweight, they used materials such as carbon fiber for its wings and fuselage, and Kevlar for the tail and nosecone, which houses the payload. The researchers designed the UAV to be easily taken apart and stored in a FedEx box, to be shipped to any disaster region and quickly reassembled.

This spring, the students refined the prototype and developed a launch system, fashioning a simple metal frame to fit on a typical car roof rack. The UAV sits atop the frame as a driver accelerates the launch vehicle (a car or truck) up to rotation speed — the UAV’s optimal takeoff speed. At that point, the remote pilot would angle the UAV toward the sky, automatically releasing a fastener and allowing the UAV to lift off.

In early May, the team put the UAV to the test, conducting flight tests at Plum Island Airport in Newburyport, Massachusetts. For initial flight testing, the students modified the vehicle to comply with FAA regulations for small unpiloted aircraft, which allow drones flying at low altitude and weighing less than 55 pounds. To reduce the UAV’s weight from 150 to under 55 pounds, the researchers simply loaded it with a smaller ballast payload and less gasoline.

In their initial tests, the UAV successfully took off, flew around, and landed safely. Hoburg says there are special considerations that have to be made to test the vehicle over multiple days, such as having enough people to monitor the aircraft over a long period of time.

“There are a few aspects to flying for five straight days,” Hoburg says. “But we’re pretty confident that we have the right fuel burn rate and right engine that we could fly it for five days.”

“These vehicles could be used not only for disaster relief but also other missions, such as environmental monitoring. You might want to keep watch on wildfires or the outflow of a river,” Hansman adds. “I think it’s pretty clear that someone within a few years will manufacture a vehicle that will be a knockoff of this.”

This research was supported, in part, by MIT Lincoln Laboratory.

A GelSight sensor attached to a robot’s gripper enables the robot to determine precisely where it has grasped a small screwdriver, removing it from and inserting it back into a slot, even when the gripper screens the screwdriver from the robot’s camera. Photo: Robot Locomotion Group at MIT

Eight years ago, Ted Adelson’s research group at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) unveiled a new sensor technology, called GelSight, that uses physical contact with an object to provide a remarkably detailed 3-D map of its surface.

Now, by mounting GelSight sensors on the grippers of robotic arms, two MIT teams have given robots greater sensitivity and dexterity. The researchers presented their work in two papers at the International Conference on Robotics and Automation last week.

In one paper, Adelson’s group uses the data from the GelSight sensor to enable a robot to judge the hardness of surfaces it touches — a crucial ability if household robots are to handle everyday objects.

In the other, Russ Tedrake’s Robot Locomotion Group at CSAIL uses GelSight sensors to enable a robot to manipulate smaller objects than was previously possible.

The GelSight sensor is, in some ways, a low-tech solution to a difficult problem. It consists of a block of transparent rubber — the “gel” of its name — one face of which is coated with metallic paint. When the paint-coated face is pressed against an object, it conforms to the object’s shape.

The metallic paint makes the object’s surface reflective, so its geometry becomes much easier for computer vision algorithms to infer. Mounted on the sensor opposite the paint-coated face of the rubber block are three colored lights and a single camera.

“[The system] has colored lights at different angles, and then it has this reflective material, and by looking at the colors, the computer … can figure out the 3-D shape of what that thing is,” explains Adelson, the John and Dorothy Wilson Professor of Vision Science in the Department of Brain and Cognitive Sciences.

In both sets of experiments, a GelSight sensor was mounted on one side of a robotic gripper, a device somewhat like the head of a pincer, but with flat gripping surfaces rather than pointed tips.

Contact points

For an autonomous robot, gauging objects’ softness or hardness is essential to deciding not only where and how hard to grasp them but how they will behave when moved, stacked, or laid on different surfaces. Tactile sensing could also aid robots in distinguishing objects that look similar.

In previous work, robots have attempted to assess objects’ hardness by laying them on a flat surface and gently poking them to see how much they give. But this is not the chief way in which humans gauge hardness. Rather, our judgments seem to be based on the degree to which the contact area between the object and our fingers changes as we press on it. Softer objects tend to flatten more, increasing the contact area.

The MIT researchers adopted the same approach. Wenzhen Yuan, a graduate student in mechanical engineering and first author on the paper from Adelson’s group, used confectionary molds to create 400 groups of silicone objects, with 16 objects per group. In each group, the objects had the same shapes but different degrees of hardness, which Yuan measured using a standard industrial scale.

Then she pressed a GelSight sensor against each object manually and recorded how the contact pattern changed over time, essentially producing a short movie for each object. To both standardize the data format and keep the size of the data manageable, she extracted five frames from each movie, evenly spaced in time, which described the deformation of the object that was pressed.

Finally, she fed the data to a neural network, which automatically looked for correlations between changes in contact patterns and hardness measurements. The resulting system takes frames of video as inputs and produces hardness scores with very high accuracy. Yuan also conducted a series of informal experiments in which human subjects palpated fruits and vegetables and ranked them according to hardness. In every instance, the GelSight-equipped robot arrived at the same rankings.

Yuan is joined on the paper by her two thesis advisors, Adelson and Mandayam Srinivasan, a senior research scientist in the Department of Mechanical Engineering; Chenzhuo Zhu, an undergraduate from Tsinghua University who visited Adelson’s group last summer; and Andrew Owens, who did his PhD in electrical engineering and computer science at MIT and is now a postdoc at the University of California at Berkeley.

Obstructed views

The paper from the Robot Locomotion Group was born of the group’s experience with the Defense Advanced Research Projects Agency’s Robotics Challenge (DRC), in which academic and industry teams competed to develop control systems that would guide a humanoid robot through a series of tasks related to a hypothetical emergency.

Typically, an autonomous robot will use some kind of computer vision system to guide its manipulation of objects in its environment. Such systems can provide very reliable information about an object’s location — until the robot picks the object up. Especially if the object is small, much of it will be occluded by the robot’s gripper, making location estimation much harder. Thus, at exactly the point at which the robot needs to know the object’s location precisely, its estimate becomes unreliable. This was the problem the MIT team faced during the DRC, when their robot had to pick up and turn on a power drill.

“You can see in our video for the DRC that we spend two or three minutes turning on the drill,” says Greg Izatt, a graduate student in electrical engineering and computer science and first author on the new paper. “It would be so much nicer if we had a live-updating, accurate estimate of where that drill was and where our hands were relative to it.”

That’s why the Robot Locomotion Group turned to GelSight. Izatt and his co-authors — Tedrake, the Toyota Professor of Electrical Engineering and Computer Science, Aeronautics and Astronautics, and Mechanical Engineering; Adelson; and Geronimo Mirano, another graduate student in Tedrake’s group — designed control algorithms that use a computer vision system to guide the robot’s gripper toward a tool and then turn location estimation over to a GelSight sensor once the robot has the tool in hand.

In general, the challenge with such an approach is reconciling the data produced by a vision system with data produced by a tactile sensor. But GelSight is itself camera-based, so its data output is much easier to integrate with visual data than the data from other tactile sensors.

In Izatt’s experiments, a robot with a GelSight-equipped gripper had to grasp a small screwdriver, remove it from a holster, and return it. Of course, the data from the GelSight sensor don’t describe the whole screwdriver, just a small patch of it. But Izatt found that, as long as the vision system’s estimate of the screwdriver’s initial position was accurate to within a few centimeters, his algorithms could deduce which part of the screwdriver the GelSight sensor was touching and thus determine the screwdriver’s position in the robot’s hand.

“I think that the GelSight technology, as well as other high-bandwidth tactile sensors, will make a big impact in robotics,” says Sergey Levine, an assistant professor of electrical engineering and computer science at the University of California at Berkeley. “For humans, our sense of touch is one of the key enabling factors for our amazing manual dexterity. Current robots lack this type of dexterity and are limited in their ability to react to surface features when manipulating objects. If you imagine fumbling for a light switch in the dark, extracting an object from your pocket, or any of the other numerous things that you can do without even thinking — these all rely on touch sensing.”

“Software is finally catching up with the capabilities of our sensors,” Levine adds. “Machine learning algorithms inspired by innovations in deep learning and computer vision can process the rich sensory data from sensors such as the GelSight to deduce object properties. In the future, we will see these kinds of learning methods incorporated into end-to-end trained manipulation skills, which will make our robots more dexterous and capable, and maybe help us understand something about our own sense of touch and motor control.”

New algorithms power a prototype system for helping visually impaired users avoid obstacles and identify objects. Courtesy of the researchers.

Computer scientists have been working for decades on automatic navigation systems to aid the visually impaired, but it’s been difficult to come up with anything as reliable and easy to use as the white cane, the type of metal-tipped cane that visually impaired people frequently use to identify clear walking paths.

White canes have a few drawbacks, however. One is that the obstacles they come in contact with are sometimes other people. Another is that they can’t identify certain types of objects, such as tables or chairs, or determine whether a chair is already occupied.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a new system that uses a 3-D camera, a belt with separately controllable vibrational motors distributed around it, and an electronically reconfigurable Braille interface to give visually impaired users more information about their environments.

The system could be used in conjunction with or as an alternative to a cane. In a paper they’re presenting this week at the International Conference on Robotics and Automation, the researchers describe the system and a series of usability studies they conducted with visually impaired volunteers.

“We did a couple of different tests with blind users,” says Robert Katzschmann, a graduate student in mechanical engineering at MIT and one of the paper’s two first authors. “Having something that didn’t infringe on their other senses was important. So we didn’t want to have audio; we didn’t want to have something around the head, vibrations on the neck — all of those things, we tried them out, but none of them were accepted. We found that the one area of the body that is the least used for other senses is around your abdomen.”

Katzschmann is joined on the paper by his advisor Daniela Rus, an Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science; his fellow first author Hsueh-Cheng Wang, who was a postdoc at MIT when the work was done and is now an assistant professor of electrical and computer engineering at National Chiao Tung University in Taiwan; Santani Teng, a postdoc in CSAIL; Brandon Araki, a graduate student in mechanical engineering; and Laura Giarré, a professor of electrical engineering at the University of Modena and Reggio Emilia in Italy.

Parsing the world

The researchers’ system consists of a 3-D camera worn in a pouch hung around the neck; a processing unit that runs the team’s proprietary algorithms; the sensor belt, which has five vibrating motors evenly spaced around its forward half; and the reconfigurable Braille interface, which is worn at the user’s side.

The key to the system is an algorithm for quickly identifying surfaces and their orientations from the 3-D-camera data. The researchers experimented with three different types of 3-D cameras, which used three different techniques to gauge depth but all produced relatively low-resolution images — 640 pixels by 480 pixels — with both color and depth measurements for each pixel.

The algorithm first groups the pixels into clusters of three. Because the pixels have associated location data, each cluster determines a plane. If the orientations of the planes defined by five nearby clusters are within 10 degrees of each other, the system concludes that it has found a surface. It doesn’t need to determine the extent of the surface or what type of object it’s the surface of; it simply registers an obstacle at that location and begins to buzz the associated motor if the wearer gets within 2 meters of it.

Chair identification is similar but a little more stringent. The system needs to complete three distinct surface identifications, in the same general area, rather than just one; this ensures that the chair is unoccupied. The surfaces need to be roughly parallel to the ground, and they have to fall within a prescribed range of heights.

Tactile data

The belt motors can vary the frequency, intensity, and duration of their vibrations, as well as the intervals between them, to send different types of tactile signals to the user. For instance, an increase in frequency and intensity generally indicates that the wearer is approaching an obstacle in the direction indicated by that particular motor. But when the system is in chair-finding mode, for example, a double pulse indicates the direction in which a chair with a vacant seat can be found.

The Braille interface consists of two rows of five reconfigurable Braille pads. Symbols displayed on the pads describe the objects in the user’s environment — for instance, a “t” for table or a “c” for chair. The symbol’s position in the row indicates the direction in which it can be found; the column it appears in indicates its distance. A user adept at Braille should find that the signals from the Braille interface and the belt-mounted motors coincide.

In tests, the chair-finding system reduced subjects’ contacts with objects other than the chairs they sought by 80 percent, and the navigation system reduced the number of cane collisions with people loitering around a hallway by 86 percent.

Engineers at MIT have come up with an algorithm to tune a Dynamic Vision Sensor (DVS) camera, simplifying a scene to its most essential visual elements and potentially enabling the development of faster drones. Image: Jose-Luis Olivares/MIT

There’s a limit to how fast autonomous vehicles can fly while safely avoiding obstacles. That’s because the cameras used on today’s drones can only process images so fast, frame by individual frame. Beyond roughly 30 miles per hour, a drone is likely to crash simply because its cameras can’t keep up.

Recently, researchers in Zurich invented a new type of camera, known as the Dynamic Vision Sensor (DVS), that continuously visualizes a scene in terms of changes in brightness, at extremely short, microsecond intervals. But this deluge of data can overwhelm a system, making it difficult for a drone to distinguish an oncoming obstacle through the noise.

Now engineers at MIT have come up with an algorithm to tune a DVS camera to detect only specific changes in brightness that matter for a particular system, vastly simplifying a scene to its most essential visual elements.

The results, which they presented at the IEEE American Control Conference in Seattle, can be applied to any linear system that directs a robot to move from point A to point B as a response to high-speed visual data. Eventually, the results could also help to increase the speeds for more complex systems such as drones and other autonomous robots.

“There is a new family of vision sensors that has the capacity to bring high-speed autonomous flight to reality, but researchers have not developed algorithms that are suitable to process the output data,” says lead author Prince Singh, a graduate student in MIT’s Department of Aeronautics and Astronautics. “We present a first approach for making sense of the DVS’ ambiguous data, by reformulating the inherently noisy system into an amenable form.”

Singh’s co-authors are MIT visiting professor Emilio Frazzoli of the Swiss Federal Institute of Technology in Zurich, and Sze Zheng Yong of Arizona State University.

Taking a visual cue from biology

The DVS camera is the first commercially available “neuromorphic” sensor — a class of sensors that is modeled after the vision systems in animals and humans. In the very early stages of processing a scene, photosensitive cells in the human retina, for example, are activated in response to changes in luminosity, in real time.

Neuromorphic sensors are designed with multiple circuits arranged in parallel, similarly to photosensitive cells, that activate and produce blue or red pixels on a computer screen in response to either a drop or spike in brightness.

Instead of a typical video feed, a drone with a DVS camera would “see” a grainy depiction of pixels that switch between two colors, depending on whether that point in space has brightened or darkened at any given moment. The sensor requires no image processing and is designed to enable, among other applications, high-speed autonomous flight.

Researchers have used DVS cameras to enable simple linear systems to see and react to high-speed events, and they have designed controllers, or algorithms, to quickly translate DVS data and carry out appropriate responses. For example, engineers have designed controllers that interpret pixel changes in order to control the movements of a robotic goalie to block an incoming soccer ball, as well as to direct a motorized platform to keep a pencil standing upright.

But for any given DVS system, researchers have had to start from scratch in designing a controller to translate DVS data in a meaningful way for that particular system.

“The pencil and goalie examples are very geometrically constrained, meaning if you give me those specific scenarios, I can design a controller,” Singh says. “But the question becomes, what if I want to do something more complicated?”

Cutting through the noise

In the team’s new paper, the researchers report developing a sort of universal controller that can translate DVS data in a meaningful way for any simple linear, robotic system. The key to the controller is that it identifies the ideal value for a parameter Singh calls “H,” or the event-threshold value, signifying the minimum change in brightness that the system can detect.

Setting the H value for a particular system can essentially determine that system’s visual sensitivity: A system with a low H value would be programmed to take in and interpret changes in luminosity that range from very small to relatively large, while a high H value would exclude small changes, and only “see” and react to large variations in brightness.

The researchers formulated an algorithm first by taking into account the possibility that a change in brightness would occur for every “event,” or pixel activated in a particular system. They also estimated the probability for “spurious events,” such as a pixel randomly misfiring, creating false noise in the data.

Once they derived a formula with these variables in mind, they were able to work it into a well-known algorithm known as an H-infinity robust controller, to determine the H value for that system.

The team’s algorithm can now be used to set a DVS camera’s sensitivity to detect the most essential changes in brightness for any given linear system, while excluding extraneous signals. The researchers performed a numerical simulation to test the algorithm, identifying an H value for a theoretical linear system, which they found was able to remain stable and carry out its function without being disrupted by extraneous pixel events.

“We found that this H threshold serves as a ‘sweet-spot,’ so that a system doesn’t become overwhelmed with too many events,” Singh says. He adds that the new results “unify control of many systems,” and represent a first step toward faster, more stable autonomous flying robots, such as the Robobee, developed by researchers at Harvard University.

“We want to break that speed limit of 20 to 30 miles per hour, and go faster without colliding,” Singh says. “The next step may be to combine DVS with a regular camera, which can tell you, based on the DVS rendering, that an object is a couch versus a car, in real time.”

This research was supported in part by the Singapore National Research Foundation through the SMART Future Urban Mobility project.

“The new frontier is learning how to design the relationships between people, robots, and infrastructure,” says David Mindell, the Dibner Professor of the History of Engineering and Manufacturing, and a professor of aeronautics and astronautics. “We need new sensors, new software, new ways of architecting systems.” Photo: Len Rubenstein

Science and technology are essential tools for innovation, and to reap their full potential, we also need to articulate and solve the many aspects of today’s global issues that are rooted in the political, cultural, and economic realities of the human world. With that mission in mind, MIT’s School of Humanities, Arts, and Social Sciences has launched The Human Factor — an ongoing series of stories and interviews that highlight research on the human dimensions of global challenges. Contributors to this series also share ideas for cultivating the multidisciplinary collaborations needed to solve the major civilizational issues of our time.

David Mindell, the Frances and David Dibner Professor of the History of Engineering and Manufacturing and Professor of Aeronautics and Astronautics at MIT, researches the intersections of human behavior, technological innovation, and automation. Mindell is the author of five acclaimed books, most recently “Our Robots, Ourselves: Robotics and the Myths of Autonomy” (Viking, 2015). He is also the co-founder of Humatics Corporation, which develops technologies for human-centered automation. SHASS Communications recently asked him to share his thoughts on the relationship of robotics to human activities, and the role of multidisciplinary research in solving complex global issues.

Q: A major theme in recent political discourse has been the perceived impact of robots and automation on the United States labor economy. In your research into the relationship between human activity and robotics, what insights have you gained that inform the future of human jobs, and the direction of technological innovation?

A: In looking at how people have designed, used, and adopted robotics in extreme environments like the deep ocean, aviation, or space, my most recent work shows how robotics and automation carry with them human assumptions about how work gets done, and how technology alters those assumptions. For example, the U.S. Air Force’s Predator drones were originally envisioned as fully autonomous — able to fly without any human assistance. In the end, these drones require hundreds of people to operate.

The new success of robots will depend on how well they situate into human environments. As in chess, the strongest players are often the combinations of human and machine. I increasingly see that the three critical elements are people, robots, and infrastructure — all interdependent.

Q: In your recent book “Our Robots, Ourselves,” you describe the success of a human-centered robotics, and explain why it is the more promising research direction — rather than research that aims for total robotic autonomy. How is your perspective being received by robotic engineers and other technologists, and do you see examples of research projects that are aiming at human-centered robotics?

A: One still hears researchers describe full autonom as the only way to go; often they overlook the multitude of human intentions built into even the most autonomous systems, and the infrastructure that surrounds them. My work describes situated autonomy, where autonomous systems can be highly functional within human environments such as factories or cities. Autonomy as a means of moving through physical environments has made enormous strides in the past ten years. As a means of moving through human environments, we are only just beginning. The new frontier is learning how to design the relationships between people, robots, and infrastructure. We need new sensors, new software, new ways of architecting systems.

Q: What can the study of the history of technology teach us about the future of robotics?

A: The history of technology does not predict the future, but it does offer rich examples of how people build and interact with technology, and how it evolves over time. Some problems just keep coming up over and over again, in new forms in each generation. When the historian notices such patterns, he can begin to ask: Is there some fundamental phenomenon here? If it is fundamental, how is it likely to appear in the next generation? Might the dynamics be altered in unexpected ways by human or technical innovations?

One such pattern is how autonomous systems have been rendered less autonomous when they make their way into real world human environments. Like the Predator drone, future military robots will likely be linked to human commanders and analysts in some ways as well. Rather than eliding those links, designing them to be as robust and effective as possible is a worthy focus for researchers’ attention.

Q: MIT President L. Rafael Reif has said that the solutions to today’s challenges depend on marrying advanced technical and scientific capabilities with a deep understanding of the world’s political, cultural, and economic realities. What barriers do you see to multidisciplinary, sociotechnical collaborations, and how can we overcome them?

A: I fear that as our technical education and research continues to excel, we are building human perspectives into technologies in ways not visible to our students. All data, for example, is socially inflected, and we are building systems that learn from those data and act in the world. As a colleague from Stanford recently observed, go to Google image search and type in “Grandma” and you’ll see the social bias that can leak into data sets — the top results all appear white and middle class.

Now think of those data sets as bases of decision making for vehicles like cars or trucks, and we become aware of the social and political dimensions that we need to build into systems to serve human needs. For example, should driverless cars adjust their expectations for pedestrian behavior according to the neighborhoods they’re in?

Meanwhile, too much of the humanities has developed islands of specialized discourse that is inaccessible to outsiders. I used to be more optimistic about multidisciplinary collaborations to address these problems. Departments and schools are great for organizing undergraduate majors and graduate education, but the old two-cultures divides remain deeply embedded in the daily practices of how we do our work. I’ve long believed MIT needs a new school to address these synthetic, far-reaching questions and train students to think in entirely new ways.

Interview prepared by MIT SHASS Communications
Editorial team: Emily Hiestand (series editor), Daniel Evans Pritchard

The Nereis virens worm inspired new research out of the MIT Laboratory for Atomistic and Molecular Mechanics. Its jaw is made of soft organic material, but is as strong as harder materials such as human dentin. Photo: Alexander Semenov/Wikimedia Commons

A new material that naturally adapts to changing environments was inspired by the strength, stability, and mechanical performance of the jaw of a marine worm. The protein material, which was designed and modeled by researchers from the Laboratory for Atomistic and Molecular Mechanics (LAMM) in the Department of Civil and Environmental Engineering (CEE), and synthesized in collaboration with the Air Force Research Lab (AFRL) at Wright-Patterson Air Force Base, Ohio, expands and contracts based on changing pH levels and ion concentrations. It was developed by studying how the jaw of Nereis virens, a sand worm, forms and adapts in different environments.

The resulting pH- and ion-sensitive material is able to respond and react to its environment. Understanding this naturally-occurring process can be particularly helpful for active control of the motion or deformation of actuators for soft robotics and sensors without using external power supply or complex electronic controlling devices. It could also be used to build autonomous structures.

“The ability of dramatically altering the material properties, by changing its hierarchical structure starting at the chemical level, offers exciting new opportunities to tune the material, and to build upon the natural material design towards new engineering applications,” wrote Markus J. Buehler, the McAfee Professor of Engineering, head of CEE, and senior author of the paper.

The research, recently published in ACS Nano, shows that depending on the ions and pH levels in the environment, the protein material expands and contracts into different geometric patterns. When the conditions change again, the material reverts back to its original shape. This makes it particularly useful for smart composite materials with tunable mechanics and self-powered roboticists that use pH value and ion condition to change the material stiffness or generate functional deformations.

Finding inspiration in the strong, stable jaw of a marine worm

In order to create bio-inspired materials that can be used for soft robotics, sensors, and other uses — such as that inspired by the Nereis — engineers and scientists at LAMM and AFRL needed to first understand how these materials form in the Nereis worm, and how they ultimately behave in various environments. This understanding involved the development of a model that encompasses all different length scales from the atomic level, and is able to predict the material behavior. This model helps to fully understand the Nereis worm and its exceptional strength.

“Working with AFRL gave us the opportunity to pair our atomistic simulations with experiments,” said CEE research scientist Francisco Martin-Martinez. AFRL experimentally synthesized a hydrogel, a gel-like material made mostly of water, which is composed of recombinant Nvjp-1 protein responsible for the structural stability and impressive mechanical performance of the Nereis jaw. The hydrogel was used to test how the protein shrinks and changes behavior based on pH and ions in the environment.

The Nereis jaw is mostly made of organic matter, meaning it is a soft protein material with a consistency similar to gelatin. In spite of this, its strength, which has been reported to have a hardness ranging between 0.4 and 0.8 gigapascals (GPa), is similar to that of harder materials like human dentin. “It’s quite remarkable that this soft protein material, with a consistency akin to Jell-O, can be as strong as calcified minerals that are found in human dentin and harder materials such as bones,” Buehler said.

At MIT, the researchers looked at the makeup of the Nereis jaw on a molecular scale to see what makes the jaw so strong and adaptive. At this scale, the metal-coordinated crosslinks, the presence of metal in its molecular structure, provide a molecular network that makes the material stronger and at the same time make the molecular bond more dynamic, and ultimately able to respond to changing conditions. At the macroscopic scale, these dynamic metal-protein bonds result in an expansion/contraction behavior.

Combining the protein structural studies from AFRL with the molecular understanding from LAMM, Buehler, Martin-Martinez, CEE Research Scientist Zhao Qin, and former PhD student Chia-Ching Chou ’15, created a multiscale model that is able to predict the mechanical behavior of materials that contain this protein in various environments. “These atomistic simulations help us to visualize the atomic arrangements and molecular conformations that underlay the mechanical performance of these materials,” Martin-Martinez said.

Specifically, using this model the research team was able to design, test, and visualize how different molecular networks change and adapt to various pH levels, taking into account the biological and mechanical properties.

By looking at the molecular and biological makeup of a the Nereis virens and using the predictive model of the mechanical behavior of the resulting protein material, the LAMM researchers were able to more fully understand the protein material at different scales and provide a comprehensive understanding of how such protein materials form and behave in differing pH settings. This understanding guides new material designs for soft robots and sensors.

Identifying the link between environmental properties and movement in the material

The predictive model explained how the pH sensitive materials change shape and behavior, which the researchers used for designing new PH-changing geometric structures. Depending on the original geometric shape tested in the protein material and the properties surrounding it, the LAMM researchers found that the material either spirals or takes a Cypraea shell-like shape when the pH levels are changed. These are only some examples of the potential that this new material could have for developing soft robots, sensors, and autonomous structures.

Using the predictive model, the research team found that the material not only changes form, but it also reverts back to its original shape when the pH levels change. At the molecular level, histidine amino acids present in the protein bind strongly to the ions in the environment. This very local chemical reaction between amino acids and metal ions has an effect in the overall conformation of the protein at a larger scale. When environmental conditions change, the histidine-metal interactions change accordingly, which affect the protein conformation and in turn the material response.

“Changing the pH or changing the ions is like flipping a switch. You switch it on or off, depending on what environment you select, and the hydrogel expands or contracts” said Martin-Martinez.

LAMM found that at the molecular level, the structure of the protein material is strengthened when the environment contains zinc ions and certain pH levels. This creates more stable metal-coordinated crosslinks in the material’s molecular structure, which makes the molecules more dynamic and flexible.

This insight into the material’s design and its flexibility is extremely useful for environments with changing pH levels. Its response of changing its figure to changing acidity levels could be used for soft robotics. “Most soft robotics require power supply to drive the motion and to be controlled by complex electronic devices. Our work toward designing of multifunctional material may provide another pathway to directly control the material property and deformation without electronic devices,” said Qin.

By studying and modeling the molecular makeup and the behavior of the primary protein responsible for the mechanical properties ideal for Nereis jaw performance, the LAMM researchers are able to link environmental properties to movement in the material and have a more comprehensive understanding of the strength of the Nereis jaw.

The research was funded by the Air Force Office of Scientific Research and the National Science Foundation’s Extreme Science and Engineering Discovery Environment (XSEDE) for the simulations.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory are trying to improve automated planners by giving them the benefit of human intuition. By encoding the strategies of high-performing human planners in a machine-readable form, they were able to improve the performance of competition-winning planning algorithms by between 10 and 15 percent on a challenging set of problems. Image: Jose-Luis Olivares/MIT

Every other year, the International Conference on Automated Planning and Scheduling hosts a competition in which computer systems designed by conference participants try to find the best solution to a planning problem, such as scheduling flights or coordinating tasks for teams of autonomous satellites.

On all but the most straightforward problems, however, even the best planning algorithms still aren’t as effective as human beings with a particular aptitude for problem-solving — such as MIT students.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory are trying to improve automated planners by giving them the benefit of human intuition. By encoding the strategies of high-performing human planners in a machine-readable form, they were able to improve the performance of competition-winning planning algorithms by 10 to 15 percent on a challenging set of problems.

The researchers are presenting their results this week at the Association for the Advancement of Artificial Intelligence’s annual conference.

“In the lab, in other investigations, we’ve seen that for things like planning and scheduling and optimization, there’s usually a small set of people who are truly outstanding at it,” says Julie Shah, an assistant professor of aeronautics and astronautics at MIT. “Can we take the insights and the high-level strategies from the few people who are truly excellent at it and allow a machine to make use of that to be better at problem-solving than the vast majority of the population?”

The first author on the conference paper is Joseph Kim, a graduate student in aeronautics and astronautics. He’s joined by Shah and Christopher Banks, an undergraduate at Norfolk State University who was a research intern in Shah’s lab in the summer of 2016.

The human factor

Algorithms entered in the automated-planning competition — called the International Planning Competition, or IPC — are given related problems with different degrees of difficulty. The easiest problems require satisfaction of a few rigid constraints: For instance, given a certain number of airports, a certain number of planes, and a certain number of people at each airport with particular destinations, is it possible to plan planes’ flight routes such that all passengers reach their destinations but no plane ever flies empty?

A more complex class of problems — numerical problems — adds some flexible numerical parameters: Can you find a set of flight plans that meets the constraints of the original problem but also minimizes planes’ flight time and fuel consumption?

Finally, the most complex problems — temporal problems — add temporal constraints to the numerical problems: Can you minimize flight time and fuel consumption while also ensuring that planes arrive and depart at specific times?

For each problem, an algorithm has a half-hour to generate a plan. The quality of the plans is measured according to some “cost function,” such as an equation that combines total flight time and total fuel consumption.

Shah, Kim, and Banks recruited 36 MIT undergraduate and graduate students and posed each of them the planning problems from two different competitions, one that focused on plane routing and one that focused on satellite positioning. Like the automatic planners, the students had a half-hour to solve each problem.

“By choosing MIT students, we’re basically choosing the world experts in problem solving,” Shah says. “Likely, they’re going to be better at it than most of the population.”

Encoding strategies

Certainly, they were better than the automatic planners. After the students had submitted their solutions, Kim interviewed them about the general strategies they had used to solve the problems. Their answers included things like “Planes should visit each city at most once,” and “For each satellite, find routes in three turns or less.”

The researchers discovered that the large majority of the students’ strategies could be described using a formal language called linear temporal logic, which in turn could be used to add constraints to the problem specifications. Because different strategies could cancel each other out, the researchers tested each student’s strategies separately, using the planning algorithms that had won their respective competitions. The results varied, but only slightly. On the numerical problems, the average improvement was 13 percent and 16 percent, respectively, on the flight-planning and satellite-positioning problems; and on the temporal problems, the improvement was 12 percent and 10 percent.

“The plan that the planner came up with looked more like the human-generated plan when it used these high-level strategies from the person,” Shah says. “There is maybe this bridge to taking a user’s high-level strategy and making that useful for the machine, and by making it useful for the machine, maybe it makes it more interpretable to the person.”

In ongoing work, Kim and Shah are using natural-language-processing techniques to make the system fully automatic, so that it will convert users’ free-form descriptions of their high-level strategies into linear temporal logic without human intervention.

Read the preprint accepted at AAAI here. 

In 2016, MIT CSAIL researchers worked on a range of projects in robotics, theory, wireless technology, software systems, and other disciplines. Image: CSAIL

Machines that predict the future, robots that patch wounds and wireless emotion-detectors are just a few of the exciting projects that came out of MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) this year. Here’s a sampling of 16 highlights from 2016 that span the many computer science disciplines that make up CSAIL.

Robots for exploring Mars — and your stomach

• A team led by CSAIL director Daniela Rus developed an ingestible origami robot that unfolds in the stomach to patch wounds and remove swallowed batteries.
• Researchers are working on NASA’s humanoid robot, “Valkyrie,” who will be programmed for trips into outer space and to autonomously perform tasks.
• A 3-D printed robot was made of both solids and liquids and printed in one single step, with no assembly required.

Keeping data safe and secure

• CSAIL hosted a cyber summit that convened members of academia, industry, and government, including featured speakers Admiral Michael Rogers, director of the National Security Agency; and Andrew McCabe, deputy director of the Federal Bureau of Investigation.
• Researchers came up with a system for staying anonymous online that uses less bandwidth to transfer large files between anonymous users.
• A deep-learning system called AI2 was shown to be able to predict 85 percent of cyberattacks with the help of some human input.

Advancements in computer vision

• A new imaging technique called Interactive Dynamic Video lets you reach in and “touch” objects in videos using a normal camera.
• Researchers from CSAIL and Israel’s Weizmann Institute of Science produced a movie display called Cinema 3D that uses special lenses and mirrors to allow viewers to watch 3-D movies in a theater without having to wear those clunky 3-D glasses.
• A new deep-learning algorithm can predict human interactions more accurately than ever before, by training itself on footage from TV shows like “Desperate Housewives” and “The Office.”
• A group from MIT and Harvard University developed an algorithm that may help astronomers produce the first image of a black hole, stitching together telescope data to essentially turn the planet into one large telescope dish.

Tech to help with health

• A team produced a robot that can help schedule and assign tasks by learning from humans, in fields like medicine and the military.
• Researchers came up with an algorithm for identifying organs in fetal MRI scans to extensively evaluate prenatal health.
• A wireless device called EQ-Radio can tell if you’re excited, happy, angry, or sad, by measuring breathing and heart rhythms.

Algorithms, systems and networks

• A system called “Polaris” was found to load web pages 34 percent faster by decreasing network trips.
• A team analyzed ant-colony behavior to create better algorithms for network communication, for applications such as social networks and collective decision-making among robot swarms.
• Researchers trained neural networks to explain themselves by providing rationales for their decisions.

MIT researchers have developed a new approach to training speech-recognition systems that doesn’t depend on transcription. Instead, their system analyzes correspondences between images and spoken descriptions of those images, as captured in a large collection of audio recordings. Image: MIT News

Speech recognition systems, such as those that convert speech to text on cellphones, are generally the result of machine learning. A computer pores through thousands or even millions of audio files and their transcriptions, and learns which acoustic features correspond to which typed words.

But transcribing recordings is costly, time-consuming work, which has limited speech recognition to a small subset of languages spoken in wealthy nations.

At the Neural Information Processing Systems conference this week, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) are presenting a new approach to training speech-recognition systems that doesn’t depend on transcription. Instead, their system analyzes correspondences between images and spoken descriptions of those images, as captured in a large collection of audio recordings. The system then learns which acoustic features of the recordings correlate with which image characteristics.

“The goal of this work is to try to get the machine to learn language more like the way humans do,” says Jim Glass, a senior research scientist at CSAIL and a co-author on the paper describing the new system. “The current methods that people use to train up speech recognizers are very supervised. You get an utterance, and you’re told what’s said. And you do this for a large body of data.

“Big advances have been made — Siri, Google — but it’s expensive to get those annotations, and people have thus focused on, really, the major languages of the world. There are 7,000 languages, and I think less than 2 percent have ASR [automatic speech recognition] capability, and probably nothing is going to be done to address the others. So if you’re trying to think about how technology can be beneficial for society at large, it’s interesting to think about what we need to do to change the current situation. And the approach we’ve been taking through the years is looking at what we can learn with less supervision.”

Joining Glass on the paper are first author David Harwath, a graduate student in electrical engineering and computer science (EECS) at MIT; and Antonio Torralba, an EECS professor.

Visual semantics

The version of the system reported in the new paper doesn’t correlate recorded speech with written text; instead, it correlates speech with groups of thematically related images. But that correlation could serve as the basis for others.

If, for instance, an utterance is associated with a particular class of images, and the images have text terms associated with them, it should be possible to find a likely transcription of the utterance, all without human intervention. Similarly, a class of images with associated text terms in different languages could provide a way to do automatic translation.

Conversely, text terms associated with similar clusters of images, such as, say, “storm” and “clouds,”  could be inferred to have related meanings. Because the system in some sense learns words’ meanings — the images associated with them — and not just their sounds, it has a wider range of potential applications than a standard speech recognition system.

To test their system, the researchers used a database of 1,000 images, each of which had a recording of a free-form verbal description associated with it. They would feed their system one of the recordings and ask it to retrieve the 10 images that best matched it. That set of 10 images would contain the correct one 31 percent of the time.

“I always emphasize that we’re just taking baby steps here and have a long way to go,” Glass says. “But it’s an encouraging start.”

The researchers trained their system on images from a huge database built by Torralba; Aude Oliva, a principal research scientist at CSAIL; and their students. Through Amazon’s Mechanical Turk crowdsourcing site, they hired people to describe the images verbally, using whatever phrasing came to mind, for about 10 to 20 seconds.

For an initial demonstration of the researchers’ approach, that kind of tailored data was necessary to ensure good results. But the ultimate aim is to train the system using digital video, with minimal human involvement. “I think this will extrapolate naturally to video,” Glass says.

Merging modalities

To build their system, the researchers used neural networks, machine-learning systems that approximately mimic the structure of the brain. Neural networks are composed of processing nodes that, like individual neurons, are capable of only very simple computations but are connected to each other in dense networks. Data is fed to a network’s input nodes, which modify it and feed it to other nodes, which modify it and feed it to still other nodes, and so on. When a neural network is being trained, it constantly modifies the operations executed by its nodes in order to improve its performance on a specified task.

The researchers’ network is, in effect, two separate networks: one that takes images as input and one that takes spectrograms, which represent audio signals as changes of amplitude, over time, in their component frequencies. The output of the top layer of each network is a 1,024-dimensional vector — a sequence of 1,024 numbers.

The final node in the network takes the dot product of the two vectors. That is, it multiplies the corresponding terms in the vectors together and adds them all up to produce a single number. During training, the networks had to try to maximize the dot product when the audio signal corresponded to an image and minimize it when it didn’t.

For every spectrogram that the researchers’ system analyzes, it can identify the points at which the dot-product peaks. In experiments, those peaks reliably picked out words that provided accurate image labels — “baseball,” for instance, in a photo of a baseball pitcher in action, or “grassy” and “field” for an image of a grassy field.

In ongoing work, the researchers have refined the system so that it can pick out spectrograms of individual words and identify just those regions of an image that correspond to them.

“Possibly, a baby learns to speak from its perception of the environment, a large part of which may be visual,” says Lin-shan Lee, a professor of electrical engineering and computer science at National Taiwan University. “Today, machines have started to mimic such a learning process. This work is one of the earliest efforts in this direction, and I was really impressed when I first learned of it.”

“Perhaps even more exciting is just the question of how much we can learn with deep neural networks,” adds Karen Livescu, an assistant professor at the Toyota Technological Institute at the University of Chicago. “The more the research community does with them, the more we realize that they can learn a lot from big piles of data. But it is hard to label big piles of data, so it’s really exciting that in this work, Harwath et al. are able to learn from unlabeled data. I am really curious to see how far they can take that.”

Read the paper here.

A new cordless virtual reality device consists of two directional “phased-array” antennas, each less than half the size of a credit card. Future versions could be small enough for users to have several in a single room, enabling multi-player gameplay. Photo: MIT CSAIL

One of the limits of today’s virtual reality (VR) headsets is that they have to be tethered to computers in order to process data well enough to deliver high-resolution visuals. But wearing an HDMI cable reduces mobility and can even lead to users tripping over cords.

Fortunately, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have recently unveiled a prototype system called “MoVR” that allows gamers to use any VR headset wirelessly.

In tests, the team showed that MoVR can enable untethered communication at a rate of multiple Gbps, or billions of bits per second. The system uses special high-frequency radio signals called “millimeter waves” (mmWaves) that many experts think could someday help deliver blazingly-fast 5G smartphones.

“It’s very exciting to get a step closer to being able to deliver a high-resolution, wireless-VR experience,” says MIT professor Dina Katabi, whose research group has developed the technology. “The ability to use a cordless headset really deepens the immersive experience of virtual reality and opens up a range of other applications.”

Researchers tested the system on an HTC Vive but say that it can work with any headset. Katabi co-wrote a paper on the topic with PhD candidate Omid Abari, postdoc Dinesh Bharadia, and master’s student Austin Duffield. The team presented their findings last week at the ACM Workshop on Hot Topics in Networks (HotNets 2016) in Atlanta.

How it works

One issue with existing wireless technologies like WiFi is that they can’t support advanced data-processing.

The “MoVR” system allows for cable-free virtual-reality gaming, eliminating the possibility of users tripping over cords. Photo: MIT CSAIL

“Replacing the HDMI cable with a wireless link is very challenging since we need to stream high-resolution multi-view video in real-time,” says Haitham Hassanieh, an assistant professor of electrical and computer engineering at the University of Illinois at Urbana Champaigna who was not involved in the research. “This requires sustaining data rates of more than 6 Gbps while the user is moving and turning, which cannot be achieved by any of today’s systems.”

Since VR platforms have to work in real-time, systems also can’t use compression to accommodate lower data rates. This has led companies to make some pretty awkward attempts at untethered VR, like stuffing the equivalent of a full PC in your backpack.

The CSAIL team instead turned to mmWaves, which have promising applications for everything from high-speed Internet to cancer diagnosis. These high-frequency waves have one major downside, which is that they don’t work well with obstacles or reflections. If you want mmWaves to deliver constant connectivity for your VR game, you would need to always have a line of sight between transmitter and receiver. (The signal can be blocked even by just briefly moving your hand in front of the headset.)

To overcome this challenge, the team developed MoVR to act as a programmable mirror that detects the direction of the incoming mmWave signal and reconfigures itself to reflect it toward the receiver on the headset. MoVR can learn the correct signal direction to within two degrees, allowing it to correctly configure its angles.

“With a traditional mirror, light reflects off the mirror at the same angle as it arrives,” says Abari. “But with MoVR, angles can be specifically programmed so that the mirror receives the signal from the mmWave transmitter and reflects it towards the headset, regardless of its actual direction.”

Each MoVR device consists of two directional antennas that are each less than half the size of a credit card. The antennas use what are called “phased arrays” in order to focus signals into narrow beams that can be electronically steered at a timescale of microseconds.

Abari says that future versions of MoVR’s hardware could be as small as a smartphone, allowing for users to put several devices in a single room. This would enable multiple people to play a game at the same time without blocking each others’ signals.

Information extraction — or automatically classifying data items stored as plain text — is a major topic of artificial-intelligence research. Image: MIT News

Of the vast wealth of information unlocked by the Internet, most is plain text. The data necessary to answer myriad questions — about, say, the correlations between the industrial use of certain chemicals and incidents of disease, or between patterns of news coverage and voter-poll results — may all be online. But extracting it from plain text and organizing it for quantitative analysis may be prohibitively time-consuming.

Information extraction — or automatically classifying data items stored as plain text — is thus a major topic of artificial-intelligence research. Last week, at the Association for Computational Linguistics’ Conference on Empirical Methods on Natural Language Processing, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory won a best-paper award for a new approach to information extraction that turns conventional machine learning on its head.

Most machine-learning systems work by combing through training examples and looking for patterns that correspond to classifications provided by human annotators. For instance, humans might label parts of speech in a set of texts, and the machine-learning system will try to identify patterns that resolve ambiguities — for instance, when “her” is a direct object and when it’s an adjective.

Typically, computer scientists will try to feed their machine-learning systems as much training data as possible. That generally increases the chances that a system will be able to handle difficult problems.

In their new paper, by contrast, the MIT researchers train their system on scanty data — because in the scenario they’re investigating, that’s usually all that’s available. But then they find the limited information an easy problem to solve.

“In information extraction, traditionally, in natural-language processing, you are given an article and you need to do whatever it takes to extract correctly from this article,” says Regina Barzilay, the Delta Electronics Professor of Electrical Engineering and Computer Science and senior author on the new paper. “That’s very different from what you or I would do. When you’re reading an article that you can’t understand, you’re going to go on the web and find one that you can understand.”

Confidence boost

Essentially, the researchers’ new system does the same thing. A machine-learning system will generally assign each of its classifications a confidence score, which is a measure of the statistical likelihood that the classification is correct, given the patterns discerned in the training data. With the researchers’ new system, if the confidence score is too low, the system automatically generates a web search query designed to pull up texts likely to contain the data it’s trying to extract.

It then attempts to extract the relevant data from one of the new texts and reconciles the results with those of its initial extraction. If the confidence score remains too low, it moves on to the next text pulled up by the search string, and so on.

“The base extractor isn’t changing,” says Adam Yala, a graduate student in the MIT Department of Electrical Engineering and Computer Science (EECS) and one of the coauthors on the new paper. “You’re going to find articles that are easier for that extractor to understand. So you have something that’s a very weak extractor, and you just find data that fits it automatically from the web.” Joining Yala and Barzilay on the paper is first author Karthik Narasimhan, also a graduate student in EECS.

Remarkably, every decision the system makes is the result of machine learning. The system learns how to generate search queries, gauge the likelihood that a new text is relevant to its extraction task, and determine the best strategy for fusing the results of multiple attempts at extraction.

Just the facts

In experiments, the researchers applied their system to two extraction tasks. One was the collection of data on mass shootings in the U.S., which is an essential resource for any epidemiological study of the effects of gun-control measures. The other was the collection of similar data on instances of food contamination. The system was trained separately for each task.

In the first case — the database of mass shootings — the system was asked to extract the name of the shooter, the location of the shooting, the number of people wounded, and the number of people killed. In the food-contamination case, it extracted food type, type of contaminant, and location. In each case, the system was trained on about 300 documents.

From those documents, it learned clusters of search terms that tended to be associated with the data items it was trying to extract. For instance, the names of mass shooters were correlated with terms like “police,” “identified,” “arrested,” and “charged.” During training, for each article the system was asked to analyze, it pulled up, on average, another nine or 10 news articles from the web.

The researchers compared their system’s performance to that of several extractors trained using more conventional machine-learning techniques. For every data item extracted in both tasks, the new system outperformed its predecessors, usually by about 10 percent.

“One of the difficulties of natural language is that you can express the same information in many, many different ways, and capturing all that variation is one of the challenges of building a comprehensive model,” says Chris Callison-Burch, an assistant professor of computer and information science at the University of Pennsylvania. “[Barzilay and her colleagues] have this super-clever part of the model that goes out and queries for more information that might result in something that’s simpler for it to process. It’s clever and well-executed.”

Callison-Burch’s group is using a combination of natural-language processing and human review to build a database of information on gun violence, much like the one that the MIT researchers’ system was trained to produce. “We’ve crawled millions and millions of news articles, and then we pick out ones that the text classifier thinks are related to gun violence, and then we have humans start doing information extraction manually,” he says. “Having a model like Regina’s that would allow us to predict whether or not this article corresponded to one that we’ve already annotated would be a huge time savings. It’s something that I’d be very excited to do in the future.”

Read the paper here.

An autonomous mobility scooter and related software were designed by researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), the National University of Singapore, and the Singapore-MIT Alliance for Research and Technology (SMART). Courtesy of the Autonomous Vehicle Team of the SMART Future of Urban Mobility Project

By: Larry Hardesty

At MIT’s 2016 Open House last spring, more than 100 visitors took rides on an autonomous mobility scooter in a trial of software designed by researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), the National University of Singapore, and the Singapore-MIT Alliance for Research and Technology (SMART).

The researchers had previously used the same sensor configuration and software in trials of autonomous cars and golf carts, so the new trial completes the demonstration of a comprehensive autonomous mobility system. A mobility-impaired user could, in principle, use a scooter to get down the hall and through the lobby of an apartment building, take a golf cart across the building’s parking lot, and pick up an autonomous car on the public roads.

The new trial establishes that the researchers’ control algorithms work indoors as well as out. “We were testing them in tighter spaces,” says Scott Pendleton, a graduate student in mechanical engineering at the National University of Singapore (NUS) and a research fellow at SMART. “One of the spaces that we tested in was the Infinite Corridor of MIT, which is a very difficult localization problem, being a long corridor without very many distinctive features. You can lose your place along the corridor. But our algorithms proved to work very well in this new environment.”

The researchers’ system includes several layers of software: low-level control algorithms that enable a vehicle to respond immediately to changes in its environment, such as a pedestrian darting across its path; route-planning algorithms; localization algorithms that the vehicle uses to determine its location on a map; map-building algorithms that it uses to construct the map in the first place; a scheduling algorithm that allocates fleet resources; and an online booking system that allows users to schedule rides.

https://www.youtube.com/watch?v=h7ehanUmDhQ

Uniformity

Using the same control algorithms for all types of vehicles — scooters, golf carts, and city cars — has several advantages. One is that it becomes much more practical to perform reliable analyses of the system’s overall performance.

“If you have a uniform system where all the algorithms are the same, the complexity is much lower than if you have a heterogeneous system where each vehicle does something different,” says Daniela Rus, the Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science at MIT and one of the project’s leaders. “That’s useful for verifying that this multilayer complexity is correct.”

Furthermore, with software uniformity, information that one vehicle acquires can easily be transferred to another. Before the scooter was shipped to MIT, for instance, it was tested in Singapore, where it used maps that had been created by the autonomous golf cart.

Similarly, says Marcelo Ang, an associate professor of mechanical engineering at NUS who co-leads the project with Rus, in ongoing work the researchers are equipping their vehicles with machine-learning systems, so that interactions with the environment will improve the performance of their navigation and control algorithms. “Once you have a better driver, you can easily transplant that to another vehicle,” says Ang. “That’s the same across different platforms.”

Finally, software uniformity means that the scheduling algorithm has more flexibility in its allocation of system resources. If an autonomous golf cart isn’t available to take a user across a public park, a scooter could fill in; if a city car isn’t available for a short trip on back roads, a golf cart might be.

“I can see its usefulness in large indoor shopping malls and amusement parks to take [mobility-impaired] people from one spot to another,” says Dan Ding, an associate professor of rehabilitation science and technology at the University of Pittsburgh, about the system.

Changing perceptions

The scooter trial at MIT also demonstrated the ease with which the researchers could deploy their modular hardware and software system in a new context. “It’s extraordinary to me, because it’s a project that the team conducted in about two months,” Rus says. MIT’s Open House was at the end of April, and “the scooter didn’t exist on February 1st,” Rus says.

The researchers described the design of the scooter system and the results of the trial in a paper they presented last week at the IEEE International Conference on Intelligent Transportation Systems. Joining Rus, Pendleton, and Ang on the paper are You Hong Eng, who leads the SMART autonomous-vehicle project, and four other researchers from both NUS and SMART.

The paper also reports the results of a short user survey that the researchers conducted during the trial. Before riding the scooter, users were asked how safe they considered autonomous vehicles to be, on a scale from one to five; after their rides, they were asked the same question again. Experience with the scooter brought the average safety score up, from 3.5 to 4.6.

Electric motor, Rotterdam Ahoy Europort 2011. Source: Wikipedia Commons

Electric motors have been around since Thomas Davenport built the first functional model in 1834, and they have played a growing part in our lives ever since. Today, they continue to replace diesel and gas engines, as well as hydraulic cylinders, while evolving into new designs optimized for robots and other technologies.

“Something like 40 percent of electric power is used to drive motors, and that number will only grow,” says James Kirtley, a professor in MIT’s Department of Electrical Engineering and Computer Science and in MIT’s Research Laboratory of Electronics. “Electric motors are being used more widely in ships, airplanes, trains, and cars. We’re also seeing a lot more electric motors in robots.”

The ongoing transition from gas to electric is primarily driven by the need for more efficient devices that run with cleaner energy sources. Yet, electric motors also tend to be more responsive, and are more adaptable to new applications, especially in smaller devices.

As one of the world’s leading experts on electric motors, Kirtley’s philosophy is that one size does not fit all. “If you take into account the specific application, you can build a motor that is far better adapted than a general purpose motor,” says Kirtley. “For example, I’m working with someone who is making robots for medical assist, and he needs motors with very special characteristics.”

Many of the newer types of electric motors tend to be much smaller and run on less power than in the past. “I started working with electric motors 40 years ago designing big nuclear generators with 20-foot long rotors that were 6 feet in diameter and could produce a gigawatt and a half of power,” says Kirtley. “I’m now building motors on the order of 100 to 200 Watts for appliance motors, which are kinder and gentler to the system powering them. In the automotive industry, the average automobile has dozens of small motors for things like door locks, wiper blades, air conditioning, and seat positioners.”

Even excluding the separate field of MEMS (micro-electromechanical systems), which Kirtley is not directly involved in, electric motors are now shrinking to as small as the 1 W devices found in cell phones. A variety of innovative new compact motors are being developed all over MIT, says Kirtley, who points to an interesting variable reluctance motor for a prosthetic foot being designed at MIT’s Center for Art, Science and Technology.

Clean transportation is another source of innovation in electric motors. “In my lab we’re doing work with a small company in Cambridge that makes bicycle assist wheels,” says Kirtley. “The wheel stores some energy, and can react to pedaling forces to help it climb hills. These are entirely new applications.”

Improving microgrids with smart motors

Kirtley’s early involvement with power-plant generators led him to study electric power systems. His research into the subject culminated in 2010 with his Wiley-published book, “Electric Power Principles.” Lately, he’s been focusing on the customer-end of the system, where he is finding a role for electric motors in helping distribution systems adapt to intermittent, user-generated solar power.

“Electric power distribution systems are being stretched by the growing use of distributed renewable generation, such as rooftop solar,” says Kirtley. “Typically, electricity is transmitted from large power plants through extra high voltage wires, and the voltage is stepped down and delivered to customers. The problem with rooftop solar is that it looks to the power system like a reduction in load, but as solar cells become more widespread, homes will at times be able to produce more power than they’re consuming. So the power flows backward, which makes everything more complicated.”

Utilities are now working on smart microgrids that can adapt to distributed, multidirectional power. The greater flexibility is primarily enabled by smarter electronics, as well as efficient, distributed battery storage. Yet, microgrids still have a problem with even brief power outages, which can sometimes cause them to shut down.

“We’re thinking about the dynamics of motors connected to microgrids, and how you can improve their stability and make them work better,” Kirtley says. “We’re concerned about continuity of supply, which is especially important with microgrids supporting large server farms. You don’t want your system to be forced into an involuntary reboot simply because you had a glitch in the electric power supply. Electric motors can add more reliability.”

Electrifying the Cheetah robot

Many of the recent innovations in electric motors are found in robotics, which need smarter motors that can reliably deliver variable levels of power on demand for short periods. Electric motors can provide mobile robots with significantly longer battery life compared to traditional hydraulic systems. “Hydraulics are controlled primarily through throttle valves, so a lot of energy is wasted pumping and controlling the hydraulic fluid,” says Kirtley.

Kirtley has been working with Professor Jeffrey Lang on developing customized electric motors for Department of Mechanical Engineering colleague Sangbae Kim’s robotic cheetah, a running, jumping quadruped that has gained widespread publicity in recent years. The cheetah’s new motor is not only more efficient, but also more powerful, although only in short intervals.

“A secondary advantage of electric motors is responsiveness and control,” says Kirtley. “We can build a motor that can produce considerable torque in short spikes, even if we can’t necessarily produce the forces for a long period of time. It’s perhaps a little too powerful for the cheetah, which can now jump so high in the air, it probably wouldn’t survive the landing if they didn’t catch it.”

Electric motors transform ships

Most commercial ships still use diesel engines, while many naval vessels use gas turbine engines. Yet shipping is quickly moving toward electric propulsion, motivated primarily by efficiency, says Kirtley. Aside from nuclear-powered vessels, these tend to be hybrid systems in which diesel or gas generators drive an electric motor.

https://www.youtube.com/watch?v=_EV2qq2-Lck

“A traditional gear drive for ships has some very decided disadvantages,” he says. “For example, most destroyers in the U.S. Navy have gearboxes with very precise machining requirements, and are therefore expensive. They also require a fixed gear ratio between the engine and the water, so the prime mover is not operating near its peak fuel efficiency. Because of that, much of commercial shipping, and virtually all cruise ships, are now moving to electric propulsion, and even the U.S. Navy is starting to use it for its latest destroyer.”

The other problem with fuel-driven gear-drive engines is “the tyranny of the shaft line,” says Kirtley. “If you’re going to use a direct gear drive, the engine, gearbox, and propeller must line up very precisely, which often takes up valuable real estate within the body of the ship. With electric propulsion, we don’t have that problem. The engines can be placed anywhere where it’s convenient. In cruise ships, for example, the motors are place in a pod underneath the ship.”

Electric cars: adapting the engine to the road

The main difference between electric propulsion on ships vs. cars is related to torque requirements, says Kirtley. “On a ship, top speed defines the torque requirement,” he explains. “Automobile propulsion occurs across a speed range of about 10 to 1, with an engine that idles at 600 rpm capable of redlining at about 6,000 rpm. The best motors for cars are those that are adaptable to a very wide speed range.”

The variable speeds used in a car require that “the gearbox adapt the engine to the road,” says Kirtley. “You can generate an electric motor that can propel an automobile without a gear shift.”

In the past, Kirtley has consulted with Tesla Motors on its electric cars, and both agree that “the induction motor is the best for electric automobiles,” says Kirtley. Many other electric car manufacturers are still using permanent magnet motors, which he says are intrinsically less efficient in the wide speed and torque range required by car propulsion.

“For any given electric motor there is a tradeoff between excitation — making the operating magnetic field within the machine — and reaction, providing current to push on that exciting field inside the machine,” explains Kirtley. “In a permanent magnet machine that field is constant and cannot be adjusted, so a machine that is turning very fast but making relatively little torque is dropping a lot of power in losses in the machine’s magnetic iron. In an induction motor you can back off on the excitation to provide torque at the energetically optimal fashion. You can improve the drive efficiency of a car over a complete drive cycle by as much of a factor of two in fewer losses.”

Induction motors aren’t optimal for all applications, however, which brings Kirtley back to his main thesis: “In the development of motors for modern applications, it is most important to understand the totality of operational requirements,” he says. “That is key to making electric motors that will accomplish what they do best: provide motion in a responsive and efficient fashion.”

Photo: Gretchen Ertl

Sarah Hensley is preparing an astronaut named Valkyrie for a mission to Mars. It is 6 feet tall, weighs 300 pounds, and is equipped with an extended chest cavity that makes it look distinctly female. Hensley spends much of her time this semester analyzing the movements of one of Valkyrie’s arms.

As a fourth-year electrical engineering student at MIT, Hensley is working with a team of researchers to prepare Valkyrie, a humanoid robot also known as R5, for future space missions. As a teenager in New Jersey, Hensley loved to read in her downtime, particularly Isaac Asimov’s classic robot series. “I’m a huge science fiction nerd — and now I’m actually getting to work with a robot that’s real and not just in books. That’s like, wow.”

Hensley is studying Valkyrie for an advanced independent research program, or SuperUROP, as one of only three undergraduate students in the Robot Locomotion Group in MIT’s Computer Science and Artificial Intelligence Laboratory. Most of her colleagues are graduate-level researchers and postdocs with extensive experience working on complex humanoids. The group is led by professor of electrical engineering and computer science Russ Tedrake, who successfully programmed Valkyrie’s predecessor (named Atlas) to open doors, turn valves, drill holes, climb stairs, and drive a car for the DARPA Robotics Challenge in 2015.

Valkyrie has 28 torque-controlled joints, four body cameras, and more than 200 individual sensors, Hensley says. The robot can walk, bend its joints, and turn a door handle. “This is one of the most advanced robots in the world. And it’s 20 feet from my desk,” she adds.

That’s largely because Valkyrie has a long way to go before it leaves for Mars. MIT is one of three institutions, including Northeastern University and the University of Edinburgh, that NASA selected to develop software enabling the robot to perform space-related tasks — open airlock hatches, attach and remove power cables, repair equipment, and retrieve samples. Oh yeah, and get to its feet when it falls down.

Hensley, who started in the lab over the summer, is intrigued by the challenge of harmonizing the movements of such a highly complex system. “I am trying to solve a very tricky problem,” she says. She’s working out how best to control Valkyrie’s elbow movements by comparing two potential approaches. One uses a main controller to gather information from the various motor systems within the arm, and then uses that data to make accurate movement decisions. The other approach is decentralized, and leaves it to each motor system to decide and act on its own.

Hensley gets animated discussing the alternatives. “Is it better to have multiple decision makers with access to different information? Or is better to have one decision maker choosing all of the motor inputs?” she asks. Hensley has already been accepted into a master’s degree program in electrical engineering at MIT. She hopes to be able to continue her work on Valkyrie.

Every day, Hensley leaves Tau Epsilon Phi, her co-ed fraternity house in the Back Bay; walks across the Massachusetts Ave. bridge to the Stata Center; and plants herself in front of two large monitors in the robotics lab. She analyzes a wealth of scientific literature and writes code for computer simulations of the equations that move the robot’s arm. Sometimes she gets up for peppermint tea, or to peer around the corner of her cubicle at Valkyrie.

One thing is certain, says Hensley. Pop culture fears that machines may soon prove superior to humans are laughable. When Valkyrie is turned on and moves, Hensley says, it often “kind of shivers and falls down. One thing you realize working in this lab is that we are really far away from the robot apocalypse,” she quips. “Sometimes robots work, and sometimes they don’t. That’s our challenge.”

Robot from Computer Science and Artificial Intelligence Lab suggests where to move patients and who should do C-sections.

By: Adam Conner-Simons

Today’s robots are awkward co-workers because they are often unable to predict what humans need. In hospitals, robots are employed to perform simple tasks like delivering supplies and medications but they have to be explicitly told what to do.

A team from MIT’s Computer Science and Artificial Intelligence Lab (CSAIL) thinks that this will soon change, and that robots might be most effective by helping humans perform one of the most complex tasks of all: scheduling.

In a pair of new papers, CSAIL researchers demonstrate a robot that, by learning from human workers, can help assign and schedule tasks in fields ranging from medicine to the military.

In one paper, the team demonstrated a robot that assisted nurses in a labor ward, where it made recommendations on everything from where to move a patient to which nurse to assign to a C-section.

“The aim of the work was to develop artificial intelligence that can learn from people about how the labor and delivery unit works, so that robots can better anticipate how to be helpful or when to stay out of the way – and maybe even help by collaborating in making challenging decisions,” says MIT professor Julie Shah, the senior author on both papers.

In a second paper, the same system was put to the test in a videogame that simulates missile-defense scenarios. In the game, which was developed by Lincoln Laboratory researchers and involves using decoy missiles to ward off enemy attacks, the system even occasionally outperformed human experts at reducing both the number of missile attacks and the overall cost of decoys.

The labor-ward paper was presented at the recent Robotics: Science and Systems (RSS) Conference and was co-written by PhD student Matthew Gombolay, CSAIL postdocs Xi Jessie Yang and Brad Hayes, and Dr. Neel Shah and Toni Golen from Beth Israel Deaconess Medical Center, which is where the study took place.

The Navy-simulation paper is being presented at this week’s International Joint Conference on Artificial Intelligence (IJCAI), and was co-written by Gombolay and the  Lincoln Lab’s Reed Jensen, Jessica Stigile and Sung-Hyun Son.

Right on schedule

From visiting hospitals and factories, Shah and Gombolay found that a subset of workers are extremely strong schedulers, but can’t easily transfer that knowledge to colleagues.

“Figuring out what makes certain people good at this often seems like a mystery,” Gombolay says. “Being able to automate the task of learning from experts – and to then generalize it across industries – could help make many businesses run more efficiently.”

A particularly tough place for scheduling are hospitals. Labor wards’ head nurses have to try to predict when a woman will arrive in labor, how long labor will take, and which patients will become sick enough to require C-sections or other procedures.

They are deluged with an endless stream of challenging split-second decisions that include assigning nurses to patients, patients to beds, and technicians to surgeries. At Beth Israel, the head nurse has to coordinate 10 nurses, 20 patients and 20 rooms at the same time, meaning that the number of distinct scheduling possibilities adds up to a staggering 2 to the one millionth power, which is more than the number of atoms in the universe.

“We thought a complex environment like a labor ward would be a good place to try to automate scheduling and take this significant burden off of workers,” says Gombolay.

How it works

Like many AI systems, the team’s robot was trained via “learning from demonstration,” which involves observing humans’ performances of tasks. But Gombolay says that researchers have never been able to apply this technique to scheduling, because of the complexity of coordinating multiple actions that can be very dependent on each other.

To overcome this, the team trained its system to look at several actions that human schedulers make, and compare them to all the possible actions that are not made at each of those moments in time. From there, it developed a scheduling policy that can respond dynamically to new situations that it has not seen before.

“Rather than considering actions in isolation of each other, we crafted a model that understands why one action is better than the alternatives,” says Shah. “By considering all such comparisons, you can learn to recommend which action will be most helpful.”

The policy is “model-free,” meaning that the nurses do not have to train the robot by painstakingly ranking each possible action in each possible scenario by hand.

“You can put the robot on the labor floor, and, just by watching humans doing the different tasks, it will understand how to coordinate an efficient schedule,” says Gombolay.

The results

With this framework, the system – which the team has dubbed “apprenticeship scheduling” – can anticipate room assignments and suggest which nurses to assign to patients for C-sections and other procedures.

The approach was evaluated through experiments in which a robot provided decision-support to nurses and doctors as they made decisions on a labor floor.

Using the system on a Nao robot, nurses accepted the robot’s recommendations 90 percent of the time. The team also demonstrated that human subjects weren’t just blindly accepting advice – the robot delivered consciously bad feedback that was rejected at the same rate of 90 percent, showing that the system was trained to distinguish between good and bad recommendations.

Nurses had almost uniformly positive feedback about the robot. One said that it would “allow for a more even dispersion of workload,” while another said that it would be particularly helpful for “new nurses [who] may not understand the constraints and complexities of the role.”

“A great potential of this technology is that good solutions can be spread more quickly to many hospitals and workplaces,” says Dana Kulic, an associate professor of computer engineering at the University of Waterloo. “ For example, innovative improvements can be distributed rapidly from research hospitals to regional health centres.”

Shah says that the new techniques have many uses, from turning robots into better collaborators to helping train new nurses, but the goal is not to develop robots that fully make decisions on their own.

“These initial results show there is tremendous potential for machines to collaborate with us in rich ways that will enhance many sectors of the economy,” says Shah. “The awkward robots of the past will be replaced by valued team members.”

The RSS paper was supported by the National Science Foundation, CRICO Harvard Risk Management Foundation, and Aldebaran Robotics Inc. The IJCAI paper was supported the National Science Foundation and the U.S. Navy.

Read the paper here. 

Photo: Xi Jessie Yang

MIT-SUTD researchers are creating improved interfaces to help machines and humans work together to complete tasks.

As machines become more intelligent, they become embedded in countless facets of life. In some ways, they can act almost as full-fledged members in human-machine teams. In such cases, as with any team, trust is a necessary ingredient for good performance.

But the dynamics of trust between people and machines are not yet well-understood. With a two-year project funded through the SUTD-MIT Postdoctoral Fellows Program, a collaboration between MIT and the Singapore University of Technology and Design (SUTD), postdoc Xi Jessie Yang, MIT Department of Aeronautics and Astronautics Assistant Professor Julie Shah, and SUTD Engineering Product Development Professor Katja Hölttä-Otto aim to develop greater knowledge in this area. During her fellowship, Yang spent one year as a postdoc at SUTD, and is now finishing her research during a second year at MIT. She and her faculty mentors hope their efforts will contribute to better user interface design and better human-machine team performance.

Some tasks can simply be accomplished more effectively and efficiently by a machine or with assistance from a machine than by a human alone. But machines can make mistakes, especially in scenarios characterized by uncertainty and ambiguity. As such, a proper calibration of trust towards machines is a prerequisite of good human-machine team performance.

A variety of factors intersect to determine the extent to which the human trusts the machine, and in turn, how well they can perform on a given task. These factors include how reliable the machine is, how self-confident the human is, and the concurrent workload (i.e. how many tasks the human is responsible for carrying out simultaneously). Teasing out the effect of any one factor can be challenging, but through intricate experimental design, Yang, Shah and Hölttä-Otto have been able to approach the topic in a more realistic way, dealing with the imperfection of machines and isolating various factors in the human-machine trust relationship to study their impact.

At SUTD, Yang and Hölttä-Otto designed an experiment involving a human participant performing a memory and recognition task with the assistance from an automated decision aid. In the experiment, participants memorized a series of images and later recognized them in a pool of similar images. The decision aid provided recommendations for the images to select, very reliably for some people and not so for others. Results revealed that participants tended to undertrust highly reliable aids and overtrust highly unreliable ones. The trust-reliability calibration also improved as humans gained more experience using the decision aid.

At MIT, Yang and Shah explored how interface design could facilitate trust-reliability calibration. Specifically, of interest to Yang and Shah is the potential benefits of likelihood alarm displays. In contrast to traditional binary alarms indicating only “threat” or “no threat”, likelihood alarms provide additional information about the confidence level and the urgency of an alerted event. Yang and Shah believe that likelihood alarm would help mitigate the “cry wolf” effect, a phenomenon commonly observed in high-risk industries. In such industries, the threshold to trigger an alarm are oftentimes set very low in order to capture every critical event. The low threshold, however, inevitably results in false alarms which cause the human users to lose trust in the alarm system and eventually harm task performance.

In another experiment, currently underway, Yang and Shah simulate a multi-task scenario in which participants act as a solider to perform a search and detection task. They have to maintain level flight of four unpiloted aerial vehicles (UAVs) and at the same time detect potential threats in the photo steams sent back from the UAVs. Either a binary alarm or a likelihood alarm helps the participants to complete their mission. The latter would presumably alleviate the “cry wolf” effect, as well as improve participants’ attention allocation, and enhance human-machine team performance.

Ultimately, this research could help inform robotics and artificial intelligence development across many domains, from military applications like the simulations in the experiments to medical diagnostics to airport security.

“Lack of trust is often cited as one of the critical barriers to more widespread use of AI and autonomous systems,” Shah says. “These studies make it clear that increasing a user’s trust in the system is not the right goal. Instead we need new approaches that help people to appropriately calibrate their trust in the system, especially considering these systems will always be imperfect. ”

Yang will be presenting her research at the 7th International Conference on Applied Human Factors and Ergonomics this July, as well as the 60th annual meeting of the Human Factors and Ergonomics Society in September.

“It’s really exciting to see our small origami robots doing something with potential important applications to healthcare,” Daniela Rus says. Pictured, an example of a capsule and the unfolded origami device.
Photo: Melanie Gonick/MIT

By: Larry Hardesty

In experiments involving a simulation of the human esophagus and stomach, researchers at MIT, the University of Sheffield, and the Tokyo Institute of Technology have demonstrated a tiny origami robot that can unfold itself from a swallowed capsule and, steered by external magnetic fields, crawl across the stomach wall to remove a swallowed button battery or patch a wound.

The new work, which the researchers are presenting this week at the International Conference on Robotics and Automation, builds on a long sequence of papers on origami robots from the research group of Daniela Rus, the Andrew and Erna Viterbi Professor in MIT’s Department of Electrical Engineering and Computer Science.

“It’s really exciting to see our small origami robots doing something with potential important applications to health care,” says Rus, who also directs MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). “For applications inside the body, we need a small, controllable, untethered robot system. It’s really difficult to control and place a robot inside the body if the robot is attached to a tether.”

Joining Rus on the paper are first author Shuhei Miyashita, who was a postdoc at CSAIL when the work was done and is now a lecturer in electronics at the University of York, in England; Steven Guitron, a graduate student in mechanical engineering; Shuguang Li, a CSAIL postdoc; Kazuhiro Yoshida of Tokyo Institute of Technology, who was visiting MIT on sabbatical when the work was done; and Dana Damian of the University of Sheffield, in England.

Although the new robot is a successor to one reported at the same conference last year, the design of its body is significantly different. Like its predecessor, it can propel itself using what’s called a “stick-slip” motion, in which its appendages stick to a surface through friction when it executes a move, but slip free again when its body flexes to change its weight distribution.

Also like its predecessor — and like several other origami robots from the Rus group — the new robot consists of two layers of structural material sandwiching a material that shrinks when heated. A pattern of slits in the outer layers determines how the robot will fold when the middle layer contracts.

Material difference

The robot’s envisioned use also dictated a host of structural modifications. “Stick-slip only works when, one, the robot is small enough and, two, the robot is stiff enough,” says Guitron. “With the original Mylar design, it was much stiffer than the new design, which is based on a biocompatible material.”

To compensate for the biocompatible material’s relative malleability, the researchers had to come up with a design that required fewer slits. At the same time, the robot’s folds increase its stiffness along certain axes.

But because the stomach is filled with fluids, the robot doesn’t rely entirely on stick-slip motion. “In our calculation, 20 percent of forward motion is by propelling water — thrust — and 80 percent is by stick-slip motion,” says Miyashita. “In this regard, we actively introduced and applied the concept and characteristics of the fin to the body design, which you can see in the relatively flat design.”

It also had to be possible to compress the robot enough that it could fit inside a capsule for swallowing; similarly, when the capsule dissolved, the forces acting on the robot had to be strong enough to cause it to fully unfold. Through a design process that Guitron describes as “mostly trial and error,” the researchers arrived at a rectangular robot with accordion folds perpendicular to its long axis and pinched corners that act as points of traction.

In the center of one of the forward accordion folds is a permanent magnet that responds to changing magnetic fields outside the body, which control the robot’s motion. The forces applied to the robot are principally rotational. A quick rotation will make it spin in place, but a slower rotation will cause it to pivot around one of its fixed feet. In the researchers’ experiments, the robot uses the same magnet to pick up the button battery.

Porcine precedents

The researchers tested about a dozen different possibilities for the structural material before settling on the type of dried pig intestine used in sausage casings. “We spent a lot of time at Asian markets and the Chinatown market looking for materials,” Li says. The shrinking layer is a biodegradable shrink wrap called Biolefin.

To design their synthetic stomach, the researchers bought a pig stomach and tested its mechanical properties. Their model is an open cross-section of the stomach and esophagus, molded from a silicone rubber with the same mechanical profile. A mixture of water and lemon juice simulates the acidic fluids in the stomach.

Every year, 3,500 swallowed button batteries are reported in the U.S. alone. Frequently, the batteries are digested normally, but if they come into prolonged contact with the tissue of the esophagus or stomach, they can cause an electric current that produces hydroxide, which burns the tissue. Miyashita employed a clever strategy to convince Rus that the removal of swallowed button batteries and the treatment of consequent wounds was a compelling application of their origami robot.

“Shuhei bought a piece of ham, and he put the battery on the ham,” Rus says. “Within half an hour, the battery was fully submerged in the ham. So that made me realize that, yes, this is important. If you have a battery in your body, you really want it out as soon as possible.”

“This concept is both highly creative and highly practical, and it addresses a clinical need in an elegant way,” says Bradley Nelson, a professor of robotics at the Swiss Federal Institute of Technology Zurich. “It is one of the most convincing applications of origami robots that I have seen.”

This 3-D hexapod robot moves via a single motor, which spins a crankshaft that pumps fluid to the robot’s legs. Besides the motor and battery, every component is printed in a single step with no assembly required. Among the robot’s key parts are several sets of “bellows” 3-D printed directly into its body. To propel the robot, the bellows uses fluid pressure that is translated into a mechanical force. (As an alternative to the bellows, the team also demonstrated they could 3-D print a gear pump that can produce continuous fluid flow.)
Photo: Robert MacCurdy/MIT CSAIL

By Adam Conner-Simons | CSAIL

One reason we don’t yet have robot personal assistants buzzing around doing our chores is because making them is hard. Assembling robots by hand is time-consuming, while automation — robots building other robots — is not yet fine-tuned enough to make robots that can do complex tasks.

But if humans and robots can’t do the trick, what about 3-D printers?

In a new paper, researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) present the first-ever technique for 3-D printing robots that involves printing solid and liquid materials at the same time.

The new method allows the team to automatically 3-D print dynamic robots in a single step, with no assembly required, using a commercially-available 3-D printer.

“Our approach, which we call ‘printable hydraulics,’ is a step towards the rapid fabrication of functional machines,” says CSAIL Director Daniela Rus, who oversaw the project and co-wrote the paper. “All you have to do is stick in a battery and motor, and you have a robot that can practically walk right out of the printer.”

To demonstrate the concept, researchers 3-D printed a tiny six-legged robot that can crawl via 12 hydraulic pumps embedded within its body. They also 3-D printed robotic parts that can be used on existing platforms, such as a soft rubber hand for the Baxter research robot.

The paper, which was recently accepted to this summer’s IEEE International Conference on Robotics and Automation (ICRA), was co-written by MIT postdoc Robert MacCurdy and PhD candidate Robert Katzschmann, as well as Harvard University undergraduate Youbin Kim.

The printing process

For all of the progress in 3-D printing, liquids continue to be a big hurdle. Printing liquids is a messy process, which means that most approaches require an additional post-printing step such as melting it away or having a human manually scrape it clean. That step makes it hard for liquid-based methods to be employed for factory-scale manufacturing.

The team’s method of printing solid and liquid materials simultaneously allows them to create robotic structures that can be hydraulically powered.
Photo: Robert MacCurdy/MIT CSAIL

With “printable hydraulics,” an inkjet printer deposits individual droplets of material that are each 20 to 30 microns in diameter, or less than half the width of a human hair. The printer proceeds layer-by-layer from the bottom up. For each layer, the printer deposits different materials in different parts, and then uses high-intensity UV light to solidify all of the materials (minus, of course, the liquids). The printer uses multiple materials, though at a more basic level each layer consist of a “photopolymer,” which is a solid, and “a non-curing material,” which is a liquid.

“Inkjet printing lets us have eight different print-heads deposit different materials adjacent to one another, all at the same time,” MacCurdy says. “It gives us very fine control of material placement, which is what allows us to print complex, pre-filled fluidic channels.”

Another challenge with 3-D printing liquids is that they often interfere with the droplets that are supposed to solidify. To handle that issue, the team printed dozens of test geometries with different orientations to determine the proper resolutions for printing solids and liquids together.

While it’s a painstaking process, MacCurdy says that printing both liquids and solids is even more difficult with other 3-D printing methods, such as fused-deposition modeling and laser-sintering.

“As far as I’m concerned,” he says, “inkjet-printing is currently the best way to print multiple materials.”

The results

To demonstrate their method, researchers 3-D printed a small hexapod robot that weighs about 1.5 pounds and is less than 6 inches long. To move, a single DC motor spins a crankshaft that pumps fluid to the robot’s legs. Aside from its motor and power supply, every component is printed in a single step with no assembly required.

Among the robot’s key parts are several set of “bellows” that are 3-D printed directly into its body. To propel the robot, the bellows uses fluid pressure that is then translated into a mechanical force. (As an alternative to the bellows, the team also demonstrated they could 3-D print a gear pump that can produce continuous fluid flow.)

Photo: Robert MacCurdy/MIT CSAIL

Lastly, the team 3-D printed a silicone-rubber robotic hand with fluid-actuated fingers. This “soft gripper” was developed for Baxter, a robot that was designed by former CSAIL director Rodney Brooks as part of his spinoff company Rethink Robotics.

“The CSAIL team has taken multi-material printing to the next level by printing not just a combination of different polymers or a mixture of metals, but essentially a self-contained working hydraulic system,” says Hod Lipson, a professor of engineering at Columbia University and co-author of “Fabricated: The New World of 3-D Printing.” “It’s an important step towards the next big phase of 3-D printing — moving from printing passive parts to printing active integrated systems.”

Compatible with any multimaterial 3-D inkjet printer, “printable hydraulics” allows for a customizable design template that can create robots of different sizes, shapes and functions.

“If you have a crawling robot that you want to have step over something larger, you can tweak the design in a matter of minutes,” MacCurdy says. “In the future, the system will hardly need any human input at all; you can just press a few buttons, and it will automatically make the changes.”

MacCurdy envisions many potential applications, including disaster relief in dangerous environments. Many nuclear sites, for example, need to be remediated to reduce their radiation levels. Unfortunately, the sites are not only lethal to humans, but radioactive enough to destroy conventional electronics.

“Printable robots like these can be quickly, cheaply fabricated, with fewer electronic components than traditional robots,” MacCurdy says.

Looking ahead

The team is eager to further build on their work. While the hexapod’s 22-hour print-time is relatively short for its complexity, researchers say that future hardware advances would improve the speed.

“Accelerating the process depends less on the particulars of our technique, and more on the engineering and resolution of the printers themselves,” says Rus, the Viterbi Professor of Electrical Engineering and Computer Science at MIT. “Printing ultimately takes as long as the printer takes, so as printers improve, so will the manufacturing capabilities.”

This isn’t Rus’ group’s first foray into 3-D printed robots. This past fall her team developed a similar gripper, while in 2014 they created an arm that can snake through a pipe and grasp an object. But where these projects still required multiple non-3-D printed objects, “printable hydraulics” gets even closer to printing all components in one step.

“Building robots doesn’t have to be as time-consuming and labor-intensive as it’s been in the past,” Rus says. “3-D printing offers a way forward, allowing us to automatically produce complex, functional, hydraulically-powered robots that can be put to immediate use.”

Read the research here

The team’s work was funded, in part, by a grant from the National Science Foundation.