A GelSight sensor attached to a robot’s gripper enables the robot to determine precisely where it has grasped a small screwdriver, removing it from and inserting it back into a slot, even when the gripper screens the screwdriver from the robot’s camera. Photo: Robot Locomotion Group at MIT

Eight years ago, Ted Adelson’s research group at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) unveiled a new sensor technology, called GelSight, that uses physical contact with an object to provide a remarkably detailed 3-D map of its surface.

Now, by mounting GelSight sensors on the grippers of robotic arms, two MIT teams have given robots greater sensitivity and dexterity. The researchers presented their work in two papers at the International Conference on Robotics and Automation last week.

In one paper, Adelson’s group uses the data from the GelSight sensor to enable a robot to judge the hardness of surfaces it touches — a crucial ability if household robots are to handle everyday objects.

In the other, Russ Tedrake’s Robot Locomotion Group at CSAIL uses GelSight sensors to enable a robot to manipulate smaller objects than was previously possible.

The GelSight sensor is, in some ways, a low-tech solution to a difficult problem. It consists of a block of transparent rubber — the “gel” of its name — one face of which is coated with metallic paint. When the paint-coated face is pressed against an object, it conforms to the object’s shape.

The metallic paint makes the object’s surface reflective, so its geometry becomes much easier for computer vision algorithms to infer. Mounted on the sensor opposite the paint-coated face of the rubber block are three colored lights and a single camera.

“[The system] has colored lights at different angles, and then it has this reflective material, and by looking at the colors, the computer … can figure out the 3-D shape of what that thing is,” explains Adelson, the John and Dorothy Wilson Professor of Vision Science in the Department of Brain and Cognitive Sciences.

In both sets of experiments, a GelSight sensor was mounted on one side of a robotic gripper, a device somewhat like the head of a pincer, but with flat gripping surfaces rather than pointed tips.

Contact points

For an autonomous robot, gauging objects’ softness or hardness is essential to deciding not only where and how hard to grasp them but how they will behave when moved, stacked, or laid on different surfaces. Tactile sensing could also aid robots in distinguishing objects that look similar.

In previous work, robots have attempted to assess objects’ hardness by laying them on a flat surface and gently poking them to see how much they give. But this is not the chief way in which humans gauge hardness. Rather, our judgments seem to be based on the degree to which the contact area between the object and our fingers changes as we press on it. Softer objects tend to flatten more, increasing the contact area.

The MIT researchers adopted the same approach. Wenzhen Yuan, a graduate student in mechanical engineering and first author on the paper from Adelson’s group, used confectionary molds to create 400 groups of silicone objects, with 16 objects per group. In each group, the objects had the same shapes but different degrees of hardness, which Yuan measured using a standard industrial scale.

Then she pressed a GelSight sensor against each object manually and recorded how the contact pattern changed over time, essentially producing a short movie for each object. To both standardize the data format and keep the size of the data manageable, she extracted five frames from each movie, evenly spaced in time, which described the deformation of the object that was pressed.

Finally, she fed the data to a neural network, which automatically looked for correlations between changes in contact patterns and hardness measurements. The resulting system takes frames of video as inputs and produces hardness scores with very high accuracy. Yuan also conducted a series of informal experiments in which human subjects palpated fruits and vegetables and ranked them according to hardness. In every instance, the GelSight-equipped robot arrived at the same rankings.

Yuan is joined on the paper by her two thesis advisors, Adelson and Mandayam Srinivasan, a senior research scientist in the Department of Mechanical Engineering; Chenzhuo Zhu, an undergraduate from Tsinghua University who visited Adelson’s group last summer; and Andrew Owens, who did his PhD in electrical engineering and computer science at MIT and is now a postdoc at the University of California at Berkeley.

Obstructed views

The paper from the Robot Locomotion Group was born of the group’s experience with the Defense Advanced Research Projects Agency’s Robotics Challenge (DRC), in which academic and industry teams competed to develop control systems that would guide a humanoid robot through a series of tasks related to a hypothetical emergency.

Typically, an autonomous robot will use some kind of computer vision system to guide its manipulation of objects in its environment. Such systems can provide very reliable information about an object’s location — until the robot picks the object up. Especially if the object is small, much of it will be occluded by the robot’s gripper, making location estimation much harder. Thus, at exactly the point at which the robot needs to know the object’s location precisely, its estimate becomes unreliable. This was the problem the MIT team faced during the DRC, when their robot had to pick up and turn on a power drill.

“You can see in our video for the DRC that we spend two or three minutes turning on the drill,” says Greg Izatt, a graduate student in electrical engineering and computer science and first author on the new paper. “It would be so much nicer if we had a live-updating, accurate estimate of where that drill was and where our hands were relative to it.”

That’s why the Robot Locomotion Group turned to GelSight. Izatt and his co-authors — Tedrake, the Toyota Professor of Electrical Engineering and Computer Science, Aeronautics and Astronautics, and Mechanical Engineering; Adelson; and Geronimo Mirano, another graduate student in Tedrake’s group — designed control algorithms that use a computer vision system to guide the robot’s gripper toward a tool and then turn location estimation over to a GelSight sensor once the robot has the tool in hand.

In general, the challenge with such an approach is reconciling the data produced by a vision system with data produced by a tactile sensor. But GelSight is itself camera-based, so its data output is much easier to integrate with visual data than the data from other tactile sensors.

In Izatt’s experiments, a robot with a GelSight-equipped gripper had to grasp a small screwdriver, remove it from a holster, and return it. Of course, the data from the GelSight sensor don’t describe the whole screwdriver, just a small patch of it. But Izatt found that, as long as the vision system’s estimate of the screwdriver’s initial position was accurate to within a few centimeters, his algorithms could deduce which part of the screwdriver the GelSight sensor was touching and thus determine the screwdriver’s position in the robot’s hand.

“I think that the GelSight technology, as well as other high-bandwidth tactile sensors, will make a big impact in robotics,” says Sergey Levine, an assistant professor of electrical engineering and computer science at the University of California at Berkeley. “For humans, our sense of touch is one of the key enabling factors for our amazing manual dexterity. Current robots lack this type of dexterity and are limited in their ability to react to surface features when manipulating objects. If you imagine fumbling for a light switch in the dark, extracting an object from your pocket, or any of the other numerous things that you can do without even thinking — these all rely on touch sensing.”

“Software is finally catching up with the capabilities of our sensors,” Levine adds. “Machine learning algorithms inspired by innovations in deep learning and computer vision can process the rich sensory data from sensors such as the GelSight to deduce object properties. In the future, we will see these kinds of learning methods incorporated into end-to-end trained manipulation skills, which will make our robots more dexterous and capable, and maybe help us understand something about our own sense of touch and motor control.”

New algorithms power a prototype system for helping visually impaired users avoid obstacles and identify objects. Courtesy of the researchers.

Computer scientists have been working for decades on automatic navigation systems to aid the visually impaired, but it’s been difficult to come up with anything as reliable and easy to use as the white cane, the type of metal-tipped cane that visually impaired people frequently use to identify clear walking paths.

White canes have a few drawbacks, however. One is that the obstacles they come in contact with are sometimes other people. Another is that they can’t identify certain types of objects, such as tables or chairs, or determine whether a chair is already occupied.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed a new system that uses a 3-D camera, a belt with separately controllable vibrational motors distributed around it, and an electronically reconfigurable Braille interface to give visually impaired users more information about their environments.

The system could be used in conjunction with or as an alternative to a cane. In a paper they’re presenting this week at the International Conference on Robotics and Automation, the researchers describe the system and a series of usability studies they conducted with visually impaired volunteers.

“We did a couple of different tests with blind users,” says Robert Katzschmann, a graduate student in mechanical engineering at MIT and one of the paper’s two first authors. “Having something that didn’t infringe on their other senses was important. So we didn’t want to have audio; we didn’t want to have something around the head, vibrations on the neck — all of those things, we tried them out, but none of them were accepted. We found that the one area of the body that is the least used for other senses is around your abdomen.”

Katzschmann is joined on the paper by his advisor Daniela Rus, an Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science; his fellow first author Hsueh-Cheng Wang, who was a postdoc at MIT when the work was done and is now an assistant professor of electrical and computer engineering at National Chiao Tung University in Taiwan; Santani Teng, a postdoc in CSAIL; Brandon Araki, a graduate student in mechanical engineering; and Laura Giarré, a professor of electrical engineering at the University of Modena and Reggio Emilia in Italy.

Parsing the world

The researchers’ system consists of a 3-D camera worn in a pouch hung around the neck; a processing unit that runs the team’s proprietary algorithms; the sensor belt, which has five vibrating motors evenly spaced around its forward half; and the reconfigurable Braille interface, which is worn at the user’s side.

The key to the system is an algorithm for quickly identifying surfaces and their orientations from the 3-D-camera data. The researchers experimented with three different types of 3-D cameras, which used three different techniques to gauge depth but all produced relatively low-resolution images — 640 pixels by 480 pixels — with both color and depth measurements for each pixel.

The algorithm first groups the pixels into clusters of three. Because the pixels have associated location data, each cluster determines a plane. If the orientations of the planes defined by five nearby clusters are within 10 degrees of each other, the system concludes that it has found a surface. It doesn’t need to determine the extent of the surface or what type of object it’s the surface of; it simply registers an obstacle at that location and begins to buzz the associated motor if the wearer gets within 2 meters of it.

Chair identification is similar but a little more stringent. The system needs to complete three distinct surface identifications, in the same general area, rather than just one; this ensures that the chair is unoccupied. The surfaces need to be roughly parallel to the ground, and they have to fall within a prescribed range of heights.

Tactile data

The belt motors can vary the frequency, intensity, and duration of their vibrations, as well as the intervals between them, to send different types of tactile signals to the user. For instance, an increase in frequency and intensity generally indicates that the wearer is approaching an obstacle in the direction indicated by that particular motor. But when the system is in chair-finding mode, for example, a double pulse indicates the direction in which a chair with a vacant seat can be found.

The Braille interface consists of two rows of five reconfigurable Braille pads. Symbols displayed on the pads describe the objects in the user’s environment — for instance, a “t” for table or a “c” for chair. The symbol’s position in the row indicates the direction in which it can be found; the column it appears in indicates its distance. A user adept at Braille should find that the signals from the Braille interface and the belt-mounted motors coincide.

In tests, the chair-finding system reduced subjects’ contacts with objects other than the chairs they sought by 80 percent, and the navigation system reduced the number of cane collisions with people loitering around a hallway by 86 percent.

In 2016, MIT CSAIL researchers worked on a range of projects in robotics, theory, wireless technology, software systems, and other disciplines. Image: CSAIL

Machines that predict the future, robots that patch wounds and wireless emotion-detectors are just a few of the exciting projects that came out of MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) this year. Here’s a sampling of 16 highlights from 2016 that span the many computer science disciplines that make up CSAIL.

Robots for exploring Mars — and your stomach

• A team led by CSAIL director Daniela Rus developed an ingestible origami robot that unfolds in the stomach to patch wounds and remove swallowed batteries.
• Researchers are working on NASA’s humanoid robot, “Valkyrie,” who will be programmed for trips into outer space and to autonomously perform tasks.
• A 3-D printed robot was made of both solids and liquids and printed in one single step, with no assembly required.

Keeping data safe and secure

• CSAIL hosted a cyber summit that convened members of academia, industry, and government, including featured speakers Admiral Michael Rogers, director of the National Security Agency; and Andrew McCabe, deputy director of the Federal Bureau of Investigation.
• Researchers came up with a system for staying anonymous online that uses less bandwidth to transfer large files between anonymous users.
• A deep-learning system called AI2 was shown to be able to predict 85 percent of cyberattacks with the help of some human input.

Advancements in computer vision

• A new imaging technique called Interactive Dynamic Video lets you reach in and “touch” objects in videos using a normal camera.
• Researchers from CSAIL and Israel’s Weizmann Institute of Science produced a movie display called Cinema 3D that uses special lenses and mirrors to allow viewers to watch 3-D movies in a theater without having to wear those clunky 3-D glasses.
• A new deep-learning algorithm can predict human interactions more accurately than ever before, by training itself on footage from TV shows like “Desperate Housewives” and “The Office.”
• A group from MIT and Harvard University developed an algorithm that may help astronomers produce the first image of a black hole, stitching together telescope data to essentially turn the planet into one large telescope dish.

Tech to help with health

• A team produced a robot that can help schedule and assign tasks by learning from humans, in fields like medicine and the military.
• Researchers came up with an algorithm for identifying organs in fetal MRI scans to extensively evaluate prenatal health.
• A wireless device called EQ-Radio can tell if you’re excited, happy, angry, or sad, by measuring breathing and heart rhythms.

Algorithms, systems and networks

• A system called “Polaris” was found to load web pages 34 percent faster by decreasing network trips.
• A team analyzed ant-colony behavior to create better algorithms for network communication, for applications such as social networks and collective decision-making among robot swarms.
• Researchers trained neural networks to explain themselves by providing rationales for their decisions.

MIT researchers have developed a new approach to training speech-recognition systems that doesn’t depend on transcription. Instead, their system analyzes correspondences between images and spoken descriptions of those images, as captured in a large collection of audio recordings. Image: MIT News

Speech recognition systems, such as those that convert speech to text on cellphones, are generally the result of machine learning. A computer pores through thousands or even millions of audio files and their transcriptions, and learns which acoustic features correspond to which typed words.

But transcribing recordings is costly, time-consuming work, which has limited speech recognition to a small subset of languages spoken in wealthy nations.

At the Neural Information Processing Systems conference this week, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) are presenting a new approach to training speech-recognition systems that doesn’t depend on transcription. Instead, their system analyzes correspondences between images and spoken descriptions of those images, as captured in a large collection of audio recordings. The system then learns which acoustic features of the recordings correlate with which image characteristics.

“The goal of this work is to try to get the machine to learn language more like the way humans do,” says Jim Glass, a senior research scientist at CSAIL and a co-author on the paper describing the new system. “The current methods that people use to train up speech recognizers are very supervised. You get an utterance, and you’re told what’s said. And you do this for a large body of data.

“Big advances have been made — Siri, Google — but it’s expensive to get those annotations, and people have thus focused on, really, the major languages of the world. There are 7,000 languages, and I think less than 2 percent have ASR [automatic speech recognition] capability, and probably nothing is going to be done to address the others. So if you’re trying to think about how technology can be beneficial for society at large, it’s interesting to think about what we need to do to change the current situation. And the approach we’ve been taking through the years is looking at what we can learn with less supervision.”

Joining Glass on the paper are first author David Harwath, a graduate student in electrical engineering and computer science (EECS) at MIT; and Antonio Torralba, an EECS professor.

Visual semantics

The version of the system reported in the new paper doesn’t correlate recorded speech with written text; instead, it correlates speech with groups of thematically related images. But that correlation could serve as the basis for others.

If, for instance, an utterance is associated with a particular class of images, and the images have text terms associated with them, it should be possible to find a likely transcription of the utterance, all without human intervention. Similarly, a class of images with associated text terms in different languages could provide a way to do automatic translation.

Conversely, text terms associated with similar clusters of images, such as, say, “storm” and “clouds,”  could be inferred to have related meanings. Because the system in some sense learns words’ meanings — the images associated with them — and not just their sounds, it has a wider range of potential applications than a standard speech recognition system.

To test their system, the researchers used a database of 1,000 images, each of which had a recording of a free-form verbal description associated with it. They would feed their system one of the recordings and ask it to retrieve the 10 images that best matched it. That set of 10 images would contain the correct one 31 percent of the time.

“I always emphasize that we’re just taking baby steps here and have a long way to go,” Glass says. “But it’s an encouraging start.”

The researchers trained their system on images from a huge database built by Torralba; Aude Oliva, a principal research scientist at CSAIL; and their students. Through Amazon’s Mechanical Turk crowdsourcing site, they hired people to describe the images verbally, using whatever phrasing came to mind, for about 10 to 20 seconds.

For an initial demonstration of the researchers’ approach, that kind of tailored data was necessary to ensure good results. But the ultimate aim is to train the system using digital video, with minimal human involvement. “I think this will extrapolate naturally to video,” Glass says.

Merging modalities

To build their system, the researchers used neural networks, machine-learning systems that approximately mimic the structure of the brain. Neural networks are composed of processing nodes that, like individual neurons, are capable of only very simple computations but are connected to each other in dense networks. Data is fed to a network’s input nodes, which modify it and feed it to other nodes, which modify it and feed it to still other nodes, and so on. When a neural network is being trained, it constantly modifies the operations executed by its nodes in order to improve its performance on a specified task.

The researchers’ network is, in effect, two separate networks: one that takes images as input and one that takes spectrograms, which represent audio signals as changes of amplitude, over time, in their component frequencies. The output of the top layer of each network is a 1,024-dimensional vector — a sequence of 1,024 numbers.

The final node in the network takes the dot product of the two vectors. That is, it multiplies the corresponding terms in the vectors together and adds them all up to produce a single number. During training, the networks had to try to maximize the dot product when the audio signal corresponded to an image and minimize it when it didn’t.

For every spectrogram that the researchers’ system analyzes, it can identify the points at which the dot-product peaks. In experiments, those peaks reliably picked out words that provided accurate image labels — “baseball,” for instance, in a photo of a baseball pitcher in action, or “grassy” and “field” for an image of a grassy field.

In ongoing work, the researchers have refined the system so that it can pick out spectrograms of individual words and identify just those regions of an image that correspond to them.

“Possibly, a baby learns to speak from its perception of the environment, a large part of which may be visual,” says Lin-shan Lee, a professor of electrical engineering and computer science at National Taiwan University. “Today, machines have started to mimic such a learning process. This work is one of the earliest efforts in this direction, and I was really impressed when I first learned of it.”

“Perhaps even more exciting is just the question of how much we can learn with deep neural networks,” adds Karen Livescu, an assistant professor at the Toyota Technological Institute at the University of Chicago. “The more the research community does with them, the more we realize that they can learn a lot from big piles of data. But it is hard to label big piles of data, so it’s really exciting that in this work, Harwath et al. are able to learn from unlabeled data. I am really curious to see how far they can take that.”

Read the paper here.

A new cordless virtual reality device consists of two directional “phased-array” antennas, each less than half the size of a credit card. Future versions could be small enough for users to have several in a single room, enabling multi-player gameplay. Photo: MIT CSAIL

One of the limits of today’s virtual reality (VR) headsets is that they have to be tethered to computers in order to process data well enough to deliver high-resolution visuals. But wearing an HDMI cable reduces mobility and can even lead to users tripping over cords.

Fortunately, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have recently unveiled a prototype system called “MoVR” that allows gamers to use any VR headset wirelessly.

In tests, the team showed that MoVR can enable untethered communication at a rate of multiple Gbps, or billions of bits per second. The system uses special high-frequency radio signals called “millimeter waves” (mmWaves) that many experts think could someday help deliver blazingly-fast 5G smartphones.

“It’s very exciting to get a step closer to being able to deliver a high-resolution, wireless-VR experience,” says MIT professor Dina Katabi, whose research group has developed the technology. “The ability to use a cordless headset really deepens the immersive experience of virtual reality and opens up a range of other applications.”

Researchers tested the system on an HTC Vive but say that it can work with any headset. Katabi co-wrote a paper on the topic with PhD candidate Omid Abari, postdoc Dinesh Bharadia, and master’s student Austin Duffield. The team presented their findings last week at the ACM Workshop on Hot Topics in Networks (HotNets 2016) in Atlanta.

How it works

One issue with existing wireless technologies like WiFi is that they can’t support advanced data-processing.

The “MoVR” system allows for cable-free virtual-reality gaming, eliminating the possibility of users tripping over cords. Photo: MIT CSAIL

“Replacing the HDMI cable with a wireless link is very challenging since we need to stream high-resolution multi-view video in real-time,” says Haitham Hassanieh, an assistant professor of electrical and computer engineering at the University of Illinois at Urbana Champaigna who was not involved in the research. “This requires sustaining data rates of more than 6 Gbps while the user is moving and turning, which cannot be achieved by any of today’s systems.”

Since VR platforms have to work in real-time, systems also can’t use compression to accommodate lower data rates. This has led companies to make some pretty awkward attempts at untethered VR, like stuffing the equivalent of a full PC in your backpack.

The CSAIL team instead turned to mmWaves, which have promising applications for everything from high-speed Internet to cancer diagnosis. These high-frequency waves have one major downside, which is that they don’t work well with obstacles or reflections. If you want mmWaves to deliver constant connectivity for your VR game, you would need to always have a line of sight between transmitter and receiver. (The signal can be blocked even by just briefly moving your hand in front of the headset.)

To overcome this challenge, the team developed MoVR to act as a programmable mirror that detects the direction of the incoming mmWave signal and reconfigures itself to reflect it toward the receiver on the headset. MoVR can learn the correct signal direction to within two degrees, allowing it to correctly configure its angles.

“With a traditional mirror, light reflects off the mirror at the same angle as it arrives,” says Abari. “But with MoVR, angles can be specifically programmed so that the mirror receives the signal from the mmWave transmitter and reflects it towards the headset, regardless of its actual direction.”

Each MoVR device consists of two directional antennas that are each less than half the size of a credit card. The antennas use what are called “phased arrays” in order to focus signals into narrow beams that can be electronically steered at a timescale of microseconds.

Abari says that future versions of MoVR’s hardware could be as small as a smartphone, allowing for users to put several devices in a single room. This would enable multiple people to play a game at the same time without blocking each others’ signals.

Information extraction — or automatically classifying data items stored as plain text — is a major topic of artificial-intelligence research. Image: MIT News

Of the vast wealth of information unlocked by the Internet, most is plain text. The data necessary to answer myriad questions — about, say, the correlations between the industrial use of certain chemicals and incidents of disease, or between patterns of news coverage and voter-poll results — may all be online. But extracting it from plain text and organizing it for quantitative analysis may be prohibitively time-consuming.

Information extraction — or automatically classifying data items stored as plain text — is thus a major topic of artificial-intelligence research. Last week, at the Association for Computational Linguistics’ Conference on Empirical Methods on Natural Language Processing, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory won a best-paper award for a new approach to information extraction that turns conventional machine learning on its head.

Most machine-learning systems work by combing through training examples and looking for patterns that correspond to classifications provided by human annotators. For instance, humans might label parts of speech in a set of texts, and the machine-learning system will try to identify patterns that resolve ambiguities — for instance, when “her” is a direct object and when it’s an adjective.

Typically, computer scientists will try to feed their machine-learning systems as much training data as possible. That generally increases the chances that a system will be able to handle difficult problems.

In their new paper, by contrast, the MIT researchers train their system on scanty data — because in the scenario they’re investigating, that’s usually all that’s available. But then they find the limited information an easy problem to solve.

“In information extraction, traditionally, in natural-language processing, you are given an article and you need to do whatever it takes to extract correctly from this article,” says Regina Barzilay, the Delta Electronics Professor of Electrical Engineering and Computer Science and senior author on the new paper. “That’s very different from what you or I would do. When you’re reading an article that you can’t understand, you’re going to go on the web and find one that you can understand.”

Confidence boost

Essentially, the researchers’ new system does the same thing. A machine-learning system will generally assign each of its classifications a confidence score, which is a measure of the statistical likelihood that the classification is correct, given the patterns discerned in the training data. With the researchers’ new system, if the confidence score is too low, the system automatically generates a web search query designed to pull up texts likely to contain the data it’s trying to extract.

It then attempts to extract the relevant data from one of the new texts and reconciles the results with those of its initial extraction. If the confidence score remains too low, it moves on to the next text pulled up by the search string, and so on.

“The base extractor isn’t changing,” says Adam Yala, a graduate student in the MIT Department of Electrical Engineering and Computer Science (EECS) and one of the coauthors on the new paper. “You’re going to find articles that are easier for that extractor to understand. So you have something that’s a very weak extractor, and you just find data that fits it automatically from the web.” Joining Yala and Barzilay on the paper is first author Karthik Narasimhan, also a graduate student in EECS.

Remarkably, every decision the system makes is the result of machine learning. The system learns how to generate search queries, gauge the likelihood that a new text is relevant to its extraction task, and determine the best strategy for fusing the results of multiple attempts at extraction.

Just the facts

In experiments, the researchers applied their system to two extraction tasks. One was the collection of data on mass shootings in the U.S., which is an essential resource for any epidemiological study of the effects of gun-control measures. The other was the collection of similar data on instances of food contamination. The system was trained separately for each task.

In the first case — the database of mass shootings — the system was asked to extract the name of the shooter, the location of the shooting, the number of people wounded, and the number of people killed. In the food-contamination case, it extracted food type, type of contaminant, and location. In each case, the system was trained on about 300 documents.

From those documents, it learned clusters of search terms that tended to be associated with the data items it was trying to extract. For instance, the names of mass shooters were correlated with terms like “police,” “identified,” “arrested,” and “charged.” During training, for each article the system was asked to analyze, it pulled up, on average, another nine or 10 news articles from the web.

The researchers compared their system’s performance to that of several extractors trained using more conventional machine-learning techniques. For every data item extracted in both tasks, the new system outperformed its predecessors, usually by about 10 percent.

“One of the difficulties of natural language is that you can express the same information in many, many different ways, and capturing all that variation is one of the challenges of building a comprehensive model,” says Chris Callison-Burch, an assistant professor of computer and information science at the University of Pennsylvania. “[Barzilay and her colleagues] have this super-clever part of the model that goes out and queries for more information that might result in something that’s simpler for it to process. It’s clever and well-executed.”

Callison-Burch’s group is using a combination of natural-language processing and human review to build a database of information on gun violence, much like the one that the MIT researchers’ system was trained to produce. “We’ve crawled millions and millions of news articles, and then we pick out ones that the text classifier thinks are related to gun violence, and then we have humans start doing information extraction manually,” he says. “Having a model like Regina’s that would allow us to predict whether or not this article corresponded to one that we’ve already annotated would be a huge time savings. It’s something that I’d be very excited to do in the future.”

Read the paper here.

An autonomous mobility scooter and related software were designed by researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), the National University of Singapore, and the Singapore-MIT Alliance for Research and Technology (SMART). Courtesy of the Autonomous Vehicle Team of the SMART Future of Urban Mobility Project

By: Larry Hardesty

At MIT’s 2016 Open House last spring, more than 100 visitors took rides on an autonomous mobility scooter in a trial of software designed by researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), the National University of Singapore, and the Singapore-MIT Alliance for Research and Technology (SMART).

The researchers had previously used the same sensor configuration and software in trials of autonomous cars and golf carts, so the new trial completes the demonstration of a comprehensive autonomous mobility system. A mobility-impaired user could, in principle, use a scooter to get down the hall and through the lobby of an apartment building, take a golf cart across the building’s parking lot, and pick up an autonomous car on the public roads.

The new trial establishes that the researchers’ control algorithms work indoors as well as out. “We were testing them in tighter spaces,” says Scott Pendleton, a graduate student in mechanical engineering at the National University of Singapore (NUS) and a research fellow at SMART. “One of the spaces that we tested in was the Infinite Corridor of MIT, which is a very difficult localization problem, being a long corridor without very many distinctive features. You can lose your place along the corridor. But our algorithms proved to work very well in this new environment.”

The researchers’ system includes several layers of software: low-level control algorithms that enable a vehicle to respond immediately to changes in its environment, such as a pedestrian darting across its path; route-planning algorithms; localization algorithms that the vehicle uses to determine its location on a map; map-building algorithms that it uses to construct the map in the first place; a scheduling algorithm that allocates fleet resources; and an online booking system that allows users to schedule rides.

https://www.youtube.com/watch?v=h7ehanUmDhQ

Uniformity

Using the same control algorithms for all types of vehicles — scooters, golf carts, and city cars — has several advantages. One is that it becomes much more practical to perform reliable analyses of the system’s overall performance.

“If you have a uniform system where all the algorithms are the same, the complexity is much lower than if you have a heterogeneous system where each vehicle does something different,” says Daniela Rus, the Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science at MIT and one of the project’s leaders. “That’s useful for verifying that this multilayer complexity is correct.”

Furthermore, with software uniformity, information that one vehicle acquires can easily be transferred to another. Before the scooter was shipped to MIT, for instance, it was tested in Singapore, where it used maps that had been created by the autonomous golf cart.

Similarly, says Marcelo Ang, an associate professor of mechanical engineering at NUS who co-leads the project with Rus, in ongoing work the researchers are equipping their vehicles with machine-learning systems, so that interactions with the environment will improve the performance of their navigation and control algorithms. “Once you have a better driver, you can easily transplant that to another vehicle,” says Ang. “That’s the same across different platforms.”

Finally, software uniformity means that the scheduling algorithm has more flexibility in its allocation of system resources. If an autonomous golf cart isn’t available to take a user across a public park, a scooter could fill in; if a city car isn’t available for a short trip on back roads, a golf cart might be.

“I can see its usefulness in large indoor shopping malls and amusement parks to take [mobility-impaired] people from one spot to another,” says Dan Ding, an associate professor of rehabilitation science and technology at the University of Pittsburgh, about the system.

Changing perceptions

The scooter trial at MIT also demonstrated the ease with which the researchers could deploy their modular hardware and software system in a new context. “It’s extraordinary to me, because it’s a project that the team conducted in about two months,” Rus says. MIT’s Open House was at the end of April, and “the scooter didn’t exist on February 1st,” Rus says.

The researchers described the design of the scooter system and the results of the trial in a paper they presented last week at the IEEE International Conference on Intelligent Transportation Systems. Joining Rus, Pendleton, and Ang on the paper are You Hong Eng, who leads the SMART autonomous-vehicle project, and four other researchers from both NUS and SMART.

The paper also reports the results of a short user survey that the researchers conducted during the trial. Before riding the scooter, users were asked how safe they considered autonomous vehicles to be, on a scale from one to five; after their rides, they were asked the same question again. Experience with the scooter brought the average safety score up, from 3.5 to 4.6.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have devised a way to train neural networks so that they provide not only predictions and classifications but rationales for their decisions. Illustration: Christine Daniloff/MIT

In recent years, the best-performing systems in artificial-intelligence research have come courtesy of neural networks, which look for patterns in training data that yield useful predictions or classifications. A neural net might, for instance, be trained to recognize certain objects in digital images or to infer the topics of texts.

But neural nets are black boxes. After training, a network may be very good at classifying data, but even its creators will have no idea why. With visual data, it’s sometimes possible to automate experiments that determine which visual features a neural net is responding to. But text-processing systems tend to be more opaque.

At the Association for Computational Linguistics’ Conference on Empirical Methods in Natural Language Processing, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) will present a new way to train neural networks so that they provide not only predictions and classifications but rationales for their decisions.

“In real-world applications, sometimes people really want to know why the model makes the predictions it does,” says Tao Lei, an MIT graduate student in electrical engineering and computer science and first author on the new paper. “One major reason that doctors don’t trust machine-learning methods is that there’s no evidence.”

“It’s not only the medical domain,” adds Regina Barzilay, the Delta Electronics Professor of Electrical Engineering and Computer Science and Lei’s thesis advisor. “It’s in any domain where the cost of making the wrong prediction is very high. You need to justify why you did it.”

“There’s a broader aspect to this work, as well,” says Tommi Jaakkola, an MIT professor of electrical engineering and computer science and the third coauthor on the paper. “You may not want to just verify that the model is making the prediction in the right way; you might also want to exert some influence in terms of the types of predictions that it should make. How does a layperson communicate with a complex model that’s trained with algorithms that they know nothing about? They might be able to tell you about the rationale for a particular prediction. In that sense, it opens up a different way of communicating with the model.”

Virtual brains

Neural networks are so called because they mimic — approximately — the structure of the brain. They are composed of a large number of processing nodes that, like individual neurons, are capable of only very simple computations but are connected to each other in dense networks.

In a process referred to as “deep learning,” training data is fed to a network’s input nodes, which modify it and feed it to other nodes, which modify it and feed it to still other nodes, and so on. The values stored in the network’s output nodes are then correlated with the classification category that the network is trying to learn — such as the objects in an image, or the topic of an essay.

Over the course of the network’s training, the operations performed by the individual nodes are continuously modified to yield consistently good results across the whole set of training examples. By the end of the process, the computer scientists who programmed the network often have no idea what the nodes’ settings are. Even if they do, it can be very hard to translate that low-level information back into an intelligible description of the system’s decision-making process.

In the new paper, Lei, Barzilay, and Jaakkola specifically address neural nets trained on textual data. To enable interpretation of a neural net’s decisions, the CSAIL researchers divide the net into two modules. The first module extracts segments of text from the training data, and the segments are scored according to their length and their coherence: The shorter the segment, and the more of it that is drawn from strings of consecutive words, the higher its score.

The segments selected by the first module are then passed to the second module, which performs the prediction or classification task. The modules are trained together, and the goal of training is to maximize both the score of the extracted segments and the accuracy of prediction or classification.

One of the data sets on which the researchers tested their system is a group of reviews from a website where users evaluate different beers. The data set includes the raw text of the reviews and the corresponding ratings, using a five-star system, on each of three attributes: aroma, palate, and appearance.

What makes the data attractive to natural-language-processing researchers is that it’s also been annotated by hand, to indicate which sentences in the reviews correspond to which scores. For example, a review might consist of eight or nine sentences, and the annotator might have highlighted those that refer to the beer’s “tan-colored head about half an inch thick,” “signature Guinness smells,” and “lack of carbonation.” Each sentence is correlated with a different attribute rating.

Validation

As such, the data set provides an excellent test of the CSAIL researchers’ system. If the first module has extracted those three phrases, and the second module has correlated them with the correct ratings, then the system has identified the same basis for judgment that the human annotator did.

In experiments, the system’s agreement with the human annotations was 96 percent and 95 percent, respectively, for ratings of appearance and aroma, and 80 percent for the more nebulous concept of palate.

In the paper, the researchers also report testing their system on a database of free-form technical questions and answers, where the task is to determine whether a given question has been answered previously.

In unpublished work, they’ve applied it to thousands of pathology reports on breast biopsies, where it has learned to extract text explaining the bases for the pathologists’ diagnoses. They’re even using it to analyze mammograms, where the first module extracts sections of images rather than segments of text.

“There’s a lot of hype now — and rightly so — around deep learning, and specifically deep learning for natural-language processing,” says Byron Wallace, an assistant professor of computer and information science at Northeastern University. “But a big drawback for these models is that they’re often black boxes. Having a model that not only makes very accurate predictions but can also tell you why it’s making those predictions is a really important aim.”

“As it happens, we have a paper that’s similar in spirit being presented at the same conference,” Wallace adds. “I didn’t know at the time that Regina was working on this, and I actually think hers is better. In our approach, during the training process, while someone is telling us, for example, that a movie review is very positive, we assume that they’ll mark a sentence that gives you the rationale. In this way we train the deep-learning model to extract these rationales. But they don’t make this assumption, so their model works without using direct annotations with rationales, which is a very nice property.”

Electric motor, Rotterdam Ahoy Europort 2011. Source: Wikipedia Commons

Electric motors have been around since Thomas Davenport built the first functional model in 1834, and they have played a growing part in our lives ever since. Today, they continue to replace diesel and gas engines, as well as hydraulic cylinders, while evolving into new designs optimized for robots and other technologies.

“Something like 40 percent of electric power is used to drive motors, and that number will only grow,” says James Kirtley, a professor in MIT’s Department of Electrical Engineering and Computer Science and in MIT’s Research Laboratory of Electronics. “Electric motors are being used more widely in ships, airplanes, trains, and cars. We’re also seeing a lot more electric motors in robots.”

The ongoing transition from gas to electric is primarily driven by the need for more efficient devices that run with cleaner energy sources. Yet, electric motors also tend to be more responsive, and are more adaptable to new applications, especially in smaller devices.

As one of the world’s leading experts on electric motors, Kirtley’s philosophy is that one size does not fit all. “If you take into account the specific application, you can build a motor that is far better adapted than a general purpose motor,” says Kirtley. “For example, I’m working with someone who is making robots for medical assist, and he needs motors with very special characteristics.”

Many of the newer types of electric motors tend to be much smaller and run on less power than in the past. “I started working with electric motors 40 years ago designing big nuclear generators with 20-foot long rotors that were 6 feet in diameter and could produce a gigawatt and a half of power,” says Kirtley. “I’m now building motors on the order of 100 to 200 Watts for appliance motors, which are kinder and gentler to the system powering them. In the automotive industry, the average automobile has dozens of small motors for things like door locks, wiper blades, air conditioning, and seat positioners.”

Even excluding the separate field of MEMS (micro-electromechanical systems), which Kirtley is not directly involved in, electric motors are now shrinking to as small as the 1 W devices found in cell phones. A variety of innovative new compact motors are being developed all over MIT, says Kirtley, who points to an interesting variable reluctance motor for a prosthetic foot being designed at MIT’s Center for Art, Science and Technology.

Clean transportation is another source of innovation in electric motors. “In my lab we’re doing work with a small company in Cambridge that makes bicycle assist wheels,” says Kirtley. “The wheel stores some energy, and can react to pedaling forces to help it climb hills. These are entirely new applications.”

Improving microgrids with smart motors

Kirtley’s early involvement with power-plant generators led him to study electric power systems. His research into the subject culminated in 2010 with his Wiley-published book, “Electric Power Principles.” Lately, he’s been focusing on the customer-end of the system, where he is finding a role for electric motors in helping distribution systems adapt to intermittent, user-generated solar power.

“Electric power distribution systems are being stretched by the growing use of distributed renewable generation, such as rooftop solar,” says Kirtley. “Typically, electricity is transmitted from large power plants through extra high voltage wires, and the voltage is stepped down and delivered to customers. The problem with rooftop solar is that it looks to the power system like a reduction in load, but as solar cells become more widespread, homes will at times be able to produce more power than they’re consuming. So the power flows backward, which makes everything more complicated.”

Utilities are now working on smart microgrids that can adapt to distributed, multidirectional power. The greater flexibility is primarily enabled by smarter electronics, as well as efficient, distributed battery storage. Yet, microgrids still have a problem with even brief power outages, which can sometimes cause them to shut down.

“We’re thinking about the dynamics of motors connected to microgrids, and how you can improve their stability and make them work better,” Kirtley says. “We’re concerned about continuity of supply, which is especially important with microgrids supporting large server farms. You don’t want your system to be forced into an involuntary reboot simply because you had a glitch in the electric power supply. Electric motors can add more reliability.”

Electrifying the Cheetah robot

Many of the recent innovations in electric motors are found in robotics, which need smarter motors that can reliably deliver variable levels of power on demand for short periods. Electric motors can provide mobile robots with significantly longer battery life compared to traditional hydraulic systems. “Hydraulics are controlled primarily through throttle valves, so a lot of energy is wasted pumping and controlling the hydraulic fluid,” says Kirtley.

Kirtley has been working with Professor Jeffrey Lang on developing customized electric motors for Department of Mechanical Engineering colleague Sangbae Kim’s robotic cheetah, a running, jumping quadruped that has gained widespread publicity in recent years. The cheetah’s new motor is not only more efficient, but also more powerful, although only in short intervals.

“A secondary advantage of electric motors is responsiveness and control,” says Kirtley. “We can build a motor that can produce considerable torque in short spikes, even if we can’t necessarily produce the forces for a long period of time. It’s perhaps a little too powerful for the cheetah, which can now jump so high in the air, it probably wouldn’t survive the landing if they didn’t catch it.”

Electric motors transform ships

Most commercial ships still use diesel engines, while many naval vessels use gas turbine engines. Yet shipping is quickly moving toward electric propulsion, motivated primarily by efficiency, says Kirtley. Aside from nuclear-powered vessels, these tend to be hybrid systems in which diesel or gas generators drive an electric motor.

https://www.youtube.com/watch?v=_EV2qq2-Lck

“A traditional gear drive for ships has some very decided disadvantages,” he says. “For example, most destroyers in the U.S. Navy have gearboxes with very precise machining requirements, and are therefore expensive. They also require a fixed gear ratio between the engine and the water, so the prime mover is not operating near its peak fuel efficiency. Because of that, much of commercial shipping, and virtually all cruise ships, are now moving to electric propulsion, and even the U.S. Navy is starting to use it for its latest destroyer.”

The other problem with fuel-driven gear-drive engines is “the tyranny of the shaft line,” says Kirtley. “If you’re going to use a direct gear drive, the engine, gearbox, and propeller must line up very precisely, which often takes up valuable real estate within the body of the ship. With electric propulsion, we don’t have that problem. The engines can be placed anywhere where it’s convenient. In cruise ships, for example, the motors are place in a pod underneath the ship.”

Electric cars: adapting the engine to the road

The main difference between electric propulsion on ships vs. cars is related to torque requirements, says Kirtley. “On a ship, top speed defines the torque requirement,” he explains. “Automobile propulsion occurs across a speed range of about 10 to 1, with an engine that idles at 600 rpm capable of redlining at about 6,000 rpm. The best motors for cars are those that are adaptable to a very wide speed range.”

The variable speeds used in a car require that “the gearbox adapt the engine to the road,” says Kirtley. “You can generate an electric motor that can propel an automobile without a gear shift.”

In the past, Kirtley has consulted with Tesla Motors on its electric cars, and both agree that “the induction motor is the best for electric automobiles,” says Kirtley. Many other electric car manufacturers are still using permanent magnet motors, which he says are intrinsically less efficient in the wide speed and torque range required by car propulsion.

“For any given electric motor there is a tradeoff between excitation — making the operating magnetic field within the machine — and reaction, providing current to push on that exciting field inside the machine,” explains Kirtley. “In a permanent magnet machine that field is constant and cannot be adjusted, so a machine that is turning very fast but making relatively little torque is dropping a lot of power in losses in the machine’s magnetic iron. In an induction motor you can back off on the excitation to provide torque at the energetically optimal fashion. You can improve the drive efficiency of a car over a complete drive cycle by as much of a factor of two in fewer losses.”

Induction motors aren’t optimal for all applications, however, which brings Kirtley back to his main thesis: “In the development of motors for modern applications, it is most important to understand the totality of operational requirements,” he says. “That is key to making electric motors that will accomplish what they do best: provide motion in a responsive and efficient fashion.”

Photo: Gretchen Ertl

Sarah Hensley is preparing an astronaut named Valkyrie for a mission to Mars. It is 6 feet tall, weighs 300 pounds, and is equipped with an extended chest cavity that makes it look distinctly female. Hensley spends much of her time this semester analyzing the movements of one of Valkyrie’s arms.

As a fourth-year electrical engineering student at MIT, Hensley is working with a team of researchers to prepare Valkyrie, a humanoid robot also known as R5, for future space missions. As a teenager in New Jersey, Hensley loved to read in her downtime, particularly Isaac Asimov’s classic robot series. “I’m a huge science fiction nerd — and now I’m actually getting to work with a robot that’s real and not just in books. That’s like, wow.”

Hensley is studying Valkyrie for an advanced independent research program, or SuperUROP, as one of only three undergraduate students in the Robot Locomotion Group in MIT’s Computer Science and Artificial Intelligence Laboratory. Most of her colleagues are graduate-level researchers and postdocs with extensive experience working on complex humanoids. The group is led by professor of electrical engineering and computer science Russ Tedrake, who successfully programmed Valkyrie’s predecessor (named Atlas) to open doors, turn valves, drill holes, climb stairs, and drive a car for the DARPA Robotics Challenge in 2015.

Valkyrie has 28 torque-controlled joints, four body cameras, and more than 200 individual sensors, Hensley says. The robot can walk, bend its joints, and turn a door handle. “This is one of the most advanced robots in the world. And it’s 20 feet from my desk,” she adds.

That’s largely because Valkyrie has a long way to go before it leaves for Mars. MIT is one of three institutions, including Northeastern University and the University of Edinburgh, that NASA selected to develop software enabling the robot to perform space-related tasks — open airlock hatches, attach and remove power cables, repair equipment, and retrieve samples. Oh yeah, and get to its feet when it falls down.

Hensley, who started in the lab over the summer, is intrigued by the challenge of harmonizing the movements of such a highly complex system. “I am trying to solve a very tricky problem,” she says. She’s working out how best to control Valkyrie’s elbow movements by comparing two potential approaches. One uses a main controller to gather information from the various motor systems within the arm, and then uses that data to make accurate movement decisions. The other approach is decentralized, and leaves it to each motor system to decide and act on its own.

Hensley gets animated discussing the alternatives. “Is it better to have multiple decision makers with access to different information? Or is better to have one decision maker choosing all of the motor inputs?” she asks. Hensley has already been accepted into a master’s degree program in electrical engineering at MIT. She hopes to be able to continue her work on Valkyrie.

Every day, Hensley leaves Tau Epsilon Phi, her co-ed fraternity house in the Back Bay; walks across the Massachusetts Ave. bridge to the Stata Center; and plants herself in front of two large monitors in the robotics lab. She analyzes a wealth of scientific literature and writes code for computer simulations of the equations that move the robot’s arm. Sometimes she gets up for peppermint tea, or to peer around the corner of her cubicle at Valkyrie.

One thing is certain, says Hensley. Pop culture fears that machines may soon prove superior to humans are laughable. When Valkyrie is turned on and moves, Hensley says, it often “kind of shivers and falls down. One thing you realize working in this lab is that we are really far away from the robot apocalypse,” she quips. “Sometimes robots work, and sometimes they don’t. That’s our challenge.”

Robot from Computer Science and Artificial Intelligence Lab suggests where to move patients and who should do C-sections.

By: Adam Conner-Simons

Today’s robots are awkward co-workers because they are often unable to predict what humans need. In hospitals, robots are employed to perform simple tasks like delivering supplies and medications but they have to be explicitly told what to do.

A team from MIT’s Computer Science and Artificial Intelligence Lab (CSAIL) thinks that this will soon change, and that robots might be most effective by helping humans perform one of the most complex tasks of all: scheduling.

In a pair of new papers, CSAIL researchers demonstrate a robot that, by learning from human workers, can help assign and schedule tasks in fields ranging from medicine to the military.

In one paper, the team demonstrated a robot that assisted nurses in a labor ward, where it made recommendations on everything from where to move a patient to which nurse to assign to a C-section.

“The aim of the work was to develop artificial intelligence that can learn from people about how the labor and delivery unit works, so that robots can better anticipate how to be helpful or when to stay out of the way – and maybe even help by collaborating in making challenging decisions,” says MIT professor Julie Shah, the senior author on both papers.

In a second paper, the same system was put to the test in a videogame that simulates missile-defense scenarios. In the game, which was developed by Lincoln Laboratory researchers and involves using decoy missiles to ward off enemy attacks, the system even occasionally outperformed human experts at reducing both the number of missile attacks and the overall cost of decoys.

The labor-ward paper was presented at the recent Robotics: Science and Systems (RSS) Conference and was co-written by PhD student Matthew Gombolay, CSAIL postdocs Xi Jessie Yang and Brad Hayes, and Dr. Neel Shah and Toni Golen from Beth Israel Deaconess Medical Center, which is where the study took place.

The Navy-simulation paper is being presented at this week’s International Joint Conference on Artificial Intelligence (IJCAI), and was co-written by Gombolay and the  Lincoln Lab’s Reed Jensen, Jessica Stigile and Sung-Hyun Son.

Right on schedule

From visiting hospitals and factories, Shah and Gombolay found that a subset of workers are extremely strong schedulers, but can’t easily transfer that knowledge to colleagues.

“Figuring out what makes certain people good at this often seems like a mystery,” Gombolay says. “Being able to automate the task of learning from experts – and to then generalize it across industries – could help make many businesses run more efficiently.”

A particularly tough place for scheduling are hospitals. Labor wards’ head nurses have to try to predict when a woman will arrive in labor, how long labor will take, and which patients will become sick enough to require C-sections or other procedures.

They are deluged with an endless stream of challenging split-second decisions that include assigning nurses to patients, patients to beds, and technicians to surgeries. At Beth Israel, the head nurse has to coordinate 10 nurses, 20 patients and 20 rooms at the same time, meaning that the number of distinct scheduling possibilities adds up to a staggering 2 to the one millionth power, which is more than the number of atoms in the universe.

“We thought a complex environment like a labor ward would be a good place to try to automate scheduling and take this significant burden off of workers,” says Gombolay.

How it works

Like many AI systems, the team’s robot was trained via “learning from demonstration,” which involves observing humans’ performances of tasks. But Gombolay says that researchers have never been able to apply this technique to scheduling, because of the complexity of coordinating multiple actions that can be very dependent on each other.

To overcome this, the team trained its system to look at several actions that human schedulers make, and compare them to all the possible actions that are not made at each of those moments in time. From there, it developed a scheduling policy that can respond dynamically to new situations that it has not seen before.

“Rather than considering actions in isolation of each other, we crafted a model that understands why one action is better than the alternatives,” says Shah. “By considering all such comparisons, you can learn to recommend which action will be most helpful.”

The policy is “model-free,” meaning that the nurses do not have to train the robot by painstakingly ranking each possible action in each possible scenario by hand.

“You can put the robot on the labor floor, and, just by watching humans doing the different tasks, it will understand how to coordinate an efficient schedule,” says Gombolay.

The results

With this framework, the system – which the team has dubbed “apprenticeship scheduling” – can anticipate room assignments and suggest which nurses to assign to patients for C-sections and other procedures.

The approach was evaluated through experiments in which a robot provided decision-support to nurses and doctors as they made decisions on a labor floor.

Using the system on a Nao robot, nurses accepted the robot’s recommendations 90 percent of the time. The team also demonstrated that human subjects weren’t just blindly accepting advice – the robot delivered consciously bad feedback that was rejected at the same rate of 90 percent, showing that the system was trained to distinguish between good and bad recommendations.

Nurses had almost uniformly positive feedback about the robot. One said that it would “allow for a more even dispersion of workload,” while another said that it would be particularly helpful for “new nurses [who] may not understand the constraints and complexities of the role.”

“A great potential of this technology is that good solutions can be spread more quickly to many hospitals and workplaces,” says Dana Kulic, an associate professor of computer engineering at the University of Waterloo. “ For example, innovative improvements can be distributed rapidly from research hospitals to regional health centres.”

Shah says that the new techniques have many uses, from turning robots into better collaborators to helping train new nurses, but the goal is not to develop robots that fully make decisions on their own.

“These initial results show there is tremendous potential for machines to collaborate with us in rich ways that will enhance many sectors of the economy,” says Shah. “The awkward robots of the past will be replaced by valued team members.”

The RSS paper was supported by the National Science Foundation, CRICO Harvard Risk Management Foundation, and Aldebaran Robotics Inc. The IJCAI paper was supported the National Science Foundation and the U.S. Navy.

Read the paper here.