Andrew Vaziri – Robohub

ep.290: AI Powered Robotic Picking at Promat 2019, with Vince Martinelli, Jim Liefer, Pete Blair, Sean Davis and Erik Nieves

Andrew Vaziri — Tue, 09 Jul 2019 01:53:38 +0000

In this episode, join our interviewer Andrew Vaziri at Promat 2019, the largest expo for manufacturing and supply chain professionals in North and South America. Andrew interviews a handful of companies which provide warehouse fulfillment robots that can autonomous pick and place items. Our guests explain how advances in AI have made autonomous picking possible. They also talk about the unique technologies they use to stand out in a crowded field of competing products.

The guests in the order they appear are:
Vince Martinelli, Head of Product and Marketing, RightHand Robotics
Jim Liefer, CEO, Kindred
Pete Blair, VP of Marketing, Berkshire Grey
Sean Davis, Technical Product Manager, Osaro
Erik Nieves, CEO, Plus One Robotics

See video of each robot picking items on our YouTube page:

Links

ep.259: AI and the Law, with Nicolas Economou

Andrew Vaziri — Wed, 02 May 2018 14:58:28 +0000

The logo of the UAE’s annual World Government Summit which this year included a Global Governance of AI Roundtable for the first time.

In this episode Andrew Vaziri speaks with Nicolas Economou, CEO of the eDiscovery company H5 and co-founder and chair of the Science, Law and Society Initiative at The Future Society, a 501c3 think tank incubated at the Harvard Kennedy School of Government. Economou discusses how AI is applied in the legal system, as well as some of the key points from the recent “Global Governance of AI Roundtable”. The roundtable, hosted by the government of the UAE, brought together a diverse group of leaders from tech companies, governments, and academia to discuss the societal implications of AI.

Nicolas Economou

Nicolas Economou is the chief executive of H5 and was a pioneer in advocating the application of scientific methods to electronic discovery. He contributes actively to advancing dialogue on public policy challenges at the intersection of law, science, and technology. He is co-chair of the Law Committee of the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems; is chair of The Future Society’s Science, Law and Society Initiative and Senior Advisor to its Artificial Intelligence Initiative; and chaired the Law Committee of the Global Governance of AI Roundtable hosted in Dubai as part of the 2018 World Government Summit.

Links

ep.204: Satellite Assembly in Space, with John Lymer

Andrew Vaziri — Sat, 19 Mar 2016 17:54:09 +0000

In this episode, Andrew Vaziri speaks with John Lymer, Chief Architect of Robotics and Automation at SSL. They highlight key programs in space robotics from the 1980s through to SSL’s current program to robotically assemble satellites in space.

John Lymer
John Lymer is the Chief Architect of Robotics and Automation at SSL, a company known for its commercial communications satellites and spacecraft systems. He is an expert in automation solutions for space, terrestrial, and underwater operations, and has experience in a broad range of robotics missions including microsurgical and nuclear restoration systems.

Since joining SSL in 2013, John has been named Chief Architect for a NASA Tipping Point project called Dragonfly, which uses robotics for satellite assembly in space. He is also involved in NASA’s Asteroid Redirect Mission (ARM).

Links:

ep.202: Automation and Employment, with Michael Osborne

Andrew Vaziri — Sat, 20 Feb 2016 08:01:21 +0000

In this episode Andrew Vaziri speaks with Mike Osborne, Associate Professor in Machine Learning at the University of Oxford. They discuss how advancements in artificial intelligence may change the workforce, the economy and society based on findings from their study on the future of employment.Beyond the big headlines about automation and jobs, Mike discusses how the study was done, the emergence of new jobs, the need for improved education, and ideas for how we might reshape society to spread wealth that results from advances in technology.

Michael Osborne

Michael Osborne is the Dyson Associate Professor in Machine Learning, Official Fellow of Exeter College and Faculty Member of the Oxford-Man Institute of Quantitative Finance, all at the University of Oxford. He co-leads theMachine Learning Research Group, a sub-group of the Robotics Research Group in the Department of Engineering Science.

Links:

ep.201: The Airbus Shopfloor Challenge, with Curtis Carson

Andrew Vaziri — Sat, 06 Feb 2016 12:39:11 +0000

Airbus assembly floor. Source: Airbus Group

Transcript below

In this episode, Andrew Vaziri speaks with Curtis Carson, Head of Research and Technology in Industrial Strategy and Systems at Airbus. They speak about the Airbus Shopfloor Challenge to be held at ICRA 2016, and discuss the need for a new generation of industrial robots for aircraft manufacturing.

More information can be found on the competition website.

Curtis Carson

Curtis Carson is the Head of Research and Technology in the Industrial Strategy and Systems Centre of Competence at Airbus, a leading aircraft manufacturer headquartered in Toulouse, France. Holder of an engineering degree from Ryerson Polytechnic University in Canada, Curtis has extensive international experience in both the aerospace and automotive industries. Curtis’ current role includes developing new and innovative aircraft production processes through the application of emerging technologies. This role spans a host of technologies in the framework of the future factory, from augmented reality to collaborative robotics to digitalization to name a few examples.

Links:

Transcript

This transcript has been edited for clarity.

Andrew: Hello, welcome to Robots Podcast! Can you introduce yourself and describe what you do?

Curtis: I’m Curtis Carson and I’m responsible for the research and technology on the industrial system side with Airbus in France. My role encompasses everything you can imagine around the production of a commercial aircraft from jigs machinery equipment to robotics automation to new innovative techniques, enhancing and optimizing through IOT and connected objects.

Andrew: What is IOT?

Curtis: Internet of things, so connectedness and bridging multiple sensors and systems around our worker or automation on our shop floor.

Andrew: For people who aren’t familiar with Airbus, could you describe your products?

Curtis: There’s a high likelihood if you’ve travelled anywhere, it was one of our Airbus products. We develop commercial aircrafts from a smaller range, which we call single aisle aircraft A320, to the world’s largest passenger aircraft, the A380.

Andrew: What are the challenges in manufacturing aircraft?

Curtis: It might be hard to imagine from an outside perspective, but some of crafts have been developed years ago and we’ve been producing them for many, many years. Our new and innovative products, like our A350, have just gone to our first customers and the industrial system that we built behind to produce all this is quite large. But also expansive in terms of footprint and age and the diversity, is quite significant. We’re not building a factory every year and we’re not designing new aircraft each year. This makes it challenging to find ways to innovate.

Andrew: You said that the lifetime can be significant. What is the average time one aircraft design might be in production?

Curtis: It is quite typical for an aircraft design to be in production for 20, 30 years or more. The time frames are large to very long, the more successful the program the longer the production.

Andrew: How many aircraft does Airbus produce in a year?

Curtis: It was over 630+ aircraft produced last year. This might seem like a small number versus an automotive domain – where you’re producing thousands and thousands of cars – but this is some of the largest quantities of commercial aircrafts ever produced in a given period, or a given year.

Andrew: How many aircraft are currently in the backlog to be purchased from Airbus?

Curtis: I think it’s close to 7,000 aircrafts in backlog for production time-frame, don’t quote me on that. If you take the numbers per year, the backlog is over quite a number of years.

Andrew: To address the large number of aircraft currently in the backlog, what are some initiatives Airbus is undertaking for its manufacturing processes?

Curtis: We’re looking to innovate assembly process in the building of aircraft that are part of that building process and it might be difficult to see this without looking at really what’s inside a production system. As well as looking into automation to help people, or robotics and automation that we can use to assemble and fabricate parts of the aircraft.

We’re looking from across different facets of innovation: from smart tools, that know where they are in the aircraft; to autonomous robotics systems, that can manage and move themselves around in our production environment; to innovative technologies, like 3D printing and what it can bring to production efficiency and performance in the building of our aircraft. We’re looking across a large spectrum of technologies and see how we can effectively boost our productivity and our performance in building our aircraft.

Andrew: What robotics initiatives has Airbus undertaken in the past?

Curtis: In the area of robotics we’re using industrial robots and systems you may see if you were in an automotive environment for specific tasks – like drilling holes on a panel or a skin structure to assemble frames on the aircraft – we have big machines that we use today for building large scale composite parts up to the size of wings. We have a number of different automated systems we use. These are the typical industrial automations that we’re applying. With those typical industrial automations we can target a small percentage of the production activities to the large quantity of parts you can have in an aircraft, and the large diversity of the types of parts.

That’s what exists today. What we’re moving towards is where we can have more light-weight robotic systems that are flexible and agile. Systems that can move in and around our structures versus designing from the ground up a fixed and rigid system… instead you have a robot from within. Once you realize the size and the scale of the aircraft, you’ll see it’s not something that’s readily set up to bring the product to the robot or for the robot to do the work, and then move the product along. We have to be more agile and flexible in our building process.

Andrew: Airbus is sponsoring this year’s challenge at the International Conference on Robotics and Automation. Can you tell us about that?

Curtis: The competition we are proposing this year is drilling. Why would we target something like drilling? The simple answer is by looking at the production in Airbus in assembling the aircraft I mentioned earlier. We’re drilling over 120 million holes per year. Of those 120 million holes I would say we’re only able to automate a small percentage in terms of production tasks. There’s a high level of manual drilling we still do in the production of our aircraft today. Drilling is an interesting aspect. If we can combine what I said earlier – a lightweight and mobile robotic system – and combine with lightweight tools, then we have an opportunity to move into a new arena, and potential. We could implement an automated system without changing our infrastructure or our production lines to accommodate such a system.

A lot of these holes are drilled today by people. It’s not that a large industrial robot wouldn’t work doing that job, but it is not very mobile. And if you make the robot mobile, well it’s still heavy! If you’re drilling holes inside an aircraft structure barrel, it becomes difficult to put the robot inside without rebuilding your industrial infrastructure. This is what we’re interested to see: how can we push the boundaries of robotics capabilities today in this lightweight mobile module and solve these activities? Drilling is just the start. If we can do it with drilling, we can look to other aspects in production operations.

Andrew: In the competition itself, what are you asking teams to do?

Curtis: What we setup might look very simple, but the technical complexities come if we have a metal template. We’re asking them to produce a specific drilling pattern. In this drilling pattern we are asking to produce an automated system with bounding frame or restrictions with regards to size, mass, points for modularity. Then, looking at how quickly they can manipulate the system towards the task. If I have a pattern of holes to drill, how fast can they drill? What’s the level of quality? We’re very, very interested not only to have high speed, but quality.

We need precision. If we adjust the environment and the positions of these templates how can they master their robotic system to accommodate these different positions? How would they simulate the complexities of the geometry and the environment that we’re drilling on? I think for us and during the activities, we’re looking for a robotics system not doing just one task in a repeated mode, over and over, in one position. The types of automation we need, especially when it comes to drilling, are having high volume of repeated drilling tasks over a wide range of different positions.

Andrew: I understand that the robots are to be modular in their end effectors and weigh less than 100kg and you’ve mentioned reasons for both. I did want to ask: why are you asking people to provide open source software to actually control these robots?

Curtis: I would say the simple approach is there in the production context we have today and you see it in a number of industries. With this approach we can look at getting the best of the best. You mentioned end effector. Instead of thinking about a closed complete system that we would go buy and maybe the company we’re buying from is advanced in the modular robotic arm system. However, they don’t develop end effectors. The standard approach in most industries would mean we go buy from a supplier and package. It’s not that it doesn’t work, it can work. But what happens when better end effectors comes along? Or we have another company that’s able to do end effectors better than the company that is integrating the robot and end effectors of the 1^st system?

Well, it becomes difficult and complicated to figure out how to benefit from such an innovative change on the current system. We see the same thing in terms of the software, the control, the algorithms, whatever goes inside. Imagine also we have a huge IT infrastructure on our side across many different aspects of the production lifecycle and this is another level of complexity when it’s closed. We’re then in a mode where we develop each time we have a new system or solution we have to develop a very specific interface for such a solution.

There are a number of different reasons, but I would say 1^st and foremost we want to capture the best capability from the best in the world, wherever they are, for what they’re good at. It’s not common to find somebody that’s good at every single aspect being on all those aspects.

Andrew: It sounds like a large part of this is foreign partnerships. Are you looking more so towards academia or industry, of a mix of both?

Curtis: I would say if we’re really good at it, it will be both. Sometimes you can discover innovative things coming out of industry but you can discover interesting things coming out of academia. I don’t think that’s by surprise; academia and industry are linked in a number of different ways and means. A lot of the new businesses you see in today’s start-up world are from academia. What we’re looking for out there is how can we help stimulate towards academia and/or the outside world?

Where are our challenges, what are we looking to resolve and help that makes it feasible? This is also a good reason for the open approach. The more we are open in accessibility to the outside world, whether academia or businesses, the more opportunities and people that could potentially propose ideas that, perhaps, we didn’t think of in resolving. It creates a level playing field for small start-ups to be just as powerful and effective at proposing something valuable versus the big business and industry we’re used to working with today. It creates opportunity.

Andrew: Beyond just the drilling task that’s been selected for this competition, what are some places where you might see innovation being needed? Or a new approach beyond just making something that’s practical and useful in an industrial setting, but a whole new capability entirely?

Curtis: I guess it depends on what you mean by a whole new capability. What we look for today is predominantly driven by what we’ve already designed and producing. The aspect that we’re not deeply looking from this type of competition is okay. We’re talking about optimizing the current state of our product based upon how it designed. The design may have been a number of years ago, or 5 years ago. How do you combine those new capabilities and products, materials, or techniques? You use the capabilities in the design of the product that you’re doing and optimize both at the same time.

Andrew: As you think about your current production process, what are some examples of other low hanging fruit processes that could be adapted to use these technologies we’ve discussed?

Curtis: The minute we talk about drilling, the next thing is fastening. We have millions of fasteners we’re replacing. We do a lot of surface treatments or ceilings or applying ceilings joints. Painting requires a high demand in terms of manual effort and activity. If you’re painting, then have sanding and abrasion processes. Due to the size and the structure there are confined spaces and restricted access spaces. There are a number of non-ergonomic conditions the workers are in today. This is the whole domain and area of potential when we talk about robotics, especially light weight or new generations of robotics where you see lightweight in modular and adaptable types of systems. We have a huge playing field in terms of opportunity.

Andrew: How would you see the robots and the people interacting as they assemble an aircraft?

Curtis: The approach we focus on is that we don’t believe, at least today, in having a factory full of robots, and that’s it. We’re looking at worker-robot collaboration. We start from our worker, the person doing the work. How can we enable him to be more effective, better, efficient in his day to day job? It’s like a partner. Any difficult and non-human friendly tasks we can pass to the robot. We can ease the non-ergonomic conditions and reduce the repetitive and straining activities from workers. It’s a combination and collaboration.

We know that as robotics systems evolve they’ll be able to do a little bit more. At the same time, like what I mentioned at the beginning, it’s not just about a robot with a worker, it’s also how we can use the new connectivity of internet of things and connected objects and advanced communication to enable the worker to be more efficient and have the right information in a more dynamic way. Whether it’s through Google Glasses or wearable computing, or sensors in the manual drillers showing the quality of the hole they drilled. These types of automated systems allow a new feedback process for the operator to do the job the best he can.

Andrew: Could you give us a story or an illustration of what you imagine a worker in this future factory might to on a daily basis that would utilize these technologies, like, internet of things and the general ability to have more information?

Curtis: I would say the easiest way is just to give you a glimpse of what we’re preparing today. We have tests in preparation for one of our production environments where we will have electronically controlled smart tools. For example, a torqueing wrench for finishing the fastener or a manual drilling pistol for making the hole, with wearable devices. We will equip to three stations or more in one of our production lines and allow them to experience what tomorrow could look like.

The reason I like to focus on the aspect of tomorrow is because it’s not 5 years from now. We have the potential to incrementally put something in place today. What we’re preparing and doing now is the first production trials to see how it would be implemented in day to day for business for our workers.

Andrew: What do you see as the timeline? If tomorrow is the system you’ve just described what is 5 years from now, 10 years, or 30 years?

Curtis: It’s difficult for me to project that far ahead. But five years from I don’t think it’s too far-fetched to imagine that connected objects and types of automation will be a prevalent, common thing in any of our production facilities. It could be that tools you might see today like working from paper or drawings, you won’t see. I imagine you’d see a much cleaner environment. You’d see what we consider a high-tech environment in terms of tools, equipment, digital displays, progress tracking interfaces they’re using to prepare job flow of parts to the job that they’re doing.

I would imagine it’s hard to describe it in words unless you visualize and see what a production facility looks like today. I would imagine the proliferation of such devices is not on a linear scale. You would see it more in an expediential implementation and growth. This is one of the good reasons to consider the open approach. We must have a way, if we want to be fast. It has to be designed in a way that can allow this evolution to happen easily and quickly for robotics systems or connected objects and digital thread that we could use with our workers.

Andrew: In order to keep pace with that expediential rate of improvement, how serious is Airbus in investing in these opportunities? What would you guess would be the activity in the next few years?

Curtis: We’re serious about it, and the challenge is just a glimpse into why we’re motivated in this domain. The challenges are real, it’s a real challenge. It’s a real problem that we face today. It’s not something just made up to make a contest for the ICRA event. It’s a real potential application for our shuffler. When it comes to the digitization and the production efficiency and connectivity, we’re very serious about it and we have a number of initiatives running.

Andrew: On an economic scale people could also be interested, like how much has been spent or planned to be spent. Professors or companies look to the future and will think about possibly partnering with Airbus.

Curtis: I don’t know if I can put a figure to it, but I would say the best way to see how committed we are to it is to open up a dialogue; ask us! As us in your domain area! It really depends on the value we can potentially get, or bring from it.

We have significant portfolios of research and development activity running on platforms and cooperation with universities, technology centers, businesses. There is a significant amount of activity happening. I can’t really quote to the figures to be honest.

Andrew: ICRA is the 15 – 21 May in Stockholm, Sweden. When will you be accepting submissions to be part of this competition?

Curtis: Our submission window for ICRA is open today through March 15^th when we’ll close the submission process.

Andrew: For these teams, what kind of support is available from the competition?

Curtis: We’re offering, in certain circumstances, the possibility to any registered team to apply for help with travel, equipment, materials or tools. It depends on the potential and ideas by the teams. We judge it on the case-by-case basis with the teams who register. There is potential for support, but it requires registering and then talking with us.

Andrew: Where can people find registration forms and more information about the event?

Curtis: They can go directly to the ICRA 2016 homepage to find the link to the Airbus group webpage. The webpage provides details and process for registering, a general description about the contest, prizes and scoring. The best place is directly on the ICRA webpage.

Andrew: Thank you for joining us and putting together the Airbus Shop Floor Challenge! We look forward to seeing the results at this year’s ICRA!

Curtis: Thank you for having me! I’m excited to see the final results from our competitors.

ep.197: Multi-Agent Systems and Human-Swarm Interaction, with Magnus Egerstedt

Andrew Vaziri — Fri, 11 Dec 2015 16:51:31 +0000

Transcript below.

In this episode, Andrew Vaziri interviews Magnus Egerstedt, Professor at Georiga Tech, about his research in swarm robotics and multi-agent systems. They discuss privacy and security concerns, as well as research into interfaces designed to enable a single operator to control large swarms of robots.

The video below shows some of the strategies used by Magnus’ lab.

Magnus Egerstedt

Magnus Egerstedt is Schlumberger Professor in the School of Electrical and Computer Engineering at the Georgia Institute of Technology, where he serves as Associate Chair for Research. He received an MSc in Engineering Physics and a PhD in Applied Mathematics from the Royal Institute of Technology, Stockholm, Sweden, and a BSc in Philosophy from Stockholm University. He was then a Postdoctoral Scholar at Harvard University. Dr. Egerstedt is the director of the Georgia Robotics and Intelligent Systems Laboratory (GRITS Lab), a Fellow of the IEEE, and a recipient of a number of research and teaching awards, including the Ragazzini Award from A2C2.
Links:

Transcript

Andrew: Welcome to the Robots Podcast. Can you introduce yourself to our listeners and tell us about your research focus?

Magnus: I’m Magnus Egerstedt and I’m a swarm robotics professor from Georgia Tech. I’m interested in finding out how to get large teams of robots together, in particular when the individual robots may not be all that talented. The question: how can they make decisions so that beautiful, effective and exciting global patterns in behaviors emerge?

Andrew: Is this limited to physical robots, or are there other systems where this can be applied?

Magnus: I’m mainly interested in robotics, but this question of distributed decision making is really something. It could go from smart invertors on the smart grid to computational units and video games, to biological systems in that order. We have cells, we have birds, and we even have human societies – they’re all distributed systems as we walk through life with limited information. And yet, we’re able to build anthills and levers and human societies.

Andrew: Swarms of robots …. distributed systems … these are very complicated things. How do you generally go about trying to model those?

Magnus: I always start by pretending that I’m a somewhat stupid robot with very limited information, and start by asking: “What would I do? How would I think about this?” After I’ve tried to do that, step two is almost always look to nature, because nature is filled with examples of elegant solutions to our problems. Now, I’m not a biologist and I am by no means trying to mimic nature. I’m shamelessly stealing good ideas that social and behavioral biologists have had. Then, when I’m done stealing from nature, I sit down with pen and paper, work out the math, put it on the robots and find it doesn’t work, because it never works when you go from math to robots. You have to iterate on the actual robots a few times.

Andrew: When you’re working with these robots there are constraints that we don’t have in nature, in terms of the way they’re able to communicate with each other. Is it difficult to model that? What is the approach?

Magnus: When you actually start deploying large teams of robots there are some things that are just impossible to model.

Things happen that you didn’t foresee: robots bumping into each other saturating other communication channels, weird occlusions and the infrared sensors messing with each other. There are always things you cannot model. One of the great arts of swarm robotics is: “How do you go from rudimentary fables or models that are somewhat relevant to things that can be deployed on a large system?” This transition is a lot of times tricky, but it’s also deeply satisfying. Swarm robotics is exciting because in the last 10 or so years we’ve gone from spotty anecdotal understanding of how to structure these things, to large teams of robots on the ground, water, and air that can actually coordinate their actions and build interesting geometries.

We now understand certain things that we didn’t ten years ago. Now it’s common to see 25 robots coordinating actions on YouTube. Ten years ago this would have been unheard of. The basic functionality of how do we build these geometries or get these global behaviors out, we’ve kind of understood both on a practical and on a mathematical level. Now there are a huge number of issues we don’t understand. Let’s say for instance one of the robots is not playing by the right rules, or let’s say it’s broken – or even worse – it’s malicious. How do we identify this? How do we not let our robots be duped by these bad robots? There are a huge number of questions that we clearly don’t understand yet, but do we understand the basics of how to assemble these kinds of structures.

Andrew: I think that’s interesting you mentioned malicious actors potentially being in the system. I know you recently published a paper about differential privacy in multi-agent systems. What is differential privacy, and what is it good for?

Magnus: Let’s say you and I and other robots are trying to do something. But what we’re doing individually is really having our own cost of utility function that we’re trying to minimize. I may like to be where it’s light and you want to be positioned so that you’re not too close to flying robots, because you’re afraid of flying robots. We have to live in this space together, but we have our own objectives. Differential privacy is a way of hiding certain information by sprinkling noise onto what we do. For instance, I may move in a little bit of a noisy way so that you can’t figure out what I’m up to. We’re still respecting each other’s actions, but no one can infer what I am trying to do. The trick here is to insert the noise in a clever way so that we don’t end up doing erratic things. We’re achieving what we set out to do as individuals, but no one can tell what’s going on.

This idea of differential privacy came from the database literature. Let’s say we have a database of all our salaries. I add my salary to the database. If you know how many people are already in the database, and you know how it changed, then you can back out my salary. But if all of us sprinkle in a little bit of clever noise then certain things about the salaries – like what standard salary and so forth – you can still get from the database, but you cannot back out any individual pieces of information. It’s the same idea with swarm robotics. You sprinkle noise on the motions to hide your intentions. This is a flip-side from the malicious behavior where you want to be able to detect a robot isn’t playing by the rules. There are two sides of the same coin. One is to hide your intentions, and the other is you want to be able to figure out what bad robots are doing. Bad robot!

Andrew: I think a theme that we’ve all seen in our local big box stores is that everything wants to be smart these days: smart grid, smart homes. Is that something that we should maybe take with a grain of salt given the concerns of being able to peer into these systems and learn something or maybe we shouldn’t know?

Magnus: Fortunately yes, I think one thing in general when we talk about technology is it’s not going to be decoupled from us; we’re going to be users of technology. We need to – in my mind – build our systems so that it respects us as individuals and maintains our privacy. People are concerned about self-driving cars. We’re not going to wake up tomorrow and be sitting in self-driving cars. This is sneaking up on us whereby we get more autonomy in the cars, but people are engaged. People are almost always going to be in the mix when it comes to distributed robotics systems. Also distributed gadgets and the internet of things; people are going to be on the internet of things together with their things.

Andrew: Differential privacy could be one way to make the internet of things a little bit more private. You mentioned self-driving cars. I know that you’ve done research on the other side of the coin where malicious agents are actively trying to change or affect what the system does. It sounds like a plot from a spy movie. Could you tell us a little bit about controlling an enemy unmanned aerial vehicle by exploiting its own safety systems? How did that work?

Magnus: We were really interested in this question, as we’re about to see aerial drone delivery systems. Google and Amazon, if you believe the talk, are going to be delivering things by aerial drones. The question is then: “Is it possible to take over these systems and take over in such a way that the systems themselves do not know they’re being taken over?” This is not a classic cyber security question; it’s a cyber physical security question.

A lot of these systems including our aerial drones that we were playing on have safeguards to hear our reasonable physical behaviors. Basically it’s – here is the stuff that is within reason! And as long as we are within reason we’re going to be fine. Our question became: “Can we hide malicious intent inside the noise of what is considered a reasonable behavior, and yet, get the systems to do completely the wrong things.” I don’t want to say that we completely demonstrated this is the case, but we’ve certainly demonstrated examples where it is possible to hide and trick the system into thinking that it’s doing the right thing. We’re camouflaging the malicious trajectories inside reasonable and rational behavior.

Andrew: In this particular example how did you inject for that signal, what was the mechanism by which you influenced the aircraft?

Magnus: In my lab we’re not hackers. Since we designed that aircraft we had an error system or a safety system that detects that something was wrong. Then we transferred wirelessly over the network and injected new signals. If someone is really clever they could probably build something to detect this is going on within the cyber security side. That was not what we were interested in. What we were interested in was to see if it is possible to hide the malicious takeover within the cyber physical security system. This is similar to the idea behind Stocks net. Where nuclear centrifuges control systems were made to believe they were behaving right. Not because someone hacked in, but because you made the signals look plausible to the security systems.

We actually drew a lot of inspiration from … this is kind of amusing … from dragonflies. Dragonflies will move in such a way. Let’s say you’re a dragonfly; you’re a male dragonfly and I’m another male dragonfly and now we’re going to fight. I move in a way to have the sun in a straight line behind me, so now you can’t tell where I am. I go in making sure that no matter how you move I’m going to move, so that you only see the sun and then BAM! I’m going to get you. Similarly in control for aerial there’s a notion of an observable subspace which here are things that the trajectory of the system can do that’s hidden from the output. We exploited this by restricting emotion to these unobservable modes of the system.

Andrew: That’s very interesting. This adds a whole other layer. It’s not just cyber warfare, it’s not that you hacked; you changed the code that was running on the target. It’s exactly intact as it was designed by its creator. You’ve changed what it observes and what it thinks it observes. That seems like a very hard thing in general to defend against, to not only take into account the cyber aspects but also all the ways things could physically interact. Is there a principled way to make sure that systems aren’t vulnerable in this fashion?

Magnus: We have been looking into this question and at this point there isn’t a principled way but here are ways people have done it. We looked at what we actually called motion probes. The system every now and then does a somewhat strange manoeuvre, where it’s basically a signature manoeuvre. It could make a small sidestep, or the quad copter makes something that resembles a figure eight. This is unexpected; if the attacker isn’t expecting it then you can actually excite these previously unobservable modes and say, “Hey, something is actually not behaving the way it should be there!”

This is how we are thinking about it and it also goes back to this question of detecting not malicious takeovers, but malicious agents. If we are swarm robots, we’re supposed to do something. If I then make a figure eight I know what you’re supposed to do in response to that, if you are one of my people. If you don’t respond correctly I can detect that perhaps something is fishy here.

Andrew: I’d like to switch gears a little bit and ask about something that’s high risk, but in a different sense. I recall reading, maybe four years ago in a systems magazine. There’s a little blub about you and you wrote about your research philosophy and research style. You mentioned liking to keep a few high risk, high payoff topics on the horizon. I just wanted to ask, what does that mean? What does a high risk, high payoff topic look like to you?

Magnus: The wonderful thing about being a professor is that you’re largely free to explore things you find interesting. I have to pay the bills and, more importantly, make sure my graduate students don’t stave. I need to do things that are five years out which people care about. But I also have the freedom to think about things I truly am interested in that are way out there. If they come to fruition they’re going to be important, but if they don’t chances are they don’t. I always like to have a few of these questions that I think are fundamentally important and interesting down the line on the back burner.

Every now and then I return to them, so right now I have a few. One thing that I’m really fascinated by is this question of human swarm interactions. Let’s say you’re a person with a joystick surrounded by a million robot mosquitoes and you want them to do something. What are the right ways to engage with truly large teams of things? That’s one thing that I’m fascinated by right now. Another thing that drives me absolutely nuts is that people talk now about heterogeneous multi-robot system.

Andrew: What does that mean?

Magnus: It means different kinds of robots. In my lab, we have things on the ground and things in the air. But fundamentally, why is it good to be different? Why should I have fast and slow robots together? What is it that makes that better than just having a bunch of fast robots?

To me, this is an interesting question because we can understand what kinds of robots we should deploy. If you have an earthquake and you want to send out robots to figure out what’s going on in the rubble you can say the optimal thing to do is to have nine snakes robots, 15 aerial robots and 24 ground robots. And here’s what they should be doing. To me this is fascinating. I again look to nature like ecosystems which are highly diverse and consider, why is that? How come hummingbirds’ live next to sloths? Is there something fundamentally good about that?

Andrew: I recall from some years ago you mentioned one of these high risk areas was human swarm interaction. How has it worked out over the past few years there were moments where it seemed dire?

Magnus: First of all, human swarm interaction has become an expectable thing to spend your time on and there are other people engaged with it. I think we’ve understood things, if you have small teams of robots then an effective way of engaging with them is actually to grab hold of a leader and inject information through the leader. That works well up to maybe four or five robots, then it kind of collapses. The reason why is that people get bad at imagining what – let’s say I have ten robots around me and I grab one and I shake it. How does that propagate through the ten robots? We just lose track.

For example look at universities, militaries, big companies … we’re all hierarchically organized. I think the reason becomes we don’t know what happens if we have 250 people to manage, all at the same time. I looked at sheep herding and found you’re not controlling individual sheep. You’re pushing things at the boundary with herding dogs. Instead of controlling individuals you’re exerting forces on the outskirts of the swarm. It turns out we can do some things like that quite well. I have worked with swarming tractors, so the question becomes: What does the farmer do at the side-line of the field with an iPad? Maybe one way of engaging is not to engage with the tractors at all but to engage with the map and say, “Here are regions of interest. Now you tractors, go figure it out!”

Andrew: You did some research with a leader-follower system and in that case it was a haptic interface, could you tell us a little bit about that?

Magnus: Yeah, this goes back to this question of its very unclear for people what it means to control large teams of things. We needed some way of making this information concrete to a person.

We basically computed something that we called the manipulability, which is a measure of how easy it is to move this swarm in particular directions and this we could compute. Then we used that to generate forces. The idea is try to push the swarm in a direction in which it was hard to move it then we’re experiencing large forces. It turned out that this was fairly useful. We did user studies that people could solve tasks that were very hard to solve with only a joystick, but with a haptic joystick they could solve tasks more effectively. This goes back to the question of, what information you actually need about a large swarm. In this case it was just force. If you want them to do more elaborate things you probably need more information. But if in general you just want to move the swarm from point A to point B then we thought the findings in this study where haptic feedback serves a strong and positive purpose.

Andrew: Could you paint us a picture of what the experimental setup looked like? What were the robots like? What were the users looking at and what were they holding?

Magnus: The user was holding a phantom omni, the brand doesn’t matter. It was a haptic interface; it looks like a little pan connected to a ball. You’re moving it up, to the side, or in. You’re experiencing a force depending on what feedback that the little haptic force generating ball is giving you. That was what the user was holding. Then the users were in the lab and we had 10 robots. The robots were arranged in a particular formation. What the user could do is select which robot, he or she, wanted to use as the leader and then drag it in various directions.

You can imagine having a long and skinny formation and the leader robot is embedded inside it in the middle. If you move along the direction of the long and skinny thing then that’s going to be hard! You have to move all these things out of the way and you’re experiencing lots of forces. It has what’s called low manipulability. If you move it away from the long and skinny direction, then it’s easier to melt into a form or shape that follows the leader. The user was looking at the 10 robots, the user did not know how the robots were organized, and basically tried to get them to go from one point to another point while navigating a little bit of an obstacle course, so no robots were allowed to actually get stuck behind an obstacle.

Andrew: How did you assess if it was being effective if the person felt like it was a good interface or not?

Magnus: Yeah, this is always when you do user studies, how do you know if it’s good or not? We used three metrics. One, we just timed them. So how long does it take? Two, we looked at how much did the user have to work. We summed up the control signals that were sent to the swarm. And third, we asked them! Basically we asked: did you find this annoying? There are probably more scientific ways of asking that question, but that’s basically what it boils down to: what level of stress, engagement, or discomfort did you feel when you were doing this?

Andrew: A more advanced interface might be something you alluded to the farmer by the field with a tablet and algorithms that were suited to that. Could you tell us a little bit about that?

Magnus: The backdrop to this is rather interesting. The tractors now are really advanced pieces of machinery. They can largely drive themselves. Put in the GPS of the path and then they can, more or less, do this by themselves. Moreover, cornfields are really not that complicated they’re nice and straight. In terms of the types of environments you’re encountering they’re not that complicated. The manufacturing floor is way more complicated, or my daughter’s bedroom … way more complicated. The farm field is not that complicated. This tractor manufacturer came to me and said, “You know what we know now.” So how do we make these autonomous, which means we don’t need a driver in the tractor.

That means we can make the tractors smaller, and that’s good. Maybe because it has less environmental impact, you don’t have to go back and retail afterwards. These are all good reasons why small is better. Except that it’s not less effective. Instead of having one small tractor lets have ten small tractors. Then the question is but how do you actually engage with these? They literally came and asked, “What should the farmer do, we want to give the farmer an iPad, what should the farmer do?” That was the thing we got rather excited about.

Andrew: What did you end up making, what algorithm and what kind of user interface or display?

Magnus: We ended up going with something that’s fundamentally a good idea, which is that the farmer should actually not deal with the tractors. What the farmer should be dealing with are higher level tasks. In this case what we had is a map of the world and the farmer could outline things that were interesting, or things that were urgent. Or where are where there was lots of moisture, or not a lot of moisture. Things that basically described we used, we call density functions to describe areas of the interest. Then robots run what’s called a dynamic coverage algorithm. That means the robots disperse themselves in such a way that they cover areas of interest where we get the same amount of stuff we’re in charge of. Technically, this is called a voronoi partition. The robots computered a voronoi partition.

Then inside that partition they did their driving around. Then the farmer could do is move their finger around, and on the flight dynamically the robots reconfigured themselves. We took this and this now, which has nothing to do with tractors anymore. But we have this general way of painting event density on areas of interest and then have the robots dynamically respond to that in a distributed manner.

Andrew: Have you received any feedback from either farmers or researchers about some of these advancements?

Magnus: So I have! I did a photo shoot where I was wearing a weird baseball cap and rubber boots in the lab. Don’t make fun of me; I pretended to be a robot farmer. But seriously people are excited about it. The interesting thing about swarm robotics is that it’s up and coming and these kinds of environmental surveillance tasks may be farming or just trying to figure out what’s going on in an area. I had a NASA project where we were monitoring effects of climate change down at Antarctica. We actually never ended up going to Antarctica, unfortunately.

There, you deploy robots for fairly long periods of time and they have to just respond to what’s going on in the environment and this kind of way of painting the world at the user level with areas of interest is a very natural thing. Now if you’re flying two drones, that’s not how to do it, then you need to worry about individual aircraft. If you’re just interested in covering areas then this is a really effective way of doing it. We’ve had quite a lot of funding from the Air Force Office of Scientific Research to pursue this further because in a way it’s a very simple way of describing missions that robots can immediately execute.

Andrew: In closing, I wanted to ask you a philosophical question. Would it be fair to say that you’re a fan of soccer?

Magnus: Ha! Yes, it would be fair to say that I’m a fan of soccer. In fact as soon as we’re done with this I’m going to go put on my soccer gear and go play a soccer game for old men; you have to be over 40 to play in that league.

Andrew: You must be aware of robocup’s robotic soccer league. Currently, they’re pitting teams of robots against each other. Their goal is to field the team of robots to beat the world cup championship human team by 2050. I think that shows a high level of optimism and the future of multi-agents teams and in swarm robotics perhaps as well. What do you see the future as, how will we get there?

Magnus: Robots are already better than people at many things. Robots are really good at not getting bored. They’re really good at responding quickly to things. They’re really good at crunching big numbers. I think if you do a robotic 100 meter dash.

We’re not far away from having bipedal robots that are faster than Usain Bolt, the fastest runner in the world. As a soccer player if you look at the artistry of Messi we’re way far away from that. I think part of that is actually not an algorithmic question. Where we’re lagging right now a lot of times is that actuation and power supplies. The humanoids of today that are not tethered, they have to schlep very large battery packs with them and, if you’re lucky, 3 degrees of freedom foot. Messi’s foot has many more degrees of freedom and he’s small and agile.

Andrew: Perhaps the World Cup is megatronics challenge. What to you would be the swarm robotics coming of age moment where you say that’s the goal?

Magnus: That’s actually a really good question. A big part of what’s driving me is purely intellectual curiosity. I think there are deep scientific questions about how do you get truly emergent behaviors out from simple local rules.

Andrew: What do you mean by an emergent behavior?

Magnus: Simple emergent behavior is how do you send out robots to cover an area where all they’re doing is seeing very locally– what’s going on? A higher level question is that you have these little neurons and all they’re doing is sending spike trains around and, all of a sudden, you have conscious human beings. That’s a very high level and emergent behavior question. Once we understand that to me that’s a really interesting question and it has repercussions way beyond robotics. The big question is not, how do I get 200 robots to mow my lawn more effectively? Even though I do think we’re going to have teams of robotic lawn mowers, small little guys, which are more agile than these big robotic lawn mowers that are out there now.

But, scientifically understand, how do we make these distributed decisions? Along the way we’re going to have robots on Mars. We’re going to have swarms of robots on Mars. We’re going to have robot helpers in hospitals and on the manufacturing floor. These things are coming. Ultimately computers are useful tools, but they’re only useful in as much as we do useful things with them. I think people tend to think of robots as, “Oh my God, we’re going to have an army of terminators!” I hope not. I hope we’re going to use robots to make our lives better. And use them to find energy, food, and travel to remote locations around the globe. I hope that we, as people, will be able to use them as tools for good as opposed to tools for evil … like an army of terminators.

Andrew: A fascinating discussion, thank you for joining us!

Magnus: You bet, thank you for having me!

This audio interview was transcribed and edited for clarity.