Automated production line in modern factory.

Every few weeks, Robohub will post a roundtable chat and discuss an engaging topic relating to robotics. In this edition, we looked at the controversial job loss of 60K jobs by Foxconn. Is this substantial job loss a preview to come with automation, or largely overblown hype? We strongly encourage our Robohub readers to chime in and be part of the conversation!

This chat features Sabine Hauert, Andra Keay, Kassie Perlongo, Yannis Erripis, John Payne, providing a range of perspectives from research, business, and the general public.

Automation cutting jobs. Robots stealing jobs. The loss of human workers. How many times have different iterations appeared in the media with these all too similar headlines? When news broke about Foxconn cutting 60,000 jobs from their factory, mass speculation spread like wildfire. Chinese companies say they are looking to combine people with robotics, and that it will improve product quality whilst also ensure labour supply is not a factor to production. So, is China looking at a long-term gain, with a short-term loss, so they can continue to make their own products on their mainland?

Once the offshore labour capital of the world, China has rising costs and a higher standard of living, making it difficult to find workers. President Xi Jinping called for an “industrial robot revolution” in 2014, looking to place automation’s role in raising productivity. Due to this, Chinese companies are turning to robots, announcing measures — such as subsidies and tax incentives — to encourage ‘more industrial automation and development of home-grown robotics.’ By the end of this year, China will overtake Japan to be the world’s biggest operator of industrial robots, according to the International Federation of Robotics (IFR), an industry lobby group.

But back to Foxconn. After the news broke, they released a statement: “We are applying robotics engineering and other innovative manufacturing technologies to replace repetitive tasks previously done by employees, and through training, also enabling our employees to focus on higher value-added elements in the manufacturing process, such as research and development, process control and quality control. We will continue to harness automation and manpower in our manufacturing operations, and we expect to maintain our significant workforce in China.”

The company still employs more than 1.2 million people. If businesses grow more efficient and competitive due to robotics, are robots in some ways helping to prevent businesses from shutting down, with potentially more people losing jobs? Some tasks currently done by humans can be automated at low-cost while others, such as fine manipulation or visual inspection, will require sophisticated hardware and software to mechanise.

The roundtable began by discussing how robotics alleviates risk with people working in dirty, dangerous conditions. And also how this plays into the government’s plan to infuse capital into the mainland robotics industry over the next few years.

“The increase in automation is partly due to the increase in labour costs, but it is also in response to things that happened, like the factory explosion due to unsafe working conditions, killing people. The government pledged 2 billion yuan in subsidies for companies to install industrial robotics and production lines,” said Andra.

“I remember reading exposés about terrible working conditions. It’s not necessarily a bad thing to be introducing robots,” said Andra. “It may be ‘viewed’ as threatening, but it can be beneficial. So instead of people being pushed into horrible working conditions, robots will take care of it,” said Andra.

((As an aside, Foxconn echoes similar concerns, with Day Chia-Peng, general manager of Foxconn’s automation technology development committee, saying that the company was motivated to focus on this area due to safety concerns and manpower shortages in recent years.))

“There are always going to be people working in some of the most dangerous jobs in society, so there’s a question of whether robotics can improve the quality of some of the worst jobs for these people,” Andra continued.

“People often forget how difficult it is to make a robot. The goal is not to make robots that replace humans, but robots that work alongside humans, performing specific tasks. There is a large push to develop collaborative robots (cobots) that are easy to program and deploy by workers. This will allow the workers to focus on more fulfilling high-level tasks that require human-level cognition, and the robots, to focus on specific repetitive tasks,” said Sabine.

The roundtable also discussed China’s strategy for investing and utilising industrial robotics. As noted by the Financial Times, their working age population is expected to fall from one billion people last year to 960 million in 2030, and 800 million by 2050. Automation can be a way to fill the labour gap.

“China has increased their robotics to 1 million robots in 2 years,” said Andra. “The number of robots of China is increasing. They have a middle class that is ageing, and they are expecting a higher standard of living and a higher quality of job. The cost of labour in China has increased, and manufacturing has shifted to other parts of the world.”

“What we’re seeing today, is that robots introduced in factories have contributed overall to an increase in productivity. Higher production leads to more jobs down the road in other sectors such as sales, customer support, and for the overall deployment, installation, maintenance, and management of high-tech factories. While it’s true that the nature of work might change, the net number of jobs may not necessary decrease. The question is how to retrain workers,” said Sabine.

The roundtable mentions that China has relied on a seemingly endless supply of cheap labour for decades to power its economic expansion. “Fast forward to today and the production of robots and China is now the largest consumer of industrial robots and is approaching becoming one of the largest producers,” said Andra.

“Is there also a problem that it’s all happening too quickly and not a gradual pace for people to find new jobs?” asked Kassie.

“I think the pace the automation is increasing, but we’re still talking about a fairly small density,” said Andra.

“One hope is that the jobs of tomorrow are more fulfilling. The nature of work has always evolved, driven in part by the introduction of technology. We don’t manually wash our clothes anymore, and I don’t see why we should be performing the same repetitive task on an assembly line either,” said Sabine.

“Job displacement doesn’t mean everyone needs to become an engineer, although we are very much in need for more people, especially women, in the STEM fields. There is also room for jobs that profoundly build on human interactions and creativity. In fact, I hope all professions will become more human,” said Sabine.

“Augmented reality will make training easier and human workers more effective,” said John. “By replacing shortages of health care professionals (not doctors) robots can help here — replacing tasks, not replacing jobs. I don’t think we have to leave people out of the meaningful economy. Eyes, brains, and hands are wonderful things. Add augmented reality and you have a powerful combination.”

Final thoughts…

In 2013, the most widely noted report on the subject came from Oxford University’s Carl Benedikt Frey and Michael Osborne, “The Future of Employment: How Susceptible Are Jobs to Computerisation?” However, a new report has also been making waves. The report from Organisation for Economic Co-operation and Development (OECD), “The Risk of Automation for Jobs in OECD Countries: A Comparative Analysis,” by Melanie Arntz, Terry Gregory, and Ulrich Zierahn, finds that there actually will be jobs for people, but still foresees difficulties, particularly for low-skilled workers. It’s also important to note that these reports do not specifically discuss robotics, but also software automation.

Also, just because something can be automated doesn’t mean it necessarily will be. Although we have current technology to purchase coffee by machine, there are still people lining up at coffee houses, like Starbucks, to order their favourite brew. The human aspect is more important.

Finally, the robotic component of this overhaul in China will be about more than just installing more robots in manufacturing plants; and perhaps, robots shouldn’t be seen as a cure-all solution for the nation’s labour shortages and economic slowdown.

Read some of our previous Robohub roundtables below:

• Robohub roundtable: Why is it so difficult to define ‘robot’?
• Robohub roundtable: Robotics and space exploration

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Over the years, there has been quite a lot of talk within the robotic research community
about a lack of standard experimental procedures. A EURON (European Robotics Network) workshop back in 2010 tried to tackle this issue head on. They said that although there are many good experimental scientists in the robotic community, there has not been uniformly good experimental work and reporting within the community as a whole.

One of the reasons for this might be because experimental design is not integrated into robotic research the way it is in, for example, medicine or psychology. This could partially be due to a lack of education. Experimental practices are usually learned from existing research papers, which can be of varying quality. This means that robotic researchers sometimes don’t even learn the basics of experimental design. This post will give you a starting point from where you can learn how to implement good robotic research experiments.

Of course, experimental design will be different depending on the type of robotics you are researching. An experiment with a UAV (Unmanned Aerial Vehicle) will be very different from an experiment with surgical robotics. However, there are still common aspects between the two. For now, we will focus on the aspects of experimental design that are similar across all experiments. In future posts, we will look further into specific aspects of experimentation.

Heilmeier’s Catechism: What Are You Trying To Do?

George H. Heilmeier was a pioneering engineer and inventor of the Liquid Crystal Display (LCD). He also introduced a useful set of questions for anyone proposing a research project or new product development. These are:

1. What are you trying to do? Articulate your objectives using absolutely no jargon.
2. How is it done today, and what are the limits of current practice?
3. What’s new in your approach and why do you think it will be successful?
4. Who cares? If you’re successful, what difference will it make? What are the risks and the payoffs?
5. How much will it cost? How long will it take? What are the midterm and final “exams” to check for success?

You can use these questions to structure a good introduction for a research paper. They are also a good way to get started with experimental design. If you can’t answer these questions about your experiment before you start then you will just be fishing around in the dark.

Is Robotics Suited to Experimental Design?

You might wonder whether the lack of strict experimental design procedures might be because they are not applicable to robotics. Other fields have stricter experimental design practices, including the pure sciences, medicine, psychology and the social sciences. You might wonder: Is robotics simply different?

This is quite a reasonable question to raise, especially as there is very little information about experimental design in robotics. However, the simple answer is that although, yes, robotics is quite a different field from many others, it could still be improved with stricter experimental design. Many people within the field agree and are trying to improve the experimental quality of robotic research by introducing benchmarks, standards, competitions and common software and hardware platforms.

I would recommend you read this paper from Willow Garage’s Leila Takayama, as a good introduction to why robotics needs more rigorous experiments and how we can achieve this.

The Five Components of a Well-Designed Experiment

Now that you can answer questions about the purpose of your experiment, you should define all of the key components. One way to look at experimental design is to split it into five components:

1. The Hypothesis or Hypotheses

The hypothesis is the big question that your experiment aims to answer. It’s so big, in fact, that we will devote another post to it later. Some examples of hypotheses are:

• Our inverse kinematic algorithm calculates the result of a 6-R robot in less time than three popular existing methods.
• Our robot gripper uses less electrical power than another similar gripper doing the same task.
• Human tests subjects respond more positively to Robot Design A than they do to Robot Design B.
• etc…

Quite simply, the hypothesis answers the question: What do you want to learn from this work?

It can be tempting to pick any old hypothesis and then start experimenting. However, it is best to sit down and formulate a good hypothesis. Here is one step-by-step guide on creating a hypothesis and here is another.

2. Variables

The variables of your study are those factors you are going to test in the experiment. The classic example is from medical drug research, where you have two groups of test subjects: a “drug group” and a “placebo group”. Apart from these variables, all other factors should either remain constant in the experiment or be removed some other way.

The variables that you will change in the experiment are known as independent variables. A classic example in robotics is to change the control algorithm and keep the rest of the experimental setup the same.

The variables which you will not change (but are affected by the independent variables) are known as dependent variables. One dependent variable for an algorithm might be, for example, computation time.

It can be difficult to decide which variables to choose, especially when you are comparing robotic platforms with many differences between them. However, as Takayama discusses, you can compare different systems if (and only if) you can control or eliminate any other variables of the system that might affect your variables. Is the speed of the robot important? Is the energy consumption important? Only compare the variables that make sense.

It is important to be objective about which factors are relevant to the hypothesis.

3. Measures or Metrics

When you have decided on your variables, you need to decide how you are actually going to measure those variables. For some variables, such as the computation speed of an algorithm, this might be a simple case of “time to solution”. However, for other experiments there may be several potential options for each variable.

For example, if you were comparing “user comfort” for various robot designs, you could ask the human subjects how the design made them feel. Alternatively, you could watch their actions when placed in the room with the robot. Both metrics might be valid but could produce very different results and present various challenges. Often, good research experiments will measure a variable in several different ways.

It is important to choose metrics that are not affected by confounding variables. This means variables which are neither those being tested, nor those which are controlled.

Confounding variables can come from many sources, because robots are such complex systems. Always try to think of other explanations for any result and don’t jump to conclusions. For example, imagine you set up an experiment to test the energy consumption of a robot to compare the same algorithm on two different electronic boards. Imagine that Board-A gave you a huge energy consumption; while Board-B used hardly any energy. A confounding variable might be the fact that Board-A turned the WiFi chip on by default; whereas Board-B turned the chip off by default. The energy-hungry WiFi chip would have nothing to do with your algorithm, but you could wrongly conclude that Board-B was more energy efficient.

Always question your results and seek out confounding variables.

4. Experimental Units

Now that we’ve picked our independent variable(s) and our dependent variable(s), and worked out how we are going to measure them, the next thing is to decide your “experimental units”, which can simply be thought of as: the number of things/subjects with which you’re going to test your system. In the case of algorithm research, this often means “data sets”. With human-based research, one experimental unit is usually comparable to one human subject, such as in human-robot interaction studies. For manipulation research, a robot setup may be tested on several different test setups.

In general, the more experimental units you test with, the better chance you have at producing a publishable research study. However, just because you have many results doesn’t necessarily mean that your experiment is well designed. Your results must be well aligned with the hypothesis, free of confounding variables and repeatable.

5. Assignment Method

The final step is to decide how to apply the variables to the experimental units. For example, imagine you have 1,000 data sets and you want to compare them with 10 different robotic grasping algorithms. You might ask questions like:

• Do you run all 10 algorithms on all 1,000 data sets?
• If not, how do you decide which data sets to try with each algorithm?
• How many times do you want to run each algorithm on each data set?
• etc…

The decisions you make for each of these five steps will make up the Experimental Design section of any research paper or report you write.

It is important to be able to defend the choices you make. Even if you don’t write everything in the paper (due to lack of space usually), you may be asked questions about it by others. Thinking through each stage of the experimental design before actually doing the experiments will make your research much stronger.

Foote explains the benefits of open source and how it helps make research and development more efficient when both researchers and start-ups can focus on their added value, rather than starting from scratch. This is an advantage for the whole robotics community, enabling it to move forward much faster. 

Foote also describes how developers can contribute directly to further developing the ROS standard, for the benefit of the robotics community. 

You can support Robots in Depth on Patreon. Check out all the Robots in Depth videos here.

Vecna’s robotic logistics solutions are a family of autonomous mobile robots, built to operate within human-centric environments such as hospitals. In this in-depth interview, Daniel Theobald, the company’s CTO, explains the importance of user-centred design and standardization. The latter is a primary focus in Mass Robotics, the innovation hub he founded.

Vecna was co-founded by Theobold in 1998 with a mission to empower humanity through transformative technologies. Without any outside investment, the company has grown to be a world leader in healthcare and robotics, also making significant contributions to green energy and sustainable agriculture; Vecna’s employees are paid to spend four hours a week doing community service.

Top of his class at MIT, Theobold studied electrical, mechanical, and software engineering, worked on Mars rovers and received fellowships from Ford, Hertz, and NSF. He spoke to Robohub’s Audrow Nash at the INNOROBO conference, which took place in July  in Lyon, France.

Audrow Nash: Tell me about Vecna.

Daniel Theobald: Vecna is a robotics and automation company that has been around for about 17 years now. It started with a group of engineers out of MIT and we’ve really focused on five areas of robotics: machine perception, which is the ability of robots to perceive and understand the environment; navigation, which is the ability of the robot to move around in the environment; manipulation, which is the ability of the robot to grasp, pick up and move objects; human-robot interaction, which is how robots and humans can collaborate and interact, and then the last area is high level optimization and planning, which is about how you can use many robots or robots and people to best accomplish some goals.

We bring all of these pieces together to try and use robots to provide real value to industry.

Audrow Nash: How did you pick these?

Daniel Theobald: We looked for the best way to break up the various aspects of robotics necessary to provide complete solutions to customers; we wanted to address all of the areas of robotics.

We looked for the best way to break up the various aspects of robotics necessary to provide complete solutions to customers

Over the years we’ve been fortunate to be able to do a significant amount of research in all these areas and our focus has always been: do interesting relevant research but make sure that we’re very connected to real customers and real problems, so that ultimately the technology can provide economic value. If you’re building robots just because they’re cool, you’re not going to be building them for very long because you won’t achieve enough income to sustain the process.

Audrow Nash: Can you tell me a bit about the robots that you’re building at Vecna?

Daniel Theobald: We have an entire range of robots. We’ve been focusing on what we call logistics. Anytime things need to move from one place to another, you can talk about that in terms of logistics. We have focused on having robots that can move objects ranging from less than 5 kilograms to 4,500 kilograms.

 In hospitals, the concept we talk about is from the loading dock to the bedside. Hospitals will get large shipments of things, like pallets coming off trucks, we have robots that can move pallets and smaller robots that break down the pallets and then make deliveries to different floors and wards or even directly to a patient.

Audrow Nash: I want to focus a bit on the roaming robots. Can you describe what they look like?

Daniel Theobald: Yes, we have a robot called QC Bot, QC stands for Quality Care. In many ways it looks like an average cart that you would see in a hospital. It has a computer screen on it but it drives around by itself. It has a cabinet, with some drawers on it. These are interesting because they don’t have any handles; they’re just flat drawers. The reason they don’t have handles is because, in many cases, you need them to be secure. A human can’t just come along and pull on a handle to open it; the robot has to actively open the drawer, once the person has identified him or herself. You wouldn’t want someone opening a drawer and taking medication out, particularly as these robots roam the hallways of the hospitals where there are patients and random people, it’s not necessarily a controlled environment.

Audrow Nash: Tell me about how this affects the nurses?

Daniel Theobald: There are a couple of very useful benefits to the nurses. A really big benefit is that they can spend their time caring for patients rather than being couriers. Right now, nurses spend, in some cases, the majority of their time walking around, pushing a cart or carrying things from point A to point B; when, in fact, nurses went to school to provide care to patients and pushing a cart is taking time away from that. When they have robots that can push a cart automatically, they don’t have to walk kilometers, they can spend more time actually caring for people so we get better service; we’re able to provide better care for the patients at a lower cost because you’re not using expensive resources to push a cart around.

We’re able to provide better care for the patients at a lower cost because you’re not using expensive resources to push a cart around

In some cases people may say, “Well, why not just use less expensive labor to push the cart?” But it’s not always possible to do that because many medicines are controlled substances. It’s nice to be able to lock the medication in the robot, it does the delivery and only nurses handle the medication. It can also be used for a number of other applications: everything from delivering food and sterilized surgical instruments to the surgical suits, linens and taking out the trash.

Audrow Nash: How is this perceived by, say, an elderly person or a small child or someone who’s never seen a robot before?

Daniel Theobald: When we first started working on this problem, the idea of deploying robots in public spaces was a completely new concept that needed a lot of work because – first and foremost – robots have to be safe. But it actually goes beyond that: there are a lot of social conventions that, as human beings, we take for granted. For instance, if you and I were walking down the hallway towards each other, maybe the convention is you walk on the left side I walk on the right side, or vice versa depending on where you’re from. Maybe you want to interact with me. So, how would you do that? You might say something or make eye contact, or you might put up your hand. These are all very subtle social cues that we have adopted as human beings. Ultimately, robots will need to understand some of these social cues too. For instance, if a robot is approaching someone and the person wants the robot to stop, the universal gesture is to put up a hand to get it to stop. If the robot isn’t able to understand those gestures then people aren’t going to have a favorable impression of robots.

What’s more, it’s important that all robots start to adopt and follow the same type of understanding in their interactions with human beings.

The other aspect, of course, is safety and ensuring that we put a lot of effort into making the robot safe. One of the things that Vecna’s robots do is, when they sense the environment, they don’t just sense whether or not there’s an obstacle. When I say obstacle that can mean a wall, a piece of furniture, a human being, a cart, a wheelchair; whatever. Treating a human being and a chair equally doesn’t make sense, although that’s what most robots do right now. It’s very important to be able to distinguish a moving object from a non-moving one and, ultimately, a human from a moving but non-human object, so that we can interact with those objects safely and appropriately.

One of the problems with some delivery robots is, because they need to be safe, they drive very slowly all the time, because they don’t know if there are humans around. The only safe assumption is there must be humans, so I need to drive very slowly. Our robots are able to sense when there are people around by visually detecting human beings and then reducing their speed to a socially appropriate and safe level, given their presence. But when there are no humans around, they’re able to move much faster. This allows the robot to move as fast as possible, safely, in any given situation.

Audrow Nash: Human-robot interaction is major design consideration. Can you just talk a bit more about that and how do we generally design robots with humans in mind?

What I really encourage people to do is the minimum viable product; get the crudest, most basic thing that somebody can start playing with, get it into their hands and start getting feedback as early as possible in the process

Daniel Theobald: There are a number of important principles there but the most important one is to get the robot into the hands of human beings as quickly as possible. Because engineers think about things differently than normal people and, even if they thought the same way, through the process of developing the robot you gain certain knowledge that obscures or makes it difficult for you to have a fresh perspective as if you were seeing the robot for the first time. It’s very important to bring in outside users who haven’t been exposed to the technology and see how they respond to it, because you learn a lot about what works, what’s intuitive, what makes sense for the humans and what you have to instruct them about. Anything you have to explain to them was probably not designed particularly well. Starting that feedback loop is very important.

What we’d like to say is that Vecnas fail early and fail often. Failure is a very important ingredient to success and, if they’re afraid of failing, people will often spend way too much time and money solving the wrong problem, because they want something that’s perfect and polished and finished before they show it to somebody for the first time. All that means is you’ve spent about 90% of your time chasing the wrong problems. What I really encourage people to do is the minimum viable product; get the crudest, most basic thing that somebody can start playing with, get it into their hands and start getting feedback as early as possible in the process. Because the problems you think are super important probably aren’t the problems that are important to them and, by watching them interact with the technology, you’ll very quickly learn what the real challenges are and be able to focus on those.

Audrow Nash: Can you talk a bit about establishing standards for different robots with different industries?

Daniel Theobald: The biggest obstacle to the adoption of our robots, or really any robots right now in buildings, is, what I broadly classify as, building integration. The robots need to be able to use the same elevators, the same doors and the same security systems as human users in those environments. Because it didn’t exist, at Vecna, we had to develop our own approach to integrating with elevators. You’d think that elevators would be able to communicate through the internet, but we’re not there yet. We’re working with elevator companies to establish those standards but, for now, we had to build our own elevator integration box.

It’s a piece of hardware that actually goes into the elevator closet, wires into the system and allows our robot, or any robot, to call an elevator to a floor, hold the door open, select which floor it wants to go to, etcetera. We like to share that with other companies and organizations because there’s no value added in having everybody build their own approach to integrating with elevators. If everybody comes up with their own solution then you’re only going to be able to have one type of robot in any building, because a different type of robot won’t be able to use the elevator integration solution that the other company provided.

It’s very important for us, as an industry, to understand that sharing and collaborating on these types of standards benefits everybody

It’s very important for us, as an industry, to understand that sharing and collaborating on these types of standards benefits everybody. It  reduces the friction to robotics adoption, meaning that it will be more cost effective for customers to buy robots, which means that there will be more robots and that benefits everybody.

That’s just one example, there are numerous examples where collaborating on standards is important. The computer industry would not be where it is today if every single computer operated differently. If you had to buy software only from the company, that you bought the hardware from, the internet wouldn’t exist; the ability to download software from multiple vendors and app stores, that wouldn’t exist.

The reason that these industries have flourished is because of collaboration. We can create a new device, we can publish a programming standard for it and then a company can go and write software; they can build the apps. It’s important to realize that robotics is 95% software. We tend to think of robots as hardware, as devices, but, at the end of the day, the value is in the software, it’s in the applications. If you look at the amount of time and cost spent, overall, on hardware in the long term, right now, that often represents the biggest investment, because we’re not actually out there developing a lot of different applications for the robots.

We tend to think of robots as hardware, as devices, but, at the end of the day, the value is in the software, it’s in the applications

Ultimately, the hardware will become a commodity and all of the time and money and effort will be spent on creating new applications for the platform, and that’s all going to be software. There’s a real opportunity for companies to start to focus on, not trying to reinvent the wheel, not building new hardware platforms, but building great new applications for existing hardware. That can accelerate your time to market as a robotics company. Don’t try and start from scratch, build on existing things to provide unique value that somebody else hasn’t already provided. That’s what will move the industry forward quickly.

Audrow Nash: Would you tell me a bit about Mass Robotics?

Daniel Theobald: We learned a number of these lessons over the years and some of the frustrations that we ran into encouraged us to start to help the industry to organize.  I started to reach out to numerous CEOs of robotics companies, particularly in the Cambridge Massachusetts area in the United States, near MIT and Harvard, because that’s where we were. As I started to meet these CEOs, it became even clearer that there was a missing piece. We weren’t a mature industry, we didn’t collaborate effectively, we weren’t sharing standards, we weren’t sharing technology, where appropriate. There was no organization in Massachusetts that was the right place to do that.

Collaboratively, we came up with the concept of Mass Robotics. Mass Robotics is really a play on mass adoption – where there are lots and lots of robots – although it works nicely for Massachusetts as well. The main idea is that we want mass market adoption of robots. One of the goals is trying to overcome, as an industry, the many hurdles that are preventing that from happening as quickly as possible.

We don’t need the 20-30 years that it took for the computer industry to become really mass market and consumer focused. We can take the lessons learned from that industry, benefit from them and accelerate the mass market adoption of robotics by paying attention to some of those issues. Again, a big one here is standards.

We don’t need the 20-30 years that it took for the computer industry to become really mass market and consumer focused. We can take the lessons learned from that industry, benefit from them and accelerate the mass market adoption of robotics

There was really no efficient mechanism for promoting and sharing standards in the robotics industries. There are a lot of organizations out there but each has a very specific focus and the main goal of Mass Robotics is to move the industry forward as a whole. That’s a lot about reducing risk, and reducing risk for investors is a really big one.

One of the issues is the shiny object problem, where technologists invent cool applications for robots, but it’s not really where the economic value is. One of the main focuses of Mass Robotics is bringing the real consumers, the real users of the robotic systems, to the table and they say, “Here’s what we could use help with from automation.”

The reality right now is that robotics has to borrow its technology from other industries. We’ve been fortunate because of the cell phone industry, and others that have helped move things like accelerometers, sensors and small cameras forward. But there is still a huge gap in terms of products that are specifically designed to be used on robots. Sensors are a great example. While the Kinect sensor was great – the structured light sensor that is used on home video games – because it had low cost, in many ways it was not quite good enough for a lot of the applications people would want.

The next option is about 10 times as expensive but is more precise than we really need. We’re establishing the robotics industry together as a group, going to the sensor manufacturers, the light and camera manufacturers, and saying, “Here’s the product we really need as an industry.” It can allow the robotics companies to focus on solving problems, rather than trying to hack around the limitations and sensors. It also allows prices to come down because, as more and more companies are able to start buying the same types of sensors and not having to reinvent the wheel, it allows them to focus on solving real problems and providing real value to humanity.

Audrow Nash: Thank you.

Daniel Theobald: My pleasure.

If you liked this article, you may also be interested in:

• Another robot company acquired: Vecna acquires VGo
• Robotic research: Are we applying the scientific method?
• Repeatable robotics: Standards and benchmarks from frontier science to applied technology
• Who will buy your bot? End user opportunities
• Evaluating the effectiveness of robot behaviors in human-robot interactions

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Once upon a time validation of robotic research was relatively straightforward. Let us assume, for example, that a researcher had published in a journal a novel adaptive control law with a numerical example on a two-link robot. Beyond the formal proof of convergence, he supplied to the reader the differential equations used to model the system, including the corresponding dynamic parameters (no more than 20 numbers), the eventual quantization and discretization of the controller, the solver details of the software used, and the sensor noise statistics. Not only the reviewers, thus, but also each single reader would have the possibility to re-run the numerical simulations in a half-day of work. The community would have the possibility to test, validate, generalize, and benchmark the algorithm.

Since then, robotics has changed, the machines are now much more complex in their kinematics, number of degrees of freedom, and are filled with several sensors. Also, giant steps have been made: the robots left the confined industrial cells to jump within unstructured environments, not only in the industry but also in the houses, the museums, the airports, and the post-disaster sites; they perform a number of exciting tasks such as exploration, maintenance, interaction with humans, search and rescue … wait, is it really so? Beyond specific outstanding experiences, beyond the claims of the constructors and lab’s directors, how many robots run, autonomously or semi-autonomously, in our daily lives? Not so many, to be honest. A few vacuum cleaning robots, this is all (Guizzo, 2015). While we have several noticeable robotic tools (parking assist systems, lane keeping assist systems, space systems), where are all the learning and adaptable robot protagonists of thousands of scientific publications in the last years? Our robots can avoid the predicted unpredicted events, but what about the unpredicted? The information required to validate the two-link example above is obviously not possible any more but why are we experiencing so large a gap between claimed and real robotics? Why has it been the case for several years now that the robotics revolution is regularly postponed to the next 10 years …?

The grand challenge for the robotics community is to discuss the way its research is conducted. 

The grand challenge for the robotics community is to discuss, from its foundations up, the way its research is conducted. It is a huge effort involving complex interactions among the institutions, the ministries, the funding agencies, and the individual researchers’ careers. Research is funded by selection of proposals, at each call more and more imaginative which, however, most of the time end with more or less disappointing demos. This process includes perforce to review the validation of the research process in a wide sense and, within this, the publishing process. The latter is becoming (apparently) faster and more selective with new ideas spread out and absorbed by other researchers in a very short time during which a paper placed in the hands of a reviewer or a debatable reject may be a dramatic event.

The previous claims are intentionally provocative, and so is the title of this article: are we (still) applying the scientific method in robotics? Let us frankly discuss this question.

The Oxford English Dictionary defines the scientific method as “a method or procedure that has characterized natural science since the 17th century, consisting in systematic observation, measurement, and experiment, and the formulation, testing, and modification of hypotheses.” The strict link existing with theory and experimentation is evident and further elaborated by, for instance, Karl Popper, who claimed that the criterion of the scientific status of a theory is its falsifiability, or refutability, or testability (Popper, 1963). Within the same framework, we find the sentence attributed to Einstein – “No amount of experimentation can ever prove me right; a single experiment can prove me wrong” – and the famous Wolfgang Pauli saying, of an argument that fails to be scientific because it cannot be falsified by experiment, “it is not only not right, it is not even wrong!” The term “falsifiable” does not mean something is made false, but rather that there exists an observation or experiment that might prove it to be false. Yes, all our scientific theories are perpetually subject to falsification. What is not scientific falls within the categories of non-science or pseudsocience, with slightly different meanings and a certain negative attitude toward the term pseudoscience. Social sciences are often considered as non-science because of the impossibility to exactly replicate the experiments. Economy and medicine, for example, are characterized by the uniqueness of each single event and thus the impossibility to, for example, implement a successful macroeconomic approach from one country to another.

The scientific method is not universally recognized as the sole medium for advancing knowledge.

The scientific method is not universally recognized as the sole medium for advancing knowledge, Paul Feyerabend’s opinion is that epistemological anarchism is probably the only paradigm that allowed science and knowledge to reach their current levels (Feyerabend, 1993). He criticized any standardized methodology or any possible authority for science. This opinion stands on various considerations, one of which is that a large number of all scientific discoveries are estimated to have been stumbled upon, rather than sought out. Several prestigious scientists consider themselves lucky; Louis Pasteur is credited with the saying “Luck favors the prepared mind” and, more recently, Nassim Nicholas Taleb, a former market analyst, developed the concept of “anti-fragility” as the capability to be ready to profit from chance.

Within a robotic framework, the considerations above bring us toward two aspects: interaction and replication. We need, as a research community, to exchange our experiences, our algorithms, the pieces of code since we need to replicate the experiments, to find bugs or theoretical flaws, and to benchmark. Let us thus have a closer look at the robotics case.

The robotic case

Conferences coffee-breaks are well-known critical sources of information. Among the various chats, few sentences collected are as pertinent here as: “I asked a Ph.D. student to implement the method of xyz but it never worked, we wasted 6 months and we do not know why.” Or “we implemented the method of xyz but it appears to work only on full moon nights. We could not prove it and we never published the results.” It is clear that the community is wasting resources; we know that only formal Comments on … can be published for major analytical flows but it is quite hard to find papers commenting on the implementation issues (robustness, sensitivity to parameters, etc.). On the other hand, each of us experienced the issue of having asked a junior to implement an algorithm and, at its failure, to remain with the unpleasant flavor of not having additional resources to investigate why and not trusting both the junior and the algorithm.

It is clear that the community is wasting resources; we know that only formal Comments on … can be published for major analytical flows but it is quite hard to find papers commenting on the implementation issues (robustness, sensitivity to parameters, etc.).

A different sentence came from a senior colleague involved in robotic applications of social sciences that, pretending not to need to follow the scientific method paradigm, finally claimed, “asking for formal proof is unfair, the world is not mathematics.” The former words, taken alone, might also be debated but, in the context of an escape-from-validation strategy, appear to be debatable. Sometimes, researchers from the social sciences raise as a flag a personal interpretation of the Gödel’s incompleteness theorems such that mathematics is once and for all kept out from discussion on their view of artificial life systems.

Another risk pushing the research away from the scientific method is the so called a posteriori reasoning, affecting a surprisingly large number of domains. Let us consider the classical example of the group of monkeys playing the stock market. Each morning, the monkeys forecast whether the market will rise or fall at the end of the day. Assuming a pure casual choice, it is reasonable to assume that they will split in half and thus half will be correct. The following day the same operation is repeated, still half of the monkeys will be correct; one-quarter of the initial set, moreover, will guess for two consecutive days. With a large enough set of monkeys, a certain number of them will guess for several days, or weeks, and will be considered as smart enough to interpret the stock market. The flaw is that only a posteriori we know which monkey will win. Similarly, the human body is considered as a perfect machine because it adapts perfectly to environmental conditions. This is one possibility but we cannot discern it from another possibility: the physical characteristics are spread totally casually among the species. The species which do not adapt to survive on this planet disappear and only a few of them, among which are listed humans, happily survive. A posteriori a perfect design …. This risk strongly affects the area of behavioral robotics. Let us assume that we implement via software some kind of behavior that pushes the robot towards a goal while pulling it far from obstacles. We run the experiment and then, a posteriori, whatever happens is interpreted in the framework of our solid behavioral structure (“the robot is attracted by the food,” “it is afraid of the obstacle,” “it is curious …”). Let us increment the number of robots and behaviors; isn’t it similar to what is considered “emergence behavior”? Arkin (1998) wrote: “Emergence is often invoked in an almost mystical sense regarding the capabilities of behavior-based systems. Emergent behavior implies a holistic capability where the sum is considerably greater than its parts …. The notion of emergence as a mystical phenomenon needs to be dispelled, but the concept in a well-defined sense can still be useful …. Coordination functions as defined in this chapter are algorithms and hence contain no surprise and possess no magical perspective.”

Words are important …

Etymology and epistemology

Etymology is the origin and meaning of words, epistemology studies the nature and scope of knowledge. One interesting issue concerning etymology and epistemology has been raised by Heisenberg (1958), when he addressed the problem of describing new concepts with existing words. Heisenberg was walking at the edge of human knowledge and comprehension and thus his communicability trouble was deep and severe.

Surprisingly, in robotics, it seems that a similar problem is traversing the community. The following is a list of terms used in recent scientific papers dealing with robots: intelligence, cognition, collective cognition, meta-cognition, smartness, consciousness, pre-consciousness, awareness, self-awareness, collective awareness, collective identity, collective memory, mood, emotion, and so forth (basically a two-combination of a huge set …).

Heisenberg claimed that a formal language, alone, would not have the same powerful communication capability of the human language.

When reading those words we all are influenced by the etymology, of course, even if they are supposed to represent new concepts, basically algorithms. Given their algorithmic nature one would expect a clear definition of these terms, both in terms of a linguistic description and in terms of any analytical language (mathematics or code-like reasoning). However, a large part of the community is using these terms by circular definitions, i.e., by defining one using a second that uses a third and so on while the final term is defined using the first again, and intentionally refusing to deal with any formal description. Heisenberg claimed that a formal language, alone, would not have the same powerful communication capability of the human language; he was talking about the Copenhagen theory, however, and not of algorithms to be implemented in C.

The proportional-integral-derivative (PID) controller dates 1922 (Minorsky, 1922), do we need pages of dialectic to come out with always the same three actions? What is the difference between a master/slave architecture with respect to a father/son one? What is the added value for the yet-another-controller based on the optimization of a yet-another-functional in which the interpretation is given a new, fancy, name?

Interesting enough, the misuse of words is not specific to Robotics, as an example in Economy, there is a debate about the possible malicious use of the term efficiency (Buchanan, 2013).

This poses a big ethical issue since most of the foundations of the scientific method are neglected. On the other hand, this naturally takes us to the next section dealing with the role of the media in robotics research.

The role of the media

There is a huge gap between the expectations of non-experts and what a robotic system can actually do  . Often, scientific journalists push for impressive titles and videos while totally neglecting the scientific and technological aspects. The language used by the journalists is different from the one used by the research community and is mainly aimed at achieving a larger impact rather than an adherence to reality. On the other hand, the funding agencies ask for a diffusion of the results to the general public in order to justify the use of public money. Multimedia interpretation of the results can be very informative, however, the temptation to use science-fiction terminology and to enrich the experimental video with post-processing editing is difficult to avoid, as it is clear by looking at the numerous attractive robotic cases and advertising-like projects’ spots.

Replicability

The main road to follow the scientific method is to allow the replicability of the experiments. Two aspects need to be considered separately, perception and control. For the perception case, the real-time issue is not a critical variable to replicate or benchmark a certain algorithm. Raw data exists that allows the community to compare their own works. One such example is the RawSeeds project funded by the European Community under the FP6 program [1] that permits the downloading of datasets composed by several sensors such as ultrasounds, inertial measurement units, vision and laser range finders, together with a given ground truth. Unfortunately, when dealing with control, the possibility to exactly replicate a certain experiment is much more difficult.

Human behavior is never the same in two consecutive tests, thus forbidding the possibility of an exact test replication.

To the best of our knowledge, in cases of control software being released to the community, and for obvious reasons, it is very rare that the hardware is provided together with the code. Even in that unlikely case, to replicate the experiment would involve a cost in terms of human resources that would discourage it. In recent years, the adoption of common hardware and software platforms has made it possible to reduce differences among laboratories and thus it will probably have a positive impact on replicability.

In case it is necessary to replicate a robotics test, both for perception or control, involving humans makes the issue still more delicate. Similarly, to the other social sciences, human behavior is never the same in two consecutive tests, thus forbidding the possibility of an exact test replication. The next natural step is to move toward massive experimental campaigns and statistical analysis, but this brings into play additional problems, for example, the increased costs involved in large-dimensional surveys.

Competitions

One possible approach is to evaluate from the results, no matter what path and terminology has been used to achieve them, within the same environmental situation, i.e., in the same moment with the same constraints. We are talking about robotic competitions. Currently, there are a dozen of them worldwide, with different structures, such as the DARPA challenge in the USA [2] or EuRoC [3], Eurathlon [4] and RockIn [5] in Europe or the Mohamed Bin Zayed International Robotics Challenge in Abu Dhabi, plus several others mainly involving mainly student teams.

If competitions are the tool to benchmark and evaluate and to allow interaction within the community, other captivating strategies should be envisaged.

The competitions differ from each other in several aspects, including the possible reward or eventual financial contribution to the participating teams. In all the competitions, there is always a trade-off between objective metrics and a peer review aspect. In EuRoC, for example, the Consortium is also in charge of hosting the hardware platforms and providing the teams with more or less basic robotic functionalities. In a way, the teams are leveled and the focus is on the algorithms and applications.

The critical aspects concerning competitions are the motivation that pushes the researchers to participate. When the value is solely appeal, the competition will probably attract teams with a commitment proportional to the reward. However, if competitions are the tool to benchmark and evaluate and to allow interaction within the community, other captivating strategies should be envisaged. Also, competitions embed an inherent risk of wasting effort in quick-and-dirty fixes, employed to match the realms and deadlines of the challenges, that could potentially have an impact on longer term research.

The Grand Challenge

Will the future bring an independent institution receiving funding to falsify the theories or to provide a standard for robotic algorithms? It is clear that research on research evaluation is a topic itself; it is dynamic and it changes with time. Even research in robotics, thus, is entering a new, challenging phase, for which we might need to refine our assessed evaluation convictions and even our beliefs  and try to adapt our work to new environmental conditions. It is probably the moment to enlarge the amount of people that can provide feedback on scientific publications by providing access to all and by including metrics for the impact of the papers after their publication, following a technical-only threshold within a solid validation discussion. Open access, open review, competitions and full disclosure policies can be viewed in the perspective of trying to level the playing field amongst competitors in the research adventure, with the ultimate conviction being that this will definitively provide new and exciting results.

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Footnotes

• http://www.rawseeds.org
• http://www.darpa.mil
• http://www.euroc-project.eu
• http://www.eurathlon.eu
• http://rockinrobotchallenge.eu

References

Arkin, R. (1998). Behavior-Based Robotics. Cambridge, MA: The MIT Press. Google Scholar

Buchanan, M. (2013). Forecast: What Physics, Meteorology, and the Natural. Sciences can Teach Us About Economics. Bloomsbury Publishing. Google Scholar

Feyerabend, P. (1993). Against Method. Verso.Google Scholar

Guizzo, E. (2015). “A robot in the family,” in IEEE Spectrum, 52, 28–58. doi: 10.1109/MSPEC.2015.6995630 CrossRef Full Text | Google Scholar

Heisenberg, W. (1958). Physics and Philosophy. New York, NY: Harper and Row. Google Scholar

Minorsky, N. (1922). Directional stability of automatically steered bodies. J. Am. Soc. Nav. Eng. 34, 280–309. doi:10.1111/j.1559-3584.1922.tb04958.x CrossRef Full Text | Google Scholar

Popper, K. (1963). Conjectures and Refutations: The Growth of Scientific Knowledge. New York: Routledge. Google Scholar

Original article:  http://journal.frontiersin.org/article/10.3389/frobt.2015.00013/full#

Link to pdf: Robotic research: are we applying the scientific method?

Citation: Antonelli G (2015) Robotic research: are we applying the scientific method? Front. Robot. AI 2:13. doi: 10.3389/frobt.2015.00013

Edited by: Torsten Kroeger, Google Inc., USA

Reviewed by: Vincent Creuze, University of Montpellier, France
Sylvain Calinon, Idiap Research Institute, Switzerland

Copyright: © 2015 Antonelli. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

Correspondence: Gianluca Antonelli, antonelli@unicas.it

If you liked this post, you may also be interested in:

• Repeatable robotics: Standards and benchmarks from frontier science to applied technology
• Towards standardized experiments in human robot interactions
• People and Robots: UC’s new multidisciplinary CITRIS initiative wants humans in the loop
• euRathlon Competition: A source of real data for robotics research
• NSF’s 40-year history supporting US robotics research

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Is robotics a science, or is it a technology, and how do we demonstrate/benchmark either? Scientific method exists to test science; standards and regulations exist to benchmark technologies. Researchers face the issue of repeatability at the experimental level. Product developers face the issue at the prototype level, and producers of consumer product face the issue at a commercial level. Robotics is an emerging technology – caught between the research lab and the real world – and frequently falls into the abyss between.

Publishing and peer review alone cannot keep pace with the full spectrum robotics developments, and not everything can be contained in a paper. Indeed, robotic systems entwine both the physical and the non-physical in ways that are difficult to catalogue entirely in journal format. For example, code can not be adequately cited or valued academically, yet is fundamental to robotics and AI research. At the same time, standards and regulatory frameworks push robotics in certain directions that might not be optimal. As open source code increasingly becomes the foundation for commercial products, the need to ensure repeatability, robustness and validation becomes essential, not just for commercially scaled producers, but for regulators as well.

At the same time, academic development requires flexibility, creativity, and unboundedness. How else could you mix chemistry and robotics, as in the recent paper on designing soft jumping robots published in Science? Breakthroughs require state-of-the-art knowhow that is sometimes difficult to reproduce outside the original lab setting, but shows potential for developments further down the line.

We know the robotics community understands this need, hence the call for research benchmarks, competitions, current ROS efforts, and new standards for collaborative robotics. But approaches are fragmented and neither widely established nor confirmed. We believe it is time to review these issues, and to examine the various solutions proposed to date. Over the next several months Robohub will be delving more deeply into how these issues are being played out between the research lab and the real world.

Interested in participating in the discussion? Leave your comments below, or send a draft proposal to info[at]robohub.org.

Relevant links:

http://clawar.org/towards-standarised-experiments-in-human-robot-interactions/

http://www.robot-standards.eu/

http://rockinrobotchallenge.eu/

http://www.ros.org/

With standardized means of assessing robot products and applications in terms of safety, performance, user experience, and ergonomics, the community would be able to produce comparable data. In the standardization community, such data is labeled “normative”, meaning that it has been formulated via wide consultation in an open and transparent manner. In this way, the results become widely acceptable, and can be exploited for the creation of international quality norms and standards, which in turn would mean measurable robot performances in terms of HRI.

Experts from academia, industry, and standardization have joined to launch the euRobotics AISBL topic group in “Standardization” that strives to develop standardized HRI experiments. Such experiments will allow the community to assess robotic solutions and compare data over different projects. Some of the topics the group is working on are safety, performance, user experience, and modularity of robots and robotic components.

The group has organized several workshops on standardized HRI experiments, and our next event is a workshop at this year’s IROS conference in Hamburg, Germany on September 28, 2015: “Towards Standardized Experiments in Human-Robot Interaction”.

We invite interested parties to participate and contribute in our effort to tackle HRI as a horizontal topic across all robotic domains.
For more information refer to the workshop website.