Robohub.org
How to design a robotics experiment
The ability to design good scientific experiments is a key skill for any robotic researcher. Without a grasp of experimental design, even the best technological developments may never succeed in being published or turning into real products. In this post, we show why this is a vital skill for researchers and we show you how to implement good experimental design in 5 steps.
Over the years, there has been quite a lot of talk within the robotic research community
about a lack of standard experimental procedures. A EURON (European Robotics Network) workshop back in 2010 tried to tackle this issue head on. They said that although there are many good experimental scientists in the robotic community, there has not been uniformly good experimental work and reporting within the community as a whole.
One of the reasons for this might be because experimental design is not integrated into robotic research the way it is in, for example, medicine or psychology. This could partially be due to a lack of education. Experimental practices are usually learned from existing research papers, which can be of varying quality. This means that robotic researchers sometimes don’t even learn the basics of experimental design. This post will give you a starting point from where you can learn how to implement good robotic research experiments.
Of course, experimental design will be different depending on the type of robotics you are researching. An experiment with a UAV (Unmanned Aerial Vehicle) will be very different from an experiment with surgical robotics. However, there are still common aspects between the two. For now, we will focus on the aspects of experimental design that are similar across all experiments. In future posts, we will look further into specific aspects of experimentation.
Heilmeier’s Catechism: What Are You Trying To Do?
George H. Heilmeier was a pioneering engineer and inventor of the Liquid Crystal Display (LCD). He also introduced a useful set of questions for anyone proposing a research project or new product development. These are:
- What are you trying to do? Articulate your objectives using absolutely no jargon.
- How is it done today, and what are the limits of current practice?
- What’s new in your approach and why do you think it will be successful?
- Who cares? If you’re successful, what difference will it make? What are the risks and the payoffs?
- How much will it cost? How long will it take? What are the midterm and final “exams” to check for success?
You can use these questions to structure a good introduction for a research paper. They are also a good way to get started with experimental design. If you can’t answer these questions about your experiment before you start then you will just be fishing around in the dark.
Is Robotics Suited to Experimental Design?
You might wonder whether the lack of strict experimental design procedures might be because they are not applicable to robotics. Other fields have stricter experimental design practices, including the pure sciences, medicine, psychology and the social sciences. You might wonder: Is robotics simply different?
This is quite a reasonable question to raise, especially as there is very little information about experimental design in robotics. However, the simple answer is that although, yes, robotics is quite a different field from many others, it could still be improved with stricter experimental design. Many people within the field agree and are trying to improve the experimental quality of robotic research by introducing benchmarks, standards, competitions and common software and hardware platforms.
I would recommend you read this paper from Willow Garage’s Leila Takayama, as a good introduction to why robotics needs more rigorous experiments and how we can achieve this.
The Five Components of a Well-Designed Experiment
Now that you can answer questions about the purpose of your experiment, you should define all of the key components. One way to look at experimental design is to split it into five components:
1. The Hypothesis or Hypotheses
The hypothesis is the big question that your experiment aims to answer. It’s so big, in fact, that we will devote another post to it later. Some examples of hypotheses are:
- Our inverse kinematic algorithm calculates the result of a 6-R robot in less time than three popular existing methods.
- Our robot gripper uses less electrical power than another similar gripper doing the same task.
- Human tests subjects respond more positively to Robot Design A than they do to Robot Design B.
- etc…
Quite simply, the hypothesis answers the question: What do you want to learn from this work?
It can be tempting to pick any old hypothesis and then start experimenting. However, it is best to sit down and formulate a good hypothesis. Here is one step-by-step guide on creating a hypothesis and here is another.
2. Variables
The variables of your study are those factors you are going to test in the experiment. The classic example is from medical drug research, where you have two groups of test subjects: a “drug group” and a “placebo group”. Apart from these variables, all other factors should either remain constant in the experiment or be removed some other way.
The variables that you will change in the experiment are known as independent variables. A classic example in robotics is to change the control algorithm and keep the rest of the experimental setup the same.
The variables which you will not change (but are affected by the independent variables) are known as dependent variables. One dependent variable for an algorithm might be, for example, computation time.
It can be difficult to decide which variables to choose, especially when you are comparing robotic platforms with many differences between them. However, as Takayama discusses, you can compare different systems if (and only if) you can control or eliminate any other variables of the system that might affect your variables. Is the speed of the robot important? Is the energy consumption important? Only compare the variables that make sense.
It is important to be objective about which factors are relevant to the hypothesis.
3. Measures or Metrics
When you have decided on your variables, you need to decide how you are actually going to measure those variables. For some variables, such as the computation speed of an algorithm, this might be a simple case of “time to solution”. However, for other experiments there may be several potential options for each variable.
For example, if you were comparing “user comfort” for various robot designs, you could ask the human subjects how the design made them feel. Alternatively, you could watch their actions when placed in the room with the robot. Both metrics might be valid but could produce very different results and present various challenges. Often, good research experiments will measure a variable in several different ways.
It is important to choose metrics that are not affected by confounding variables. This means variables which are neither those being tested, nor those which are controlled.
Confounding variables can come from many sources, because robots are such complex systems. Always try to think of other explanations for any result and don’t jump to conclusions. For example, imagine you set up an experiment to test the energy consumption of a robot to compare the same algorithm on two different electronic boards. Imagine that Board-A gave you a huge energy consumption; while Board-B used hardly any energy. A confounding variable might be the fact that Board-A turned the WiFi chip on by default; whereas Board-B turned the chip off by default. The energy-hungry WiFi chip would have nothing to do with your algorithm, but you could wrongly conclude that Board-B was more energy efficient.
Always question your results and seek out confounding variables.
4. Experimental Units
Now that we’ve picked our independent variable(s) and our dependent variable(s), and worked out how we are going to measure them, the next thing is to decide your “experimental units”, which can simply be thought of as: the number of things/subjects with which you’re going to test your system. In the case of algorithm research, this often means “data sets”. With human-based research, one experimental unit is usually comparable to one human subject, such as in human-robot interaction studies. For manipulation research, a robot setup may be tested on several different test setups.
In general, the more experimental units you test with, the better chance you have at producing a publishable research study. However, just because you have many results doesn’t necessarily mean that your experiment is well designed. Your results must be well aligned with the hypothesis, free of confounding variables and repeatable.
5. Assignment Method
The final step is to decide how to apply the variables to the experimental units. For example, imagine you have 1,000 data sets and you want to compare them with 10 different robotic grasping algorithms. You might ask questions like:
- Do you run all 10 algorithms on all 1,000 data sets?
- If not, how do you decide which data sets to try with each algorithm?
- How many times do you want to run each algorithm on each data set?
- etc…
The decisions you make for each of these five steps will make up the Experimental Design section of any research paper or report you write.
It is important to be able to defend the choices you make. Even if you don’t write everything in the paper (due to lack of space usually), you may be asked questions about it by others. Thinking through each stage of the experimental design before actually doing the experiments will make your research much stronger.
tags: benchmarking, c-Education-DIY, cx-Research-Innovation, robohub focus on standardization
