Previously known for consumer products for smartphones, tablets and cars, Paris-based Parrot recently branched into both the consumer and commercial drone businesses.

Parrot created the AR.Drone quadcopter and revealed it at CES 2010 in Las Vegas. Since then they have sold over a million of them – 700M in 2014 alone! In 2014, drones generated 34% of Parrot’s total revenue. At CES 2014 they launched two new mini drones for the retail consumer market. All appear to have done quite well during the Christmas season.

But what is particularly interesting is Parrots growth In the commercial drone sector. Parrot is acquiring companies and developing products to provide drones, software and data solutions for the agriculture, mapping and surveillance industries. To jumpstart their movement toward commercial drones, two spinoffs from the Swiss Federal Institute of Technology (EPF), senseFly and Pix4D, were invested in in 2012.

• senseFly designs and develops a line of winged drones and is soon to release a quadcopter as well. All their drones are for commercial use. senseFly’s $25K eBee drone was rated one of the Top 7 Drones for Agriculture in 2014 by DroneLife.
• Pix4D provides image processing software for mapping and modeling geo-referenced 2D images into 3D models and for a wide range of GIS and CAD applications.

Two more acquisitions in B2B (business-to-business) drones happened in 2014: Parrot invested in MicaSense and Airinov.

• MicaSense is a Seattle-based software and systems company providing advanced data gathering and processing, coupled with professional analysis tools, to provide accurate and repeatable information on the status of a crop. Raw data from the MicaSense camera, a lightweight, multispectral camera, is transformed into vegetation index maps. Powerful analytics provide time-based trends and change maps. Analysis tools such as plant population counts enable optimized farm management. Parrot invested $2M in a Series A round of funding in November, 2014.
• Airinov is a French provider of UAV farming applications and sensors for use on senseFly and other UAS. Their software can process flight and sensor data and produce fertilization recommendations which are compatible with most brands of dispenser equipment. Parrot invested $2.2M for a 20.9% equity interest in Airinov.

Henri Seydoux, Parrot’s founder, chairman and CEO, said:

We are moving forward with our external growth policy initiated in 2011 and focusing on new products with high-potential, outstanding and complementary technological expertise, applications for commercial and retail customer segments, strong operational and financial synergies. Parrot intends to meet the needs of professionals moving into the civil drone age and firmly believes in the commercial potential of this market, (on which the Parrot AR.Drone has already enabled us to gain global recognition) and I am very pleased that we can also serve the commercial drone market as well.

Agriculture is one of the fastest-growing market segments for unmanned aircraft commercial applications. The sensors required to capture accurate data are a critical part of the solution, and [our recent acquisition of MicaSense and Airinov] brings this technology to the table.

As much fun as Parrot’s drones are, there is a growing market for professional service drones for mapping, surveying, protection, real estate photography and agricultural uses. Shenzen-based Dajiang Innovation Technology (DJI) has sold over 400M of their line of consumer and B2B drones; San Francisco startup Skycatch has partnered with one of the largest heavy machinery makers, Komatsu, to automate construction and mining job sites world wide using drones, mapping software and various sensors.

• There is also a steadily growing defense and first-responder market led by Lockheed Martin, Boeing, AeroVironment, Elbit and Northrup Grumman. The Global UAV Market 2015-2025 projects the global defense UAV market to grow at a 5.66% CAGR from 2014 to 2025 with North America and Europe the largest markets and the military drone segment (UCAV – unmanned combat aerial vehicle) to dominate the UAV market.
• But the real good news is Wintergreen Research’s report entitled “Commercial Drones: Highways in the Sky, UAS, Market Shares, Strategies and Forecasts, Worldwide, 2015 to 2021” (whew!) which projects commercial drone growth at 34.3% CAGR over the period from 2014 to 2021. That’s game-changing growth!

This is the third tutorial in the Up and flying with the AR.Drone and ROS series.

In this tutorial we will:

1. Learn about the AR.Drone’s state feedback (and how it is handled by ROS)
2. Learn about the AR.Drone’s tag detection
3. Program our first ROS nodes: A subscriber and a publisher

In the previous tutorials we:

1. Installed ROS, the AR.Drone driver and AR.Drone keyboard controller then flew the AR.Drone using the provided keyboard controller (link)
2. Learned about ROS communication, setup and then flew the AR.Drone using a joystick (link)

In the next tutorials, we will:

• Write a controller to enable tag following
• Write a controller for the drone which allows us to control drone velocity, rather than body angle

What you’ll need for the tutorial

1. An AR.Drone!
2. A computer or laptop with WiFi
3. Linux, either the ARDroneUbuntu virtual machine, or a custom installation
4. Either the keyboard or joystick controller from the previous tutorials

I also assume that you’ve completed either the keyboard or joystick tutorial, which detail how to setup the software, connect to the drone, etc, and provide a method of controlling the AR.Drone from your computer.

Updating

If you have not completed the joystick tutorial:

Please go back and update your installation according to the instructions listed there.

If you downloaded the ARDroneUbuntu virtual machine:

To update your installation for this tutorial, click the “Update Tutorials” button in the ARDroneUbuntu Virtual Machine

If you use a custom installation:

At the terminal, enter the following:

$ roscd ardrone_autonomy
$ git pull
$ roscd ardrone_tutorials
$ git pull
$ roscd
$ rosmake ardrone_tutorials

AR.Drone State Feedback – A First Look

Preparation:

1. Find the Orange-[Green|Yellow|Blue]-Orange tag that came with the AR.Drone.
2. In your ARDroneUbunutu virtual machine (or custom installation), open a new terminal window and type:

   $ roscd ardrone_tutorials/launch

3. If you intend to fly with the keyboard, type:

   $ geany keyboard_controller_with_tags.launch

   Otherwise, if you intend to fly with the joystick:

   $ geany joystick_controller.launch

4. Now find the line:

   <param name="enemy_colors" value="3" />

5. And change the value to “1”, “2”, or “3”, depending on whether your tags are Orange-Green-Orange, Orange-Yellow-Orange or Orange-Blue-Orange respectively.

Tag Detection and Feedback Data

1. Run the system (but don’t takeoff with the drone!) using either the keyboard_controller:

   $ roslaunch ardrone_tutorials keyboard_controller_with_tags.launch

   or the joystick_controller:

   $ roslaunch ardrone_tutorials joystick_controller.launch

2. Open a new terminal window and type:

   $ rostopic list

   • This list shows all the active ROS “topics” used for communication to and from the AR.Drone
3. In the same terminal window, type:

   $ rostopic echo /ardrone/navdata

   • This echoes the AR.Drone’s feedback data into the terminal window.
   • Note that many items will be zero because the AR.Drone is not yet flying
4. Move the Orange-Color-Orange tag in front of the AR.Drone’s front camera. It should be detected, and the console output (from the previous step), should update to reflect this detection.
   • Note the tags_xc and tags_yc values. These represent the x/y location of the tag within the image.
   • The video display window will show a dot at these coordinates, and the distance measurement above the dot
5. Now place the tag somewhere that can be seen by the flying drone. After this is done, take off with the drone.
6. Watch the console output. What happens if you fly forwards? sideways? What variable changes if you rotate the drone?

Plotting the AR.Drone’s State

1. A more intuitive way of representing this feedback data is using a plot. PressCtrl-C to terminate the echo process from the last stage, and then type:

   $ rxplot /ardrone/navdata/rotX,/ardrone/navdata/rotY

   • This launches a plotting window showing the AR.Drone’s estimated roll (rotX) and pitch (rotY).
   • Similar plots could, for example, be generated for yaw (rotZ), x and y velocity (vx and vy), altitude (altd), etc.
2. Similarly to displaying the AR.Drone’s feedback data, we can use the plotting method to investigate how we are controlling the drone:

   $ rxplot /cmd_vel/linear/y,/cmd_vel/linear/x

   • This plots the roll (linear/y) and pitch (linear/x) commands.
   • Note that these are scaled with a value of $\pm1$ being the maximum commandable angle (set by the euler_angle_max in the launch file).
   • Note also that the direction of AR.Drone movement is commanded (ie. move along the x-axis = /cmd_vel/linear/x = commanded pitch), whereas the AR.Drone’s feedback data is named with reference to axis of rotation (ie. tilt around y-axis = /ardrone/navdata/rotY = feedback pitch).

Logging Data

1. To log data, we can use the inbuilt ROS command rosbag:

   $ rosbag record -O ARDroneFlight.bag /ardrone/navdata /cmd_vel

   • This will record the topics /ardrone/navdata and /cmd_vel to a “bag-file”, stamped with the current date and time.
   • Leave this process running for a few seconds, fly the AR.Drone around, and when finished press Ctrl-C to terminate.
   • Recording all topics is possible with rosbag record -a, however this also records the AR.Drone’s HD video stream, thus bag-files will quickly become very large
   • The advantage of using rosbag to log data, is that bag-files can be replayed at a later time, and interact very nicely with the existing ROS infrastructure.
2. Land the AR.Drone and close any windows / terminate any processes relating to the controller. Now, into the terminal window that ran the AR.Drone controller, type:

   $ roscore

   Now open a new terminal window and type:

   $ rxplot /cmd_vel/linear/y,/cmd_vel/linear/x

   Now finally (ignoring the warning from the previous step), switch back to the terminal window that you used to run the rosbag command and type:

   $ rosbag play ARDroneFlight.bag

   Does the flight look familiar? It should! This is the flight you recorded in the last step! Use Ctrl-C to terminate the above processes.

Custom Data Handling

In the last section, we investigated the AR.Drone’s feedback data by using ROS’s inbuilt plotting (rxplot) and logging (rosbag) commands. In this section, you will learn how we get this feedback data into your own application by writing a custom ROS Node. This node will be used to compare the commanded roll and pitch angles with the AR.Drone’s roll and pitch estimates, thus laying the foundations for the next tutorial, where we will use this feedback data to programmatically control the AR.Drone.

Messages, Topics and Nodes

In ROS terminology, a program is known as a Node. To interact with other nodes (for example the AR.Drone driver), each node will have some subscribers, which listen for information messages on a specific topic (communications channel); and some publishers, which send messages to a specific topic.

These messages can be anything. An example that we saw in the previous section was the Navdata message, which is sent over the /ardrone/navdata topic and contains feedback data from the AR.Drone.

We will now investigate this message a little further. Open up a new terminal window and type

rosmsg show ardrone_autonomy/Navdata

You should be presented with the following output:

$ rosmsg show ardrone_autonomy/Navdata
std_msgs/Header header
  uint32 seq
  time stamp
  string frame_id
float32 batteryPercent
uint32 state
int32 magX
int32 magY
int32 magZ
int32 pressure
int32 temp
float32 wind_speed
float32 wind_angle
float32 wind_comp_angle
float32 rotX
float32 rotY
float32 rotZ
int32 altd
float32 vx
float32 vy
float32 vz
float32 ax
float32 ay
float32 az
uint32 tags_count
uint32[] tags_type
uint32[] tags_xc
uint32[] tags_yc
uint32[] tags_width
uint32[] tags_height
float32[] tags_orientation
float32[] tags_distance
float32 tm

This listing shows us the various members of the Navdata message, including their datatypes. Note that the Navdata message has a header, which includes the time the message is sent.

Minimum Viable Subscriber

Open up a new terminal window and type:

$ roscd ardrone_tutorials
$ geany src/minimum_viable_subscriber.py

Up will pop a geany window, ready for your to edit the new file. Enter the following into the new geany window, then save and close the file:

import roslib
import rospy

roslib.load_manifest('ardrone_tutorials')

from ardrone_autonomy.msg import Navdata

def ReceiveData(data):
  print '[{0:.3f}] Pitch: {1:.3f}'.format(data.header.stamp.to_sec(), data.rotY)

rospy.init_node('minimum_viable_subscriber')
sub_Navdata = rospy.Subscriber('/ardrone/navdata', Navdata, ReceiveData)

while not rospy.is_shutdown():
  pass

We will now use the previously recorded bag-file to test the node that we’ve just written.

1. In one terminal window, enter:

   $ roscore

   Which begins the roscore process, which coordinates the interactions between nodes.

2. Now in a new terminal window type:

   $ roscd ardrone_tutorials
   $ python src/minimum_viable_subscriber.py

   This will launch the node that we’ve just programmed.

3. Finally, in a third terminal window, begin the bag-file playback with:

   $ rosbag play ARDroneFlight.bag

   • Note you must be in the same directory as the bag-file for the above command to work
4. If you switch back to the terminal window in which our new node is running, you should see a stream of time-stamped printouts with the drone’s pitch, for example:

   [1353596639.491] Pitch: 4.375
   [1353596639.511] Pitch: 4.128
   [1353596639.531] Pitch: 3.750
   [1353596639.552] Pitch: 3.304
   [1353596639.571] Pitch: 3.063

Minimum Viable Subscriber: A Closer Look

If you made it through the previous section, congratulations, you have successfully written and tested a ROS node, which listens for Navdata measurements from the AR.Drone (or equivalently: a recorded flight) and outputs a time-stamped pitch measurement. But how does it all work?

Lets take a look at the source-code from the previous section, line by line:

import roslib
import rospy

With these two lines, we import the standard ROS packages, required for every ROS node.

roslib.load_manifest('ardrone_tutorials')

Once the required packages are imported, load the manifest file. This file details the dependencies of the current project (for example, that ardrone_tutorials is dependent on the AR.Drone driver, ardrone_autonomy), which through the above command are loaded into the path.

from ardrone_autonomy.msg import Navdata

Once the project dependencies have been loaded, we can import those which are needed by the current node. In the above case, we load the Navdata message definition from the ardrone_autonomy package.

def ReceiveData(data):
  print '[{0:.3f}] Pitch: {1:.3f}'.format(data.header.stamp.to_sec(), data.rotY)

Here we define a callback function to handle the Navdata messages. This function prints a formatted string containing the message’s timestamp and the AR.Drone’s pitch angle. This callback function will be used later in the program.

rospy.init_node('minimum_viable_subscriber')

Before issuing any commands to the rospy package, we need to initialize a ROS node.

sub_Navdata = rospy.Subscriber('/ardrone/navdata', Navdata, ReceiveData)

We can now subscribe the node to various topics. Above we subscribe the node to the /ardrone/navdata topic, which carries Navdata messages. Received messages should then be handled by the previously-defined ReceiveData function.

while not rospy.is_shutdown():
  pass

Because our program is relying on callbacks for data processing, we need to stop the program from terminating. The above defines a loop that runs until the node is shut down (for example with Ctrl-C). In most situations the above loop would perform some sort of processing, however in our example, we simply pass (do nothing).

Publishing Data

In this section we will introduce the concept of publishing data to a topic (the opposite of subscribing). We will be using this more extensively in the next tutorial, where we will be programmatically controlling the AR.Drone by publishing messages to the /cmd_vel topic.

Minimum Viable Publisher

The goal of this section is to create a minimum_viable_publisher.py file such that Navdata messages are averaged over a second and the average pitch published to the (new) /ardrone/average_pitch topic. Modify the file as follows:

import roslib
import rospy

roslib.load_manifest('ardrone_tutorials')

from ardrone_autonomy.msg import Navdata
from std_msgs.msg import String

messages = []

def ReceiveData(data):
  messages.append(data)

rospy.init_node('minimum_viable_publisher')
sub_Navdata = rospy.Subscriber('/ardrone/navdata', Navdata, ReceiveData)
pub_Average = rospy.Publisher('/ardrone/average_pitch', String)

r = rospy.Rate(1)
while not rospy.is_shutdown():
  if len(messages)>0:
    avg = sum([m.rotY for m in messages])/len(messages)
    messages = []

    avgmsg = String()
    avgmsg.data = 'Average Pitch: {0:.3f}'.format(avg)
    pub_Average.publish(avgmsg)
  r.sleep()

You can run the program using the following series of commands:

1. In one terminal, run:

   $ roscore

2. In the next terminal, run the new program

   $ roscd ardrone_tutorials
   $ python src/minimum_viable_publisher.py

3. We now echo the published data to the topic we’ve created:

   $ rostopic echo /ardrone/average_pitch

4. And finally, like we’ve done twice before, we begin the playback of our recorded flight

   $ rosbag play ARDroneFlight.bag

5. Switching back to the terminal in which we ranrostopic echo, you should see a new message with the average pitch appearing once a second:

   data: Average Pitch: 0.649
   ---
   data: Average Pitch: 1.078
   ---
   data: Average Pitch: 3.781
   ---
   data: Average Pitch: 3.285
   ---

Minimum Viable Publisher: A Closer Look

Comparing minimum_viable_publisher.py to minimum_viable_subscriber.py from the last section, we can see many similarities:

from ardrone_autonomy.msg import Navdata
from std_msgs.msg import String

We additionally import the String message, required for our new topic.

messages = []
def ReceiveData(data):
  messages.append(data)

We have updated the navdata callback function to save the messages in a queue, rather than process them straight away.

Note: because we’re accessing the messages list in both a callback function and the main loop, we should be using Python’s threading.Lock to control access to the list and prevent race conditions. See ardrone_tutorials/src/drone_video_display.py for an example.

pub_Average = rospy.Publisher('/ardrone/average_pitch', String)

We create a new publisher, responsible for publishing messages of the String type to the /ardrone/average_pitch topic. Note that we do not need to explicitly initialize this topic, it will be initialized automatically on first usage.

The biggest change is in the program’s main loop, which I will now cover in a little more detail:

r = rospy.Rate(1)

Firstly we define a Rate object. We use this to control the frequency of the main processing loop. In our case, we want to print a message every second, thus we want a frequency of $1 \text{Hz}$.

while not rospy.is_shutdown():
  if len(messages)>0:
    avg = sum([m.rotY for m in messages])/len(messages)
    messages = []

    avgmsg = String()
    avgmsg.data = 'Average Pitch: {0:.3f}'.format(avg)
    pub_Average.publish(avgmsg)

If messages have been received, we calculate the average pitch using a python list comprehension to extract the pitch (.rotY) from every Navdata message. We then empty the messages queue.

We then construct a new String message and set its data property, before publishing it to the topic we previously created.

r.sleep()

Finally, we tell the rate object to sleep. This command will calculate how long it needs to sleep, such that the desired loop frequency is achieved.

Conclusions

I hope you’ve enjoyed working through this tutorial and learning about data flow in the ROS environment. In the next tutorial, which I will release in a month’s time, we will use what we have learned today to programmatically control the AR.Drone, while at the same time learning about PID control.

If you have any questions about the tutorial, please leave me a comment below and I will get back to you as soon as possible!

You can see all the tutorials in here: Up and flying with the AR.Drone and ROS.

If you liked this article, you may also be interested in:

• Parrot AR.Drone app harnesses crowd power to fast-track vision learning in robotic spacecraft
• Taking your first steps in robotics with the Thymio II
• Discovering social robotics with the Aisoy1
• Classroom robotics: Motivating independent learning and discovery

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Parrot’s AR.Drone without styrofoam bumper guard (top) and with (bottom).

As an aside, part of the business plan for selling the AR.Drones is to also sell parts to repair the devices when they crash – an inevitable outcome of the adventure of flying them. Parrot has made buying replacement parts easy and provides visual tutorials on how to install replacement parts and make needed repairs. A bit like printer companies selling their own brand of paper and ink.

Henri Seydoux, Founder, Chairmen and CEO, Parrot SA.

As researchers at the European Space Agency’s Advanced Concepts Team, we wanted to study how visual cues could be used by robotic spacecraft to help them navigate unknown, extraterrestrial environments. One of our main research goals was to explore how robots can share knowledge about their environments and behaviors to speed up this visual learning process.

© European Space Agency, Anneke Le Floc’h

Similar to the RoboEarth project, the central idea is that a group of robots sharing visual information such as raw camera images or abstracted mathematical image features would have a much broader visual experience to learn from than a single robot operating on its own.

We had already studied the use of motion-based cues in navigation, such as optic flow and the size changes of visually salient features like SURF, but we we wanted to verify our findings in this area and at the same time investigate if the appearance of visual features could be used to aid navigation as well. To this end, we needed a very large robotic data set of visually salient image features paired with corresponding robotic state estimates, and it was simply infeasible to gather all this data by ourselves.

That’s where the Parrot AR.Drone entered into play. The AR stands for Augmented Reality, and the drone is indeed a toy quad rotor meant for playing games. We thought we could develop a game for the Parrot that would serve as a means for crowd-sourcing the data we needed. And because the Parrot was designed to be used in augmented reality games and can be controlled by an iOS device, data-gathering could be done in a gaming context, making it fun for people to participate in the experiment.

To this end, we designed a mission-based game where players simulate docking the Parrot with the International Space Station as quickly as possible while maintaining good control. Bonus points are given for correct orientation and low speed on the final approach.

At the end of the mission, players can log their high scores on a score board, and at the same time contribute to the experiment by anonymously sharing abstract mathematical image features and velocity readings.

The Parrot AR.Drone is uniquely suitable for this purpose because:

• It is widely available to the public, with over a half-million units sold world-wide.
• It is a real robot with many useful sensors, including two cameras (at the front and on the bottom), sonar, accelerometers, gyrometers, and (on the Parrot AR drone 2.0) a pressuremeter and magnetometer. Further, it already uses these sensors to achieve accurate onboard state estimation and (consequently) stable hovering.
• It can be controlled by smart devices such as the iPhone or Android devices, which can be used send data over the internet.
• The code for communicating with Parrot is open source.

Methodology
The question we faced experimentally was how to couple the real-world object to the virtual space in which the drone would be flying. Having a reference object of known size was a goal since it would provide some form of ground-truth measurement. As it happens, the Parrot AR.Drone is delivered with a marker that is recognized in the images by the onboard firmware. The recognition gives an image position (x,y) and a distance to the marker. Note, however, that this information is not enough to disambiguate both the position and orientation of the drone with respect to the marker. Figure 1 shows two configurations that give exactly the same marker detection data. In the ‘red’ case, the marker is in the left part of the image, because the drone is located on the right of the marker. In the ‘blue’ case, the marker is in the left part of the image, because the drone has turned to the right. Note that the distances to the marker are the same.

Figure 1: Ambiguity of retrieved marker data. The angle β and distance d are the same for the red and the blue case, but their position / heading are different.

In order to render the virtual space environment, we perform state estimation on the basis of the drone’s estimates of its height, speeds, angles, and its detection of the marker. The docking port of the ISS (the marker) is assumed to be at location (0,0,0). The state estimation uses an Extended Kalman Filter (EKF), and in the light of the discussion of the marker’s ambiguity, we decided to only update the position on the basis of the marker’s readings, and not the heading. The heading is estimated quite well on board the drone, where the Parrot AR.Drone 2 also uses a magnetometer.

Figure 2 shows what computational processes are happening on which device during the game. The marker detection and the estimation of the drone’s attitude, height, and speeds are performed by the firmware on board the drone. The app is running the drone’s control interface, sending and receiving data from the drone, performing state estimation (X,Y,Z) with respect to the marker and rendering the 3D world.

Figure 2: Distribution of processes over the devices involved in the game.

Of course, for the experiment we also need to process the drone’s onboard images. Because of the iPhone’s computational constraints, we decided to save 5 subsequent images while the player is trying to dock and to process the images after the flight. This processing now happens when the player visits the highscore table and agrees to join the experiment. After extracting the vision data and concatenating this with the state estimates, the data is encrypted for transmission. Finally, since some players may use 3G for sending their data, we considered that the data should be as compact as possible. Therefore, the data is compressed on board the iPhone before being sent. The data from 5 textured images on average takes around 77 kb.

© European Space Agency, Anneke Le Floc’h

Testing and Launch
Our original plan was to work on the game from February to April 2012, launch it in May or June, and analyze the data from August to September. In reality, we didn’t converge on the final code until November 2012.

Of course, when you are developing the app, you know the program inside out and by habit you may avoid playing the game in such a way that would make the program malfunction. So in November we invited other people try out and rate the app. The ratings and comments were used to perform some final adjustments, and we sent the app to the Apple store for review in January 2013. At the same time, we began to execute our plans for the launch and related PR-activities. We saw this as an essential part of the project, as the “crowd” first has to know about a project in order to contribute to it.

The app was launched on the 15^th of March 2013, accompanied by a press release about AstroDrone from the European Space Agency and the video shown below. Parrot also promoted our app on their site. The news was soon picked up by BBC Technology, Wired, and the Verge, as well as news papers, and television shows such as ARD’s NachtMagazin. This (ongoing) media attention has helped us to reach the public: just one week after the launch, some 4,000 persons downloaded the AstroDrone app, and 458 of these people had already contributed to our experiment by sending their data. This is a nice start, but of course we hope there is more to come.

Next Steps
We have not arrived at the end of this project, but rather at the beginning. We are first solving some small issues by means of app updates. For example, if an iPad or iPod is connected to the drone’s WiFi, it cannot send its data. There was no message that showed the absence of an internet connection (solved in version 1.2). At the same time, we have started analyzing all the data that is coming in.

In the future, we may also improve our crowd-sourcing game in various ways, in order to make it as fun and easy as possible for players, and as useful as possible for our research. In any case, it’s an adventure to work with this exciting new way of gathering robotic data!

[Ed. Hallie Siegel]

This is the second tutorial in the Up and flying with the AR.Drone and ROS series.

In this tutorial we will:

1. Talk about the ROS communication hierarchy
2. Setup a joystick to work with ROS
3. Fly the AR.Drone with a joystick

In the previous tutorial we:

1. Installed ROS, the AR.Drone driver and AR.Drone keyboard controller then flew the AR.Drone using the provided keyboard controller (link)

In the next tutorials, we will:

• Look more closely at the data sent back from the drone and play with the on-board tag detection (link)
• Write a controller for the drone to enable tag following
• Write a controller for the drone which allows us to control drone velocity, rather than body angle

What you’ll need for the tutorial

To complete this tutorial you will need:

1. An AR.Drone!
2. A computer or laptop with WiFi
3. Linux, either the ARDroneUbuntu virtual machine, or a custom installation
4. A  (linux compatible) USB joystick or control pad

I also assume that you’ve completed the previous tutorial, which details how to setup the software, connect to the drone, etc.

Updating

It has been three weeks since the launch of the first tutorial and I am very happy with the response so far. The tutorial has been viewed over 720 times and ARDroneUbuntu downloaded over 50 times. I was also given the honor of partaking in RobotGarden’s very first Quadrocoptor Workshop, where we worked through the tutorial (and ironed out the bugs) together.

This response has taught me a few things about how to structure a tutorial series, and how to release a virtual machine. Based on what I have learned, I will be restructuring the source-code for the following tutorials. I will also be updating the virtual machine to reflect these changes.

When writing the first tutorial, I thought it would be better to have a separate repository for each tutorial to keep everything atomic and isolated to ensure the addition of a new tutorial would not affect anything in the past – However having learned from the video-hangout with RobotGarden, I realize this could cause an update nightmare down the track. For this reason, I have decided to operate from a single repository and will maintain a stable branch for each tutorial, with the master branch always synchronized with the stable version of the latest tutorial. The source-code for each tutorial will also contain all source-code required to run the previous tutorials.

What this means for you:

• If you downloaded the ARDroneUbuntu virtual machine before the release of this tutorial, you will need to click the “Update Tutorials” button twice. After this, your virtual machine will be updated to reflect the new changes.
• If you installed the tutorials manually, you will need to run the following at the terminal:

  # roscd
  # rm -rf ardrone_tutorials_getting_started
  # git clone https://github.com/mikehamer/ardrone_tutorials

• If you have not yet downloaded the VM, no problems! I have released an up-to-date version that you can download from here

Sorry for any hassle caused by the above, I hope this new structure will allow me to more easily deliver the next tutorials!

So without further ado, lets check out some data!

1. Connect to the AR.Drone as detailed in the first tutorial (remembering to disconnect other network adaptors if they use the 192.168.1.x address range)
2. From within your Linux installation, launch two terminal windows (ARDroneUbuntu users, refer to the first tutorial if you’re not sure what I mean)
3. In the first window, launch the KeyboardController from the first tutorial, using the command:

   $ roslaunch ardrone_tutorials keyboard_controller.launch

4. Wait for everything to load and the video display to become active, then in the second terminal window enter the command:

   $ rostopic list

   This command lists all active “topics”, the communication channels through which ROS programs (called “nodes”) communicate.

5. To view the currently running ROS Nodes, type:

   $ rxgraph

   The window that appears will show a network of nodes, and how they are connected through the topics that we saw in the previous step.

6. When you’re ready, close this window. We’re now going to have a look at the messages that are being sent between nodes. At the terminal, type:

   $ rostopic echo /ardrone/navdata

   The navdata message is published by the AR.Drone driver at a rate of 50Hz.

7. Press Ctrl-C to stop echoing the navdata message then switch to the other terminal window and press Ctrl-C again, to stop the roscore program

So what are we setting up today?

After that whirlwind tour of the ROS communication hierarchy, lets take a look at what we’ll be building today:

As shown above, today we’ll be setting up the joy_node (a standard ROS package) to listen for input from a joystick or control-pad. This input will be converted into a ROS message, then sent to our controller over the joy topic.

Our controller subscribes to the joy topic and reacts to the received messages by translating joystick movement into drone movement commands, which are then published to the cmd_vel topic; and button presses to takeoff, land, or emergency commands, which are each published to a respective channel.

The ardrone_driver receives these various commands and then sends them via the wireless network to the physical drone, which in turn sends back a video stream. This video stream is published by the ardrone_driver onto the /ardrone/image_raw topic, from where it is received and subsequently displayed by our controller.

Setting up a joystick

Firstly, we need to connect the joystick and set it up to work with ROS.

1. Open a terminal window (or reuse one from before), then type:

   $ ls /dev/input

2. Take note of the jsX devices (where X is a number). These are not your joystick.
3. Now connect your joystick to the computer and again type:

   $ ls /dev/input

4. If you see a new jsX device appear, congratulations, you’ve successfully connected the joystick, take note of which device this is, then progress to the next section of the tutorial
5. If no new jsX device appears when you connect the joystick, you’re most likely using a virtual machine and may need to add the USB device manually.
6. If you are using the ARDroneUbuntu virtual machine, open the Machine > Settings window, then the Ports > USB tab. With the joystick plugged in, add a new USB device. Your joystick should appear in the dropdown list.
7. Click OK, then unplug the joystick from the computer
8. Repeat steps 3 and 4, plugging the joystick into a different USB port if it is not detected in step 4.

Testing the joystick

Now that we have the joystick installed, let’s check to make sure everything is working. Type the following at a terminal, substituting jsX for the joystick device you previously identified:

$ jstest /dev/input/jsX

If you are greeted with a “jstest: Permission denied” error, type:

$ sudo chmod a+r /dev/input/jsX

Which enables all users to read the joystick device (ARDroneUbuntu users, the password is: ardrone). After running jstest, moving the joystick should cause the numbers on the screen to update, if not, try running jstest for the other jsX devices. Congratulations, your joystick device is working! Press Ctrl-C to exit.

Using the joystick with ROS

Now we’re ready to integrate the joystick device with the ROS ecosystem using ROS’s “Joy” package.

1. Open up three terminal windows. In the first, launch roscore:

   $ roscore

2. In the second, we launch ROS’s joystick launcher, replacing jsX with the joystick device you previously identified:

   $ rosparam set joy_node/dev "/dev/input/jsX"
   $ rosrun joy joy_node

3. Now in the third terminal window, we echo the Joy messages, sent out by the joy_node that we just launched:

   $ rostopic echo joy

4. If all is working, you should see a Joy message print on the console whenever the joystick is moved (or a button is pressed).

Identifying Control Buttons & Axes

We now have the task of identifying the various axis and buttons used to fly the drone. All in all, you will need to identify axes to control the AR.Drone’s roll, pitch, yaw-velocity and z-velocity; and buttons to trigger an emergency, command a takeoff, and command a landing.

To identify the emergency button for example, press the button you would like to trigger an emergency, and watch the terminal to see which button is enabled. In my case, I use the joystick’s trigger button for emergency (being able to react quickly to an emergency is very important). With the button pressed, the following is printed in the terminal window:

axes: [-0.0, -0.0, -0.0, -0.0, 0.0, 0.0]
buttons: [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

From this I can identify that: Emergency = Button 0 (note that indexing starts from zero). The process follows similarly for identifying an axis:

1. Press the joystick fully left. The axis that changes  (the first number in the list is axis 0) is the roll axis, the number if changes to is your roll scale (usually + or -1)
2. Press the joystick fully forward. The axis that changes is your pitch axis, and the number it changes to is the pitch scale (usually + or -1)
3. Rotate your joystick fully counter clockwise, the axis that changes is your yaw axis and the number it changes to is the yaw scale (usually + or -1)
4. Push your throttle stick fully forwards. The axis that changes is your z-axis and the number it changes to is the z-scale (usually + or -1).

axes: [1.0, -0.0, -0.0, -0.0, 0.0, 0.0]
buttons: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

From this I identify that the Roll Axis = 0 and Roll Scale = 1.

After you have identified all axes and buttons, press Ctrl-C in all terminal windows to stop the processes.

Configuring the Joystick Controller

In ROS, a “Launch File” is used to configure and launch a set of nodes. We’ve already seen this in the first tutorial (and at the beginning of this one), where we typed:

$ roslaunch ardrone_tutorials keyboard_controller.launch

In this command, keyboard_controller.launch configures the ARDrone driver for indoor flight, then launches both the driver and keyboard controller. Similarly in this tutorial, we will be launching the joystick controller (and related nodes) using a launch file.

Go ahead and open a terminal window, then type:

$ roscd ardrone_tutorials
$ geany launch/joystick_controller.launchtemplate

This will open the joystick_controller.launch file in the text editor (non ARDroneUbuntu users can use a text editor of choice). You will need to customize this file for your setup.

• Update line 21 to reflect your joystick device (the /dev/input/jsX that you identified earlier).
• Update lines 27-29 to reflect your button mapping
• Update lines 32-35 to reflect your axis mapping
• Update lines 38-41 to reflect your scales (usually either 1 or -1)
  These scaling factors determine for example the drone roll direction when the joystick is tilted along the roll axis. If your drone does the opposite of what you expect, change the sign on the relevant scaling factor (eg 1 becomes -1)

Feel free to have a look at some of the parameters used on the drone. The drone’s control parameters can potentially be increased to allow for more aggressive flight.

Once you’re happy with the settings, save this file as joystick_controller.launch, then close the geany application.

Finally flying!

So, after that rather long endeavor (which you should only need to do once), lets fly!

Connect to the drone, disconnect other network devices, then open a terminal and type:

$ roslaunch ardrone_tutorials joystick_controller.launch

This will launch the nodes as you configured in launch file. If all goes well, you should now be able to fly the drone using your joystick.

Before commanding the drone to take-off, you should first check that your emergency button works. With the drone still landed, press the emergency button once. All onboard LEDs should turn red. Pushing the button again will reset the drone, and the LEDs will once again turn green. In addition to this, whenever a (mapped) button is pressed, the joystick_controller will print a message to the terminal, such as “Emergency Button Pressed”. This is a further indication that things are working!

You’re now ready to take off. Have fun!

Going Further

The above child’s play? Heres some ideas for exploring the ROS and ARDrone environment:

• While the joystick controller is running, open a new terminal window and type:

  $ rxgraph

  This will display the node communication diagram that we investigated at the start of the post

• Take a look at the data being sent back from the drone:

  $ rostopic echo /ardrone/navdata

• Plot your flight speeds (or any other data of interest) with:

  $ rxplot /ardrone/navdata/vx,/ardrone/navdata/vy

• Adjust drone parameters in the joystick_controller.launch file to fly more aggressively

Troubleshooting

• The joystick doesn’t do anything
  Check that the joystick device you configured in the launch file corresponds to your joystick. To do this, repeat the section “Testing the Joystick”
• My joystick device is correct, but the joystick doesn’t do anything
  Assuming you’ve gone through the “Testing the Joystick” section successfully, launch the joystick controller again and in a separate terminal, type:

$ rostopic echo joy

Assuming everything is configured correctly, this will print joystick messages to the terminal when the joystick is removed.

• If nothing is printed, check to make sure you’ve configured the correct joystick device in the launch file, and that the joystick permissions are correct (see “Testing the Joystick”).
• If something is printed, check to make sure your axis and button mappings in the launch file are correct. If this is the case, leave a comment to this tutorial and I will be happy to help!

• The drone does the opposite of what I expect
  For example flying backwards when the joystick is tilted forwards. To solve this, determine the axes that respond counterintuitively then invert the sign of the respective scaling factors in the launch file. For example, if the pitch axis (forward-backward) is the opposite of what you would expect, the ScalePitch parameter would change from 1 to -1.
• I cannot connect to the drone
  Check that your computer is connected to the drone via wireless, and that all other network devices are disconnected
• The drone will not take off

• Are all the onboard LEDs red? If so, try pressing the emergency button. Do they toggle to green? Perfect – you’re ready to fly!
• If the LEDs are green and the drone still won’t take off, check that your button mapping is correct in the launch file

• Other issues?
  If you have any questions, or comments about what you would like to see in the next tutorials, please send me a message or leave a comment to this post!

You can see all the tutorials in here: Up and flying with the AR.Drone and ROS.

If you liked this article, you may also be interested in:

• Parrot AR.Drone app harnesses crowd power to fast-track vision learning in robotic spacecraft
• Taking your first steps in robotics with the Thymio II
• Discovering social robotics with the Aisoy1
• Classroom robotics: Motivating independent learning and discovery

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this tutorial (#1) we will:

1. Install ROS, the AR.Drone driver and AR.Drone keyboard controller
2. Fly the AR.Drone using the provided keyboard controller

In tutorials 2-5, we will:

2. Modify the keyboard controller to work with a joystick or control pad, giving much finer control (link)
3. Look more closely at the data sent back from the drone and play with the onboard tag detection (link)
4. Write a controller for the drone to enable tag following
5. Write a controller for the drone which allows us to control drone velocity, rather than body angle

Updates:

Based on feedback I received from a video hangout with Robot Garden (cofounded by fellow Robohub author, Andra Keay), I have made a few updates to the tutorial, summarized below and highlighted throughout the tutorial in red:

1. Fixed permissions problem which caused the keyboard button on the launch bar to do nothing
2. Added keyboard_controller_with_tags.launch file, which enables the drone’s tag detection feature
3. Updated tutorial with command-line equivalent commands, for all graphical actions
4. Included a “Going Further” section at the end of the tutorial

If you are running the ARDroneUbuntu virtual machine, updating the tutorial will download the above updates and fixes.

Command-line equivalent (updating the tutorial):
$ roscd ardrone_tutorials
$ git pull

What is the AR.Drone?

The AR.Drone is a quadrotor that can be flown from any WiFi enabled device with an appropriate application – most commonly an iPhone or Android phone.

Unlike other quadrotors, which can be difficult to fly manually, the AR.Drone is equipped with onboard sensors and controllers that stabilize the vehicle while flying and give it the ability to hover on a spot and await the next command. Because of this onboard stabilization, the AR.Drone is becoming very popular as both a toy (it can be flown by even the most inexperienced pilots), and as a research platform for computer vision and robotic exploration (researchers can focus on their algorithms without worrying that the drone will crash).

And what is ROS?

ROS, the “Robot Operating System”, is a software framework that enables the development of robotic applications – applications that control and interact with robots. A large number of groups using the AR.Drone for research purposes, use ROS to control the drone, which is why we are focusing on it here.

What you’ll need for the tutorial

To complete this tutorial you will need:

1. An AR.Drone!
2. A computer or laptop with WiFi

For future tutorials, you will need:

1. USB Joystick or control pad  (linux compatible)

Getting started

For these tutorials, we will be running the ROS software and AR.Drone controllers using the Linux operating system.

If you don’t have Linux, want to get flying quickly or don’t know what I’m talking about:

To get started, we’ll be downloading an “Xubuntu Virtual Machine”, which has been created especially for these tutorials. This virtual machine package contains everything that you need to get up and flying – fast!

To run the virtual machine package, you will first need to download and install the free Virtual Box software. You will also want to download the ARDroneUbuntu virtual machine package.

Once you have Virtual Box installed, double click the ARDroneUbuntu.ova file that you downloaded. This will open it in the Virtual Box program.

A settings/import dialog should open. You should leave these settings as they are, unless you know what you’re doing.

After completing the import process, click start in the Virtual Box program to begin running ARDroneUbuntu.

Congratulations, you’re now running Xubuntu 12.04 (username: ardrone, password: ardrone)!  This has been preconfigured with all the software required for this series of tutorials, so you’re now good to go!

Once the virtual machine is running, you will be brought to the desktop. At the bottom of the screen you will see the launch bar:

The launch bar contains a number of icons, which I will be referring to throughout the tutorial:

1. Terminal – in this program, you run text-based programs and commands
2. Text Editor – we will be using this in future tutorials to write some basic programs
3. Keyboard Controller – this launches the keyboard controller
4. Update Tutorials – this will update the AR.Drone driver and tutorial source code, it will also install new tutorials once they have been released.
5. Quick Command Box – this is like the terminal and can be used for sending a quick command to the operating system

Great, now that you’re up and running, lets update the tutorial files. Click the Update Tutorials (#4) button in the launch bar. This will launch a terminal window in which it will download and install any required software updates. You should be connected to the internet for this step.

Command-line equivalent (updating the driver):
$ roscd ardrone_autonomy
$ git pull
$ rosmake ardrone_autonomy

Command-line equivalent (updating the tutorial):
$ roscd ardrone_tutorials
$ git pull
$ rosmake ardrone_tutorials

Now you’re ready to fly and can skip the next section of the tutorial.

If you have Linux (skip this section if you just downloaded the virtual machine):

I’ll assume you understand enough about linux to follow the ROS installation instructions (summarized here). I also assume you are running Ubuntu (or a deviant). If not, you should modify the apt-get commands to suit your system.

In addition to the summarized ROS installation instructions above, you will need to run:

$ sudo rosdep init
$ rosdep update
$ sudo apt-get install ros-fuerte-joystick-drivers
$ rosdep install joy

This will install the joystick drivers which are used in a future tutorial.

Now to install the AR.Drone driver’s prerequisites:

$ sudo apt-get install daemontools libudev-dev libiw-dev

Next, we will download the AR.Drone driver (full disclosure: I am a regular contributor to the driver) and the keyboard controller used for this tutorial:

$ roscd
$ pwd
~/ros_workspace
$ git clone https://github.com/AutonomyLab/ardrone_autonomy.git
$ git clone https://github.com/mikehamer/ardrone_tutorials.git
$ ls
...  ardrone_autonomy  ardrone_tutorials  ...
$ rospack profile

And now to compile the packages:

$ roscd ardrone_auto<TAB should autocomplete>
$ ./build_sdk.sh
$ rosmake -a

If TAB does not autocomplete the above-indicated command, or if rosmake reports any failures, you may have a problem with your installation. If this is the case, you should check the installation guide linked above.

If you’ve completed all of the above successfully, you’re ready to move on!

Warning!

We are now going to fly the drone. Flying inside can be dangerous. When flying inside, you should always use the propellor-guard and fly in the largest possible area, making sure to stay away from walls and delicate objects. Even when hovering, the drone tends to drift, so always land the drone if not being flown.

I accept no responsibility for damages or injuries that result from flying the drone.

If you have a laptop and are thus able to fly outside, check the last section of this tutorial for instructions on reconfiguring the drone.

First connection

Great, we’ve got the software, now lets have some fun!

Turn on the drone and wait until the propellors vibrate, indicating that the drone is ready.

Now connect your computer’s wireless to the AR.Drone’s network, just like you would connect to a normal wireless network. The network will be called ardrone<something>.

If your computer is also connected to a network via ethernet (or any other network adaptor), disconnect it. We only want the wireless network to be running*

* Advanced users: The drone acts as a wireless server and assigns itself (via its DHCP server) a fixed IP address of 192.168.1.1 and your wireless card an address from 192.168.1.2-255. For this reason, there can be routing issues if your ethernet card is also in the 192.168.1.0 subnet.

Running the keyboard controller

If you’re using the supplied virtual machine:

Click on the keyboard icon (#3) in the launch bar to run the keyboard controller program:

Note that a connection can take up to 10 seconds. If after 10 seconds the window still says “Disconnected”, check the wireless connection to the drone, and check that any network cables have been unplugged from the computer.

If you’re using a custom Linux installation or want to use the command line:

Open a terminal window and type the command:

roslaunch ardrone_tutorials keyboard_controller.launch

Pressing TAB at various stages throughout the command should autocomplete. If this is not the case, or if this command causes errors, check that you have setup your .bashrc file as per the ROS installation guide, and that you ran rospack profile after cloning the two git repositories.

If successful, this command will print out many startup messages, including:

Exception AttributeError: AttributeError("'_DummyThread' object has no attribute '_Thread__block'",)...

This exception is a Python bug and won’t affect the program.

You will also see a printout:

process[ardrone_driver-2]: started with pid [6178]
===================+> 192.168.1.1

Within a few seconds, the output should progress well past this stage and begin printing configuration messages as the initialization of the AR.Drone progresses. If this is not the case, check the wireless connection to the drone, and check that any network cables have been unplugged from the computer.

Up, up and away!

If the keyboard controller launched successfully, you will now be seeing a live video stream from the drone. If this is the case, we’re ready to fly!

The controls (defined in ardrone_tutorials/src/keyboard_controller.py) are:

• E – Pitch Forward
• D – Pitch Backward
• S – Roll Left
• F – Roll Right
• W – Yaw Left
• R – Yaw Right
• Q – Increase Altitude
• A – Decrease Altitude
• Y – Takeoff
• H – Land
• SPACEBAR – EMERGENCY STOP

Update 17/12:
Note: Pressing the Spacebar will toggle the drone between an emergency state (all onboard LEDs are red), and the ready state (all onboard LEDs are green). If your drone isn’t responding to commands, make sure it is in the ready state!

With that said, I now hand the controls to you. Remember to keep a thumb on the Spacebar at all times, just in case something goes wrong.

Have fun!

Going Further (Added 17/12/2012)

This tutorial was fairly straight forward, the purpose being simply to get you up and running with the right software. Future tutorials will be more technical, however if you’re wanting to scratch the itch, try running some of the following commands at the terminal:

With a controller running:

$ rostopic echo /ardrone/navdata

Will print the data packets that are being sent back from the drone at 50Hz.

$ rosservice call /ardrone/togglecam

Will toggle the drone’s video stream between front and bottom cameras.

$ rxplot /ardrone/navdata/ax:ay /ardrone/navdata/vx:vy

Will plot the x and y acceleration on one plot, and the estimated x and y velocity on a second plot.

Now without a controller running (ie close the keyboard_controller window)

$ roslaunch ardrone_tutorials keyboard_controller_with_tags.launch

This runs a new launch file that enables tag detection. Now run:

$ rostopic echo /ardrone/navdata

And have a look at the various tag detection data that is returned (you have to be holding tags in front of or below the drone for this to work).

By default, this launch file configures the drone to detect Orange-Blue-Orange stickers on the front facing camera, and the A4 “Oriented Roundel” that comes with the AR.Drone 2.0 on the downward facing camera.

If your stickers are a different color, change the launch-file parameter enemy_colors to 1 for Orange-Green-Orange, 2 for Orange-Yellow-Orange and 3 for Orange-Blue-Orange.

Coming up in tutorial #2 – Joystick control

By now you may have noticed that flying with the keyboard isn’t so easy. This is largely due to the binary nature of a keypress (not pressed/pressed), which translates into 0%/100% control. Although we could implement software remedies such as input windup based on how long the key has been held, this is for the most part only a band-aid solution; what we really need is an analogue input device. Enter the Joystick (or control pad).

In the second tutorial of this series, we will be covering a little ROS theory (topics, messages and launch files), before installing a joystick, linking it up to the ROS system and then using it to fly the drone.

Source code and configuration

For those of you interested in the source code or configuration, I will briefly discuss the various options below. We will be going into much more detail in the next tutorial.

Full source code for this tutorial is available from the GitHub repository.

src/drone_controller.py

This file provides an abstracted interface to the AR.Drone driver and shouldn’t need to be modified

src/drone_video_display.py

This file implements the video display window and shouldn’t need to be modified

src/keyboard_controller.py

This file brings everything together. The KeyboardController class implemented in this file extends the DroneVideoDisplay class, adding a keyboard handler for control. Key mappings are defined in this file and can be changed to suit your preferences.

launch/keyboard_controller.launch

This is the default configuration file, which sets the drone’s configuration and launches all required processes

launch/keyboard_controller_outdoor.launch

This configuration file implements a more aggressive set of flight parameters for outdoor flight. If you are wanting to fly outdoors, the keyboard controller can be launched with the command:

roslaunch ardrone_tutorials keyboard_controller_outdoor.launch

For more information about configuring the AR.Drone, have a look at the AR.Drone driver’s readme.

That’s all folks

I hope you’ve had fun with this tutorial and I look forward to bringing you the next one in two weeks time!

If you have any questions, or comments about what you would like to see in the next tutorials, please send me a message or leave a comment to this post! You can see all the tutorials in here: Up and flying with the AR.Drone and ROS.

If you liked this article, you may also be interested in:

• Parrot AR.Drone app harnesses crowd power to fast-track vision learning in robotic spacecraft
• Taking your first steps in robotics with the Thymio II
• Discovering social robotics with the Aisoy1
• Classroom robotics: Motivating independent learning and discovery

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In July, the Parrot Group partnered with senseFly, a Swiss start-up and spin-off from the Swiss EPFL, and invested $5.3 million to help senseFly grow. Their initial plan was to develop a high volume manufacturing capability and fund rapid growth and new product development. The eBee is the first such product from that venture. The just announced 3′ wingspan eBee can fly for 45 minutes in 45km/hr winds, and will enable users to take their own aerial photos and produce precise ortho-mosaics and 3D models. senseFly has also established an impressive global network of resellers which presently excludes the US (because of FAA regulatory limitations). The eBee meets the special needs of mining and surveying professionals because of its wind resistance and robustness, high precision 3D image processing and mission planning software.

senseFly, working with another Parrot Group partner and another EPFL spin-off (with a similarly intentioned $2.5 million equity investment from Parrot to spur growth and development), Pix4D, has developed a suite of software which can create a flight path and take 2D geotagged images and then process them into 3D maps.

Below is an EPFL/Pix4D video displaying additional information about the process of filming, tagging and creating 3D maps and images.

When senseFly’s UAVs are allowed into civilian airspace, there are a myriad uses:

• Aerial photography
• Mapping
• Mine and quarry surveys
• Emergency response
• Agricultural and cattle monitoring
• Remote sensing
• Surveying (general and archeological)
• Urban and regional planning
• Infrastructure management
• Etc.

 

As civilian airspace opens to drones like those from senseFly, the sky begins to offer unique perspectives to view and understand our environment. One can see a near-term future where images and data can be captured at low cost, anywhere, any time, and without complex infrastructure or long preparation time and transformed into useful management tools, maps and surveys.

 

“We are witnessing a revolution whereby flying drones are no longer exclusive to the military but are quickly spreading into the civilian world. In 2010, Parrot surprised competitors and fellow researchers alike when they hit the market with their iPhone-piloted quadcopter for filming and gaming. The same year, senseFly revolutionized the job of surveyors by providing fully autonomous flying cameras capable of producing precise 2D and 3D maps within minutes. Both companies are focused on developing very lightweight drones with the strong belief that this is critical to their use by the public at large – far beyond the military”, says Jean-Christophe CEO and co-founder of senseFly.

 

Researchers at the Exertion Games Lab at RMIT University in Melbourne, Australia have created a robot to support you while exercising. Joggobot is a modified version of the popular AR Drone quadrocopter platform developed by French company Parrot. The robot will track a marker pattern printed on your t-shirt and fly ahead of you when you go out for a run. The researchers describe Joggobot as an “exertion game”. They believe that jogging is play – we are not jogging to get from A to B, but for the experience of jogging – and point out that jogging with a physical device that reacts to its environment and, similar to a human jogger, has a limited amount of energy for exercise creates a very different interaction experience than pure audio-visual stimuli such as aerobic videos. They hope that the robot can improve the jogging experience and enhance our understanding of why we jog (and hence why we do not jog enough).

In July of 2008, the app store concept was launched by Apple and it has dramatically changed the mobile industry. Over 1.2 billion apps were downloaded between Christmas and New Year according to Flurry Analytics. That’s about 60 percent higher than every week of December before Christmas, and marks the largest number of apps ever downloaded in a single week.

Robot app stores are now beginning to happen as well and visions of apps for tele-robotic safety and surveillance pop into mind, or light domestic duties, or…? My imagination runs wild at the thought and I imagine these new apps being easy to use, having multiple functions and a low cost.

But that is then and now is now. At present there are three significantly different approaches to an app store for robots (although additional “stores” are in the planning stage and many existing stores are making arrangements to expand beyond their own boundaries (like iRobot’s online store and Willow Garage’s application chooser)):

1. MyRobots, by Canadian RobotShop, has opened a combination app store and online environment to permit robotics products to be monitored and updated seamlessly as well as providing a Cloud repository for applications that run from the Cloud instead of the robot itself. MyRobots’ slogan is “Connect your robots; reap the benefits.” The online monitoring aspect of MyRobots is similar to the MyMotoman Remote Monitoring Service initiated my Yaskawa Motoman America in August.
2. Robots App Store, by San Francisco serial entrepreneur Elad Inbar, is developing their store to be an online marketplace for selling apps and also providing developers their own sales portal. Robots App Store, in addition to attempting to trademark the hyphenated words “Robot-app”, has a wonderfully informative info-graphic explaining the need and market for robot apps. What it doesn’t explain is how the process will be different than what most of us have come to expect when buying apps from Apple’s App Store. It involves tinkering with development tools and operating system platforms – a turnoff for all but the most avid hobbyist.
3. RoboEarth, an EU public-private consortium, is using the Internet to create a giant open source network and database repository that can be accessed and updated by robots around the world. Focused more on academia and industrial applications, RoboEarth speeds up robot learning and adaption in complex tasks and can execute tasks that were not explicitly planned for when the robot was designed.

Service robotics – as differentiated from industrial or defense/security/space robotics – needs to provide inexpensive and efficient assistance to humans to be profitable in the consumer marketplace and emerge as a serious industry. We already have vacuum robots and adaptive cruise control bots, but there are thousands of other possible applications. Entertainment, embedded systems, eldercare and home assistance are certain to be amongst the first groups to benefit as the service robotics industry materializes.

There are two different approaches to providing those services:

C3P0 The Ultimate Humanoid Robot

1. A universal robotic tool that will be able to perform all the tasks we ask it to do, including social communication – a robot like C3P0. This is the goal of humanoid robots. At this time, humanoid robots are costly, difficult to control, slow and intrusive. Although human-robot communication is part of the lure, and there is much progress in that area, we are at least five years away from devices that not only speak but understand. IBM’s Watson and Apple’s Siri are proving that it can be done but their software isn’t yet available to front-end robotic applications and there is, at present, little comparable in the robotics world (although the recent ABC Nightline interview of Hanson Robotics’ Philip Dick robot shows there is serious ongoing development). Even when the two-way communication aspect is further developed and made available to roboticists, it will be many more years until the goal of a C3P0-like sophisticated humanoid robot will be ready.

• A more near-term and progressive solution is to forego universality and design specific devices that provide limited, task-specific means to communicate and complete specific tasks… think Roomba.

Certainly #2 has more near-term promise and is where most of the market is focused for the rest of this decade. As service robots begin to meet the four core criteria of low cost, low intrusiveness, safety and modularity – which is happening almost daily with technology changes – an incredible number of opportunities for service robot apps can be foreseen. The key will be to make them practical, affordable and acceptable by humans as part of their day-to-day life.

Robots App Store:

At discussions with Elad Inbar, CEO of Robots App Store, I asked about the size of the marketplace and was surprised to find that there are 14 million service robots out there now, and it’s estimatited that by the end of 2014 there will be 24 million. If only 15% of them were able to buy two apps at an average price of $9.99, that’s $72 million in revenue. If they each bought five apps instead of two, the figure would be $180 million. If 20% purchased five apps it would be a $¼ billion business. How real and applicable those figures are depends on many complex variables: willingness of the buyer to tinker, quality of the apps, flexibility of the robots, price, disposable income, etc. In fact, apps for robots with an Apple or Android front end will not be available in the Robots App Store but will onlybe available through the Apple (or Android) App Store – which limits the market metrics… an example of this Apple exclusivity is the hundreds of thousands of Parrot’s AR.Drone quadcopters.

Regardless of the metrics, the process of developing and sharing apps can provide a giant leap toward rejuvenating the robots we have and teaching the audience (the buyers) that they can get what they want from their robot(s). Thus the buyer becomes more educated and demanding and the producers less capricious — and more profitable.

From an infographic by Robots App Store showing possible app prices for various robots

Inbar’s Robots App Store is “focused solely on the personal and service robots marketplace because “industrial robots are very limited in their functionality… not to mention that they are oriented to quickly repeat the same tasks 24/7 unlike personal and service robots that can be extended to virtually everything! I believe in the consumers market – as they will grow like a hockey-stick!”

When I asked what kind of apps we will see once the store opens, Elad said, “We have more than 100 approved robotic developers working on apps for Romba, Pleo, AIBO, NAO, DARwin and others arranged in categories like Entertainment (dances, etc.), Story Telling (for kids), Utilities (like home search engines), Tools (like face recognition and such) and many others.”

Finally, I asked about the mechanics of porting an app or game from one robot to another. “ROS by Willow Garage, Gostai, URBI and others are trying to build a standard open platform OS that will run across all robots. We, as a marketplace, don’t take sides in this battle. We are serving every robot, ever developer, and every OS. But surely we’ll have to take sides one day when the critical mass will decide which is the widely spread and used OS…. Personally I believe that ROS is in the best position to become the de-facto standard and I personally love to work with their code. It is easy, simple and their libraries can be ported quickly between different robots – a huge advantage for every robot apps developer.”

MyRobots:

The goal for MyRobots is to get all robotic platforms to collaborate and operate together (regardless of their operating system) via the use of Cloud robotics. Although like other stores MyRobots will sell downloadable apps too, their main interest is to develop applications that will run on the Cloud thereby freeing computation time and resources on the robot and will make a single app compatible with many different robotic platforms.

Says MyRobot’s Mario Tremblay:

Cloud Robotics is the application of the cloud computing concept to robots. This means using the Internet to augment the robots capabilities by off-loading computation and providing services on demand. Being connected to the cloud also helps robots to collaborate with other machines, smart objects and humans. Through this collaboration, robots transcend their physical limitations and become more useful and capable since they can delegate parts of their tasks to more suitable parties. By combining increased communication capabilities to the ability and flexibility of running and storing part of their intelligence (i.e. software, behaviors and apps) on the MyRobots infrastructure, smart objects and robots become augmented and constitute a new and revolutionary concept for the future of robotic intelligence: Cloud Robotics.

RoboEarth:

RoboEarth is an EU-funded consortium of educators, research facilities and corporations working together to provide a world-wide-web for robot applications. RoboEarth, in development for the last two years, can be seen as a giant network and database repository where robots share information and learn from each other about their behavior and environment thereby showing a better way of robot collaboration and information sharing. RoboEarth’s premise is to out-source application development by tapping creativity and experience from wherever they can. Further, they believe (as do I) that more apps sold equates to more robots sold. Quoting Markus Waibel:

The central beauty of the App Store idea is that it enables a feedback cycle: The more devices (phones/robots) you sell, the more lucrative the app store becomes for 3rd party developers to program new apps. And the more new apps, the more devices you sell. In my opinion, robotics does not yet have critical mass to jump start this at the present time.

RoboEarth is fast becoming a large shared repository for maps, object models, task descriptions, robot models, semantic relations, probabilistic relations and “action recipes” (apps) within the global academic community. It remains to be seen whether the big robot manufacturers will embrace the technology for their products.

One unintended consequence of RoboEarth’s efforts may be within the used robots marketplace. Imagine the financial gain if an older, unused robot is rejiggered to have new equipment as well as easy to train software to put the robot back to work on new projects. Another possible beneficiary is the end user who will be less likely to retain an intergrator/consulting service if the robot can handle ongoing changes on its own.

The ultimate goal of RoboEarth is to enable robots to share and store what they discover about the world on their own Internet.

As a personal aside, I live in a rural area outside of a medium-sized city. 4G LTE is planned for 2012-2013 for the city but not my area and I have to pay for a very unreliable T1 line to my office. It’s hard to imagine having a service robot doing its thing in my world while relying on the Cloud (as represented by my T1). Maybe the concept will grow on me with time but right now I’m wary and a bit afraid of the consequences.

What do you want from a robot app store? Let’s compile a long list of “wants” and send them off to these app stores and see whether they can get them for us. Why should we wait to see if our app magically appears in their store? Let’s ask for what we want. Send them to me and I’ll combine them all and send them off to the stores.

Here’s what I would like:

• For my old Pleo, anything that makes it do things it doesn’t already do.
• For my AR.Drone, a programmable camera shoot over a grid that I draw onscreen from a starting point where I place the drone.
• For my little iSobot, a complicated routine of fetch, grab and return.
• And for my new Swivl, something sexy where it follows my girlfriend around the room and refocuses as the distance changes.

But I also want to buy a used two-armed Yaskawa Motoman and put it to work in my workshop doing ad hoc tasks. I’d love to be able to add a hardware component add-on (or two), be able to use ROS or some simulation software to test my apps, buy some additional apps that do the kind of tasks I need to do in the shop, and have all that work seamlessly and without line-for-line coding, excessive cost or too much fiddling.

What do you want? Send them to me and I’ll combine them all and send them off to the stores.