Nico, Emil, and Moritz founded ReRun with the mission of making powerful visualization tools free and easily accessible for roboticists. Nico and Emil talk about how these powerful tools help debug the complex problem scopes faced by roboticists. Tune in for more.

Nikolaus West
Co-Founder and CEO
Niko is a second-time founder and software engineer with a computer vision background from Stanford. He’s fanatic about bringing great computer vision and robotics products to the physical world.

Emil Ernerfeldt
Co-Founder and CTO
Emil fell in love with coding over 20 years ago and hasn’t looked back since. He’s the creator of egui, an easy-to-use immediate mode GUI in Rust, that we’re using to build Rerun. He brings a strong perspective from the gaming industry, with a focus on great and blazing fast tools.

Links

• ReRun
• Download mp3
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

The team at Retrocausal built a computer vision platform that allows any manufacturer to rapidly set up an activity recognition pipeline to assist with manual assemblies. Using any standard camera pointed at an assembly station, the system can learn the assembly procedure and give real-time feedback.

Deci:

At Deci, they are tackling the problem of taking a Machine Learning model optimized for a single hardware platform and adapting it to work on a different hardware platform.

When developing an ML model running on the edge, a time-consuming development cycle is made to optimize the model to run at peak capacity. However, over time hardware platforms get updated and companies need to make the decision of whether or not to update their hardware and dedicate engineering resources to manually tweak a multitude of settings.

Deci allows companies to upload their models to the cloud, then automatically optimize them for running on a variety of different hardware platforms that they own in their facilities.

StegAI:

Dr. Eric Wengrowski co-founded StegAI after publishing a paper in CVPR 2019 on embedding “fingerprints” that are embedded in images and videos that are invisible to the human eye, but detectable by their software.

Their product modulates the pixels in an image in such a way that even after compressing, resizing, or printing out the images, their deep learning algorithms will still be able to detect the origin of the image.

This powerful tool is used by social media firms to protect against potential copyright infringement when users upload content to their platforms

Marine environments present a unique set of challenges that render several technologies that were developed for land applications completely useless. Communicating using sound, or at least as people use sound to communicate, is one of them.

Michael Fulton tackles this challenge with his presentation at ICRA 2022 by using body language to communicate with an AUV underwater. Tune in for more.

His poster can be viewed here.

Michael Fulton

Michael Fulton is a Ph.D. Candidate at the University of Minnesota Twin Cities. His research focuses primarily on underwater robotics with a focus on applications where robots work with humans. Specifically, human-robot interaction and robot perception using computer vision and deep learning, with the intent of creating systems that can work collaboratively with humans in challenging environments.

At ICRA 2022, the researchers weren’t the only ones working with cutting-edge technology. We spoke to the exhibitors to get real-life demos of their products.

Tangram Vision

Tangram Vision is a hardware-agnostic sensor fusion platform. It streamlines the development and deployment of critical sensor infrastructure like calibration, fusion, and monitoring for any number of cameras, depth sensors, LiDAR, radar, and IMU. Their co-founder, Adam Rodnitzky, walks us through their sensor fusion platform.

FLX Solutions

Matt Bilsky, Founder and CEO of FLX Solutions gives us a live demo of their robot, the FLX BOT. Matt Bilsky applied his Ph.D. in Mechanical Engineering to create a novel, highly compact robot that is designed to reach and inspect parts of a building that a human cannot reach. The FLX BOT is one inch in diameter and is made of modular links that can be attached one after the other to extend the reach and degrees of freedom of the robot.

Exyn Technologies

Exyn Technologies specializes in aerial robotics, drone swarms, multi-modal sensor fusion, 3D mapping, obstacle avoidance, and autonomous navigation & planning.

Exyn’s focus is on developing software for aerial robots so they can operate in GPS-denied environments, without human control, prior information, or pre-existing infrastructure (e.g. no motion capture system).

Pollen Robotics

Pollen Robotics developed the robot, Reachy, an open-source humanoid robot with a quirky appearance. Reachy’s primary userbase is researchers studying fields such as teleoperation. Using mixed autonomy, Reachy is able to be teleoperated by humans, while still having onboard intelligence to autonomously infer the actions you want it to do.

Underwater Autonomous Vehicles face challenging environments where GPS Navigation is rarely possible. John McConnell discusses his research, presented at ICRA 2022, into fusing overhead imagery with traditional SLAM algorithms. This research results in a more robust localization and mapping, with reduced drift commonly seen in SLAM algorithms.

Satellite imagery can be obtained for free or low cost through Google or Mapbox, creating an easily deployable framework for companies in industry to implement.

Links

• Download mp3 (12.2 MB)
• Sonar SLAM opensource repo
• Research Paper
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

In this episode, Audrow Nash speaks to Tim Chung, Program Manager in the Tactical Technology Office at the Defense Advanced Research Projects Agency (DARPA), on the DARPA Subterranean (SubT) Challenge. The SubT Challenge is a robotics challenge that aims to develop innovative technologies that would augment operations underground. In this conversation, they talk about the motivation of the SubT Challenge, its systems (hardware) and virtual (simulation) challenges, how the resulting technology has been shared with the world, on benchmarking the robotics behaviors in simulation, on the challenge of scoping the SubT Challenge, and on building the final environment for the systems challenge.

Episode Links

• Download the episode
• The SubT Challenge’s Website
• Tim’s LinkedIn
• DARPAtv’s YouTube Channel
• SubT’s Repository on OSRF

Podcast info

• Podcast website
• Apple Podcasts
• Spotify
• Podcast RSS feed
• Full episodes
• Clips

In this episode, Audrow Nash speaks to Steve Macenski, who is the Open Source Robotics Engineering Lead at Samsung Research America. Steve leads the Nav2 project, which is an open source robot navigation framework. In this conversation, they talk about the problem and challenges of robot navigation, how Nav2 works at a high level, hybrid planners, and about Steve’s experience working with the Nav2 community.

Episode Links

• Download the episode
• Nav2’s Website
• Steve’s Website
• Steve’s LinkedIn
• Samsung Research’s Website

Podcast info

• Podcast website
• Apple Podcasts
• Spotify
• Podcast RSS feed
• Full episodes
• Clips

Amit Goel, Director of Product Management for Autonomous Machines at NVIDIA, discusses the new collaboration between Open Robotics and NVIDIA. The collaboration will improve the way ROS and NVIDIA’s line of products such as Isaac SIM and the Jetson line of embedded boards operate together.

NVIDIA’s Isaac SIM lets developers build robust and scalable simulations. Dramatically reducing the costs of capturing real-world data and speeding up development time.

Their Jetson line of embedded boards is core to many robotics architectures, leveraging hardware-optimized chips for machine learning, computer vision, video processing, and more.

The improvements to ROS will allow robotics companies to better utilize the available computational power, while still developing on the robotics-centric platform familiar to many.

For the developers out there, check out the ISAAC ROS Github repository to implement this yourself.

Amit Goel

Amit Goel is Director of Product Management for Autonomous Machines at NVIDIA, where he leads the product development of NVIDIA Jetson, the most advanced platform for AI computing at the edge.

Amit has more than 15 years of experience in the technology industry working in both software and hardware design roles. Prior to joining NVIDIA in 2011, he worked as a senior software engineer at Synopsys, where he developed algorithms for statistical performance modeling of digital designs.

Amit holds a Bachelor of Engineering in electronics and communication from Delhi College of Engineering, a Master of Science in electrical engineering from Arizona State University, and an MBA from the University of California at Berkeley.

Links

• Download mp3 (29.8 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

In this episode, Audrow Nash interviews Dave Coleman, Chief Executive Officer at PickNik Robotics. Dave speaks at a high level about what MoveIt is and what problems it helps roboticists solve, they talk about supervised autonomy, including a collaboration with NASA and MoveIt Studio, and Dave talks about MoveIt 3.0.

Episode links

• Download the episode
• Dave Coleman’s LinkedIn profile
• PickNik’s website
• MoveIt’s website
• MoveIt Studio’s website

Subscribe

• YouTube
• Apple Podcasts
• Spotify
• Google Podcasts
• Overcast
• Pocket Casts

Anthony DiMare and Charles Chiau deep dive into how Bedrock Ocean is innovating in the world of Marine Surveys. At Bedrock Ocean, they are developing an Autonomous Underwater Vehicle (AUV) that is able to map the seafloor autonomously and at a high resolution. They are also developing a data platform to access, process, and visualize data captured from other companies at the seafloor.

Bedrock Ocean is solving two problems in the industry of Marine Surveying.
1. The vast majority of the seafloor is completely unmapped
2. The data that is captured from the seafloor is not standardized or centralized.

Seafloor data conducted by two different companies with the same or different hardware to capture the data can vary significantly in the calculated seafloor profile

Anthony DiMare
Anthony previously founded Nautilus Labs, a leading maritime technology company advancing the efficiency of ocean commerce through artificial intelligence. While at Nautilus, Anthony helped global companies solve challenges with distributed, siloed maritime data systems and built the early team that launched Nautilus Platform into large publicly listed shipping companies.

Charles Chiau
Charles, Bedrock’s CTO, was previously at SpaceX where he helped design the avionics systems for Crew Dragon. He also was a system integration engineer at Reliable Robotics working on their autonomous aviation system and was the CTO of DeepFlight where he worked on manned submersibles including ones for Tom Perkins, Richard Branson, and Steve Fossett.

Links

• Download mp3 (48.0 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

RE2 Sapien 6M Robot Arm

In this episode, Shihan Lu interviews Travis Schneider, Business Development Manager at RE2 Robotics, focusing on their work on mobile outdoor manipulation. Travis introduces several robotic products and services of RE2, including RE2 Sapien robotic arms and human-centered system integration, and how they can be used to help rather than replace workers. He also shares his thoughts about challenges in human-robot collaboration.

Travis Schneider
Travis Schneider, Business Development Manager at RE2 Robotics (RE2), works to foster commercial B2B partnerships with groups interested in leveraging RE2’s robotic technology for applications in a wide variety of markets, including aviation, construction, energy, and medical. With a background in Mechanical Engineering, Travis has an intimate knowledge of many of the fundamentals associated with robotic systems including electromechanical design, motion control, and associated software.

Links

• RE2 Robotics
• Download mp3 (12.7 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

What is the best simulation tool for robotics? This is a hard question to answer because many people (or their companies) specialize in one tool or another. Some simulators are better at one aspect of robotics than at others. When I’m asked to recommend the best simulation tool for robotics I have to find an expert and hope that they are current and across a wide range of simulation tools in order to give me the best advice, which was why I took particular note of the recent review paper from Australia’s CSIRO, “A Review of Physics Simulators for Robotics Applications” by Jack Collins, Shelvin Chand, Anthony Vanderkop, and David Howard, published in IEEE Access (Volume: 9).

“We have compiled a broad review of physics simulators for use within the major fields of robotics research. More specifically, we navigate through key sub-domains and discuss the features, benefits, applications and use-cases of the different simulators categorised by the respective research communities. Our review provides an extensive index of the leading physics simulators applicable to robotics researchers and aims to assist them in choosing the best simulator for their use case.”

Simulation underpins robotics because it’s cheaper, faster and more robust than real robots. While there are some guides that benchmark simulators against real world tasks there isn’t a comprehensive review. A more thorough review can address gaps and needs in research and research challenges for simulation. The authors focus on seven sub-domains: Mobile Ground Robotics; Manipulation; Medical Robotics; Marine Robotics; Aerial Robotics; Soft Robotics and Learning for Robotics.

I’m going to cut to the chase and provide a copy of the final comparison tables of each sub-domain but for anyone interested in utilizing these recommendations, then I recommend reading the rationale behind the rankings in the full review article. The authors also consider whether or not a simulator is actively supported. Handy to know! And the paper is also an excellent source of information about various historic and current robotics competitions.

Mobile Ground Robotics:

TABLE 1 Feature Comparison Between Popular Robotics Simulators

TABLE 2 Feature Comparison Between Popular Robotics Simulators Used for Mobile Ground Robotics

Manipulation:

TABLE 3 Feature Comparison for Popular Robotics Simulators Used for Manipulation

Medical Robotics:

TABLE 4 Feature Comparison of Popular Robotics Simulators Used for Medical Robotics

Marine Robotics:

TABLE 5 Feature Comparison of Popular Simulators Used for Marine Robotics

Aerial Robotics:

TABLE 6 Feature Comparison of Popular Simulators Used for Aerial Robotics

Soft Robotics:

TABLE 7 Feature Comparison of Popular Simulators Used for Soft Robotics

Learning for Robotics:

TABLE 8 Feature Comparison of Popular Simulators Used in Learning for Robotics

Conclusions:

As robotics makes more use of deep learning, simulators that can deal with data on the fly become necessary, and also a potential solution for simulation problems regarding points of contact or collisions. Rather than utilize multiple simulation methods to make a clearer abstraction of the real world in these boundary situations, the answer may be to insert neural networks trained to replicate the properties of difficult phenomena into the simulator. There is further discussion on differentiable simulation, levels of abstraction and the expansion of libraries, plug-ins, toolsets, benchmarking and algorithmic integration, all increasing both the utility and complexity of simulation for robotics.

As the field of simulation for robotics grows, so does the need for metrics that capture the accuracy of the real world representation.  “Finally, we predict that we will see further research into estimating and modeling uncertainty of simulators.”

This may have been the first review article on simulation for robotics but hopefully not the last. There’s a clear need to study and measure the field. I found the sections on soft robotics and learning for robotics particularly interesting, as the paper discussed the difficulties of simulation in these fields. And please attribute any errors in this summary to my mistakes. Read the full review here: https://ieeexplore.ieee.org/document/9386154

In this episode, Lauren Klein interviews Muralidharan Arikara, CEO of Xarpie Labs. Xarpie Labs creates visualization and simulation experiences within the retail, healthcare, and defense industries. Arikara describes how Xarpie Labs grew as part of the Machani Group, which has decades of experience in automotive manufacturing, into an innovator in virtual and augmented reality. He elaborates on the role of Xarpie Lab’s virtual reality and augmented reality experiences in allowing real estate customers a vision of properties from a distance. Arikara also paints a picture of Xarpie Labs’ augmented reality tools, including a project for a museum to visualize old gramophones for visitors and an air conditioning troubleshooting tool for technicians.

Muralidharan Arikara

Muralidharan Arikara is an entrepreneur and the CEO of Xarpie Labs. Arikara is an engineer by training, and worked in the clean energy industry prior to his current role. In addition to his role as CEO at Xarpie Labs, he is also a member of the advisory board of several startups in the energy and consumer segments.

Links

• Download mp3 (16.8 MB)
• Xarpie Labs Virtual Employee Onboarding Program
• LinkedIn
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, we hear from Luca Colasanto, Senior Robotic Scientist at Realtime Robotics, about real-time robot motion planning in dynamic and complex environments with human-robot collaboration. Realtime Robotics focuses on accelerating conventional motion planning through optimization of algorithms and hardware to allow safe use of robotic tools in work areas with humans. Luca spoke to our interviewer Kate about Realtime Robotic’s fast motion planning technology, including key aspects, such as perception, algorithms and custom hardware.

Luca Colasanto

Luca Colasanto is a Sr. Scientist at Realtime Robotics focusing on AI-based grasping and multi-robot optimization. Luca completed his PhD in Humanoid Robotics at Italian Institute of Technology, focusing on control systems for bipedal walking machines and compliant actuators. During the past ten years, Luca has worked with successful startups to design and prototype new robotics products, and with science and technology leaders, such as Robert Bosch LLC (CA), Institute for Human and Machine Cognition (FL), and École Polytechnique Fédérale de Lausanne (Switzerland), He is the listed inventor in two patent applications and the author of several scientific publications. Luca speaks four languages and is a keen traveler, having visited or lived in more than 20 countries.

Links

• Download mp3 (15.1 MB)
• Realtime Robotics (Videos, Twitter, LinkedIn)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Audrow Nash speaks with Dave Coleman, CEO of PickNik Robotics, about the open source robotics manipulation platform called MoveIt. Coleman talks about MoveIt’s story, from inception and the early days to development and maintenance, as well as how MoveIt relates to the Robot Operating System (ROS) and their move to support ROS-2. He also speaks about MoveIt’s implementation, including global versus local planners and what that means. Coleman concludes by talking about World MoveIt Day and how those interested can begin learning MoveIt and contributing.

Here is a 2017 montage of how MoveIt has been used:

Dave Coleman

Dave Coleman completed his PhD in Computer Science at CU Boulder focusing on motion planning and his B.S. at Geogia Tech in Mechanical Engineering. Coleman has 12 years experience working in the field of robotics automation and is a leader in the open source MoveIt and ROS communities. His insights into robot-agnostic platforms for different morphologies, theoretical approaches, and different end-user requirements give him a well-rounded understanding for powerful robotic software. He has worked and consulted for all types of robotics companies including Google Robotics, Open Robotics, and Willow Garage.

Links

• Download mp3 (17.8 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Shihan Lu interviews Dr. Heather Culbertson, Assistant Professor in the Computer Science Department at the University of Southern California, about her work in haptics. Dr. Culbertson discusses the data-driven realistic texture modeling and rendering, haptic technologies in the social touch, the combination of haptics and robots, expectations and obstacles of haptics in the next 5 years.

Heather Culbertson

Heather Culbertson is a Gabilan Assistant Professor of Computer Science at the University of Southern California. Her research focuses on the design and control of haptic devices and rendering systems, human-robot interaction, and virtual reality. Particularly she is interested in creating haptic interactions that are natural and realistically mimic the touch sensations experienced during interactions with the physical world. Previously, she was a research scientist in the Department of Mechanical Engineering at Stanford University where she worked in the Collaborative Haptics and Robotics in Medicine (CHARM) Lab. She received her PhD in the Department of Mechanical Engineering and Applied Mechanics (MEAM) at the University of Pennsylvania in 2015 working in the Haptics Group, part of the General Robotics, Automation, Sensing and Perception (GRASP) Laboratory.

Links

• Download mp3 (14.8 MB)
• Culbertson’s research website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Lauren Klein interviews Human-Robot Interaction researcher Patrícia Alves-Oliveira. Alves-Oliveira tells us about the upcoming RSS Pioneers workshop at the 2020 Robotics: Science and Systems Conference; the workshop brings senior PhD students and postdoctoral researchers together to collaborate and discuss their work with distinguished members of the robotics field. She also describes her own research designing robots to encourage creativity in children.

Patrícia Alves-Oliveira

Patrícia Alves-Oliveira is an upcoming postdoctoral researcher working with Professor Maya Cakmak in the Human-Centered Robotics Lab at the University of Washington. She recently completed her PhD in Human-Robot Interaction at the Lisbon University Institute supervised by Profs. Ana Paiva and Patrícia Arriaga, also working as a visiting scholar in the Human-Robot Collaboration and Companionship Laboratory at Cornell University, supervised by Professor Guy Hoffman. Patrícia’s research focuses on the use of social robots as intervention tools to enrich creative behaviors in children. Patrícia’s overarching goal as a Human-Robot Interaction researcher is to investigate how and where robots can be used to empower innate human qualities and experiences.

Links

• Apply to RSS Pioneers
• Download mp3 (15.0 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Ron Vanderkley interviews Mark Pivac, Chief Technical Officer and co-founder of FBR (formerly Fastbrick Robotics) about the world’s first end-to-end autonomous bricklaying robot, ‘Hadrian X’. Three years after his first interview, we catch up with Pivac to see how FBR has expanded its operation and chat about their latest commercial prototype, ‘Hadrian X’, as well as the future of the robotic construction industry.

Mark Pivac is the primary inventor of FBR’s automated bricklaying technology. He is an aeronautical and mechanical engineer with over 25 years’ experience working on the development of high technology equipment ranging from lightweight aircraft to heavy off-road equipment. He has 20 years’ experience working with pro/engineer 3D CAD software as well as high-level mathematics, including matrix mathematics, robot transformations and vector mathematics for machine motion. Mark has also worked extensively with design, commissioning and fault finding on servo controlled motion systems achieving very high dynamic performance.

Links

• FBR Website
• Marc Pivac Profile Page
• Hadrian X in action (Video)
• Download mp3 (13.3 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Audrow Nash interviews Brian Gerkey, CEO of Open Robotics about the Robot Operating System (ROS) and Gazebo. Both ROS and Gazebo are open source and are widely used in the robotics community. ROS is a set of software libraries and tools, and Gazebo is a 3D robotics simulator. Gerkey explains ROS and Gazebo and talks about how they are used in robotics, as well as some of the design decisions of the second version of ROS, ROS2.

Brian Gerkey

Brian Gerkey is the CEO of Open Robotics, which seeks to develop and drive the adoption of open source software in robotics. Before Open Robotics, Brian was the Director of Open Source Development at Willow Garage, a computer scientist in the SRI Artificial Intelligence Center, a post-doctoral scholar in Sebastian Thrun‘s group in the Stanford Artificial Intelligence Lab.  Brian did his PhD with Maja Matarić in the USC Interaction Lab.

Links

• Download mp3 (17.4 MB)
• Open Robotics website
• Brian’s personal website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Audrow Nash interviews Bernt Børnich, CEO, CTO, and Co-founder of Halodi Robotics, about Eve (EVEr3), a general purpose full-size humanoid robot, capable of a wide variety of tasks.  Børnich discusses how Eve can be used in research, how Eve’s motors have been designed to be safe around humans (including why they use a low gear ratio), how they do direct force control and the benefits of this approach, and how they use machine learning to reduce cogging in their motors.  Børnich also discusses the longterm goal of Halodi Robotics and how they plan to support researchers using Eve.

Below are two videos of Eve. The first is a video of how Eve can be used as a platform to address several research questions.  The second shows Eve moving a box and dancing.

Bernt Børnich

Photographer: Tonje Kornelie

Bernt Børnich is the CEO and CTO of Halodi Robotics, and the inventor of their Revo1 motors. He holds a bachelor of robotics and nano-electronics from the University of Oslo.

Links

• Download mp3 (16.4 MB)
• Halodi’s homepage
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

The pilot projects

The three companies’ accomplished voucher work varies a lot and sets the basis to enlarge the array of industrial challenges tackled in the pilot work of ROBOTT-NET.

Linak’s pilot project will address some of the barriers that are still hindering the introduction of robots in High-Mix-Low-Volume (HMLV) production. This pilot will focus on the development of a modular assembly robot solution called MARS, that can be introduced in a wide-range of industries. You can check out Linak’s voucher project here:

Irabia’s pilot project UrbanPestPatrol aims to use robots for pest management in industrial facilities. The purpose of the project is to minimize the risk of a pest infestation and its resulting cost. You can learn about Irabia’s voucher project here:

https://www.youtube.com/watch?v=1zmHDOa1ErU

Lastly, Nissan will be looking into automated kitting for complex and diverse applications. This pilot will build a robotic test cell that enables Nissan Motor Manufacturting UK to develop automated kitting capability for its part families. Find out more about Nissan’s voucher project here:

ROBOTT-NET pilots help companies develop their voucher work through proof-of-concept level and accelerate it towards commercialisation. A pilot project will be a medium-scale research prototype installation, lasting for up to 18 months developing the robot technology and business case explored in the voucher stage and applying it to an industrial application demonstration at an end user’s site.

If you want to learn more about the pilot projects, you can check out ROBOTT-NETs website.

In this episode, Audrow Nash interviews Shuo Yang about DJI’s RoboMaster first-person shooter (FPS) competition, a competition designed to get people excited about robotics. For the competition, university teams build and program a robot to go against DJI’s robots in a shooting battle. Each robot has a way of propelling marble-sized plastic balls and pressure sensors on their sides to register if they’ve been hit by an opponent’s projectile. Shuo speaks about the goals of the competition, the teams that are involved, what strategies the teams use, the difficulties the team had in making their robot’s good competitors, the future of the challenge, and how people can get involved.

Note that RoboMaster’s final tournament for the FPS competition starts on July 20, 2018 and goes until July 28, 2018.

Shuo Yang

Shuo Yang is the Director of Intelligent Navigation Technologies at DJI. He obtained B.Eng and M.Phil degrees from Hong Kong University of Science and Technology (HKUST). He is involved in developing flight control and navigation technologies for several DJI flagship products, such as the Inspire 1, Phantom 4 and Matrice 100 drones and the A3 flight controller. He has coauthored 4 academic papers and obtained near 10 US patents. Shuo is also leading an educational robotics competition project called RoboMaster at DJI.

Links

• Robomaster’s Official Website
• Robomaster’s Livestream Platform
• Download mp3 (15.6 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

Image from the South China Morning Post

In this episode, Audrow Nash speaks with Karl Weaver (魏卡爾), formerly the Original Equipment Manufacturer Business Development Director for Oasis Smart SIM. Weaver discusses how wearable technology is growing as a form of payment system in China. He speaks about wireless technology, including Near-Field Communications (NFC) and Embedded SIM cards (eSIM), in wearable technology and in other applications, such as bike rental.

Karl Weaver (魏卡爾)

Since the recording of this interview, Karl has begun working for ARM promoting eSIM and iUICC to Greater China and Asia.

Karl Weaver (魏卡爾) previously worked as the Original Equipment Manufacturer Business Development Director (North America and Northeast Asia) for Oasis Smart SIM. Prior to Oasis, Karl worked for Rivetz Corp, a Massachusetts-based start-up to promote developer tools for design-in of TEE-enables applications on Smartphones for payment and security. Karl also spent 5 years working in China for Gemalto (and Trustonic) as liaison and evangelist of embedded Mobile Near-Field Communications Payments & TEE security technologies to the OEM Smartphone/Tablet PC ecosystem. Weaver has a B.S degree in Business Management from Salve Regina University and a Certification in Mandarin Chinese Language, Customs, and Culture from National Taiwan Normal University.

Links

• Download mp3 (17.5 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

Image: ICRA 2017

In this episode, Audrow Nash interviews several companies at the International Conference for Robotics and Automation (ICRA). ICRA is the IEEE Robotics and Automation Society’s biggest conference and one of the leading international forums for robotics researchers to present their work.

Interviews:

Howard Michel, Chief Technology Officer of UBTECH Education, on a humanoid robot for STEM (Science, Technology, Engineering, and Math) education for a range of ages. (Begins at 1m 20s.)

Li Bingbing, Software Engineer and Cofounder of Transforma in Singapore, on a robot for painting tall buildings. (Begins at 10m 25s.)

Xianbao Chen, Associate Researcher at Shanghai Jiao Tong University, on a hexapod robot for large parts machining. (Begins at 17m 5s.)

Lester Teh Chee Onn, Environmental Engineer at Advisian, on a watercraft for environmental monitoring. (Begins at 27m.)

Links

• Download mp3 (18.8 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, MeiXing Dong talks with Leon Kuperman, CTO of CUJO, about cybersecurity threats and how to guard against them. They discuss how CUJO, a smart hardware firewall, helps protect the home against online threats.

Leon Kuperman

Leon Kuperman is the CTO of CUJO IoT Security. Leon has a specialization in Artificial Intelligence, and is applying this expert knowledge of machine learning to his work at CUJO smart firewall. He is also an authority on Web Application Security and Payment Card Industry Data Security Standard (PCI DSS), e-commerce, cloud computing and web application architecture. Before CUJO, he co-founded ZENEDGE, a Web Application Security Platform for Enterprise. He also assisted with the launch of Modnique.com and held the position of CTO at Bidz.com. Prior to Zenedge, Leon founded and led Truition Inc., and early on demand ecommerce platform, winning major clients including Dell Financial Services, MLB, Comet, Philips, NHL, United Airlines and Disney. In his early career, he was the founding co-developer of IBM Net.Commerce currently named IBM Websphere Commerce Suite.

Links

• Download mp3 (14.4 MB)
• CUJO Website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Researchers have devised a new way to monitor sleep stages without sensors attached to the body. Their device uses an advanced artificial intelligence algorithm to analyze the radio signals around the person and translate those measurements into sleep stages: light, deep, or rapid eye movement (REM). Image: Christine Daniloff/MIT

More than 50 million Americans suffer from sleep disorders, and diseases including Parkinson’s and Alzheimer’s can also disrupt sleep. Diagnosing and monitoring these conditions usually requires attaching electrodes and a variety of other sensors to patients, which can further disrupt their sleep.

To make it easier to diagnose and study sleep problems, researchers at MIT and Massachusetts General Hospital have devised a new way to monitor sleep stages without sensors attached to the body. Their device uses an advanced artificial intelligence algorithm to analyze the radio signals around the person and translate those measurements into sleep stages: light, deep, or rapid eye movement (REM).

“Imagine if your Wi-Fi router knows when you are dreaming, and can monitor whether you are having enough deep sleep, which is necessary for memory consolidation,” says Dina Katabi, the Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science, who led the study. “Our vision is developing health sensors that will disappear into the background and capture physiological signals and important health metrics, without asking the user to change her behavior in any way.”

Katabi worked on the study with Matt Bianchi, chief of the division of sleep medicine at MGH, and Tommi Jaakkola, the Thomas Siebel Professor of Electrical Engineering and Computer Science and a member of the Institute for Data, Systems, and Society at MIT. Mingmin Zhao, an MIT graduate student, is the paper’s first author, and Shichao Yue, another MIT graduate student, is also a co-author.

The researchers will present their new sensor at the International Conference on Machine Learning on Aug. 9.

Remote sensing

Katabi and members of her group in MIT’s Computer Science and Artificial Intelligence Laboratory have previously developed radio-based sensors that enable them to remotely measure vital signs and behaviors that can be indicators of health. These sensors consist of a wireless device, about the size of a laptop computer, that emits low-power radio frequency (RF) signals. As the radio waves reflect off of the body, any slight movement of the body alters the frequency of the reflected waves. Analyzing those waves can reveal vital signs such as pulse and breathing rate.

“It’s a smart Wi-Fi-like box that sits in the home and analyzes these reflections and discovers all of these changes in the body, through a signature that the body leaves on the RF signal,” Katabi says.

Katabi and her students have also used this approach to create a sensor called WiGait that can measure walking speed using wireless signals, which could help doctors predict cognitive decline, falls, certain cardiac or pulmonary diseases, or other health problems.

After developing those sensors, Katabi thought that a similar approach could also be useful for monitoring sleep, which is currently done while patients spend the night in a sleep lab hooked up to monitors such as electroencephalography (EEG) machines.

“The opportunity is very big because we don’t understand sleep well, and a high fraction of the population has sleep problems,” says Zhao. “We have this technology that, if we can make it work, can move us from a world where we do sleep studies once every few months in the sleep lab to continuous sleep studies in the home.”

To achieve that, the researchers had to come up with a way to translate their measurements of pulse, breathing rate, and movement into sleep stages. Recent advances in artificial intelligence have made it possible to train computer algorithms known as deep neural networks to extract and analyze information from complex datasets, such as the radio signals obtained from the researchers’ sensor. However, these signals have a great deal of information that is irrelevant to sleep and can be confusing to existing algorithms. The MIT researchers had to come up with a new AI algorithm based on deep neural networks, which eliminates the irrelevant information.

“The surrounding conditions introduce a lot of unwanted variation in what you measure. The novelty lies in preserving the sleep signal while removing the rest,” says Jaakkola. Their algorithm can be used in different locations and with different people, without any calibration.

Using this approach in tests of 25 healthy volunteers, the researchers found that their technique was about 80 percent accurate, which is comparable to the accuracy of ratings determined by sleep specialists based on EEG measurements.

“Our device allows you not only to remove all of these sensors that you put on the person, and make it a much better experience that can be done at home, it also makes the job of the doctor and the sleep technologist much easier,” Katabi says. “They don’t have to go through the data and manually label it.”

Sleep deficiencies

Other researchers have tried to use radio signals to monitor sleep, but these systems are accurate only 65 percent of the time and mainly determine whether a person is awake or asleep, not what sleep stage they are in. Katabi and her colleagues were able to improve on that by training their algorithm to ignore wireless signals that bounce off of other objects in the room and include only data reflected from the sleeping person.

The researchers now plan to use this technology to study how Parkinson’s disease affects sleep.

“When you think about Parkinson’s, you think about it as a movement disorder, but the disease is also associated with very complex sleep deficiencies, which are not very well understood,” Katabi says.

The sensor could also be used to learn more about sleep changes produced by Alzheimer’s disease, as well as sleep disorders such as insomnia and sleep apnea. It may also be useful for studying epileptic seizures that happen during sleep, which are usually difficult to detect.

In recent years engineers have been developing new technologies to enable robots and humans to move faster and jump higher. Soft, elastic materials store energy in these devices, which, if released carefully, enable elegant dynamic motions. Robots leap over obstacles and prosthetics empower sprinting. A fundamental challenge remains in developing these technologies. Scientists spend long hours building and testing prototypes that can reliably move in specific ways so that, for example, a robot lands right-side up upon landing a jump.

A pair of new computational methods developed by a team of researchers from Massachusetts Institute of Technology (MIT), University of Toronto and Adobe Research takes first steps towards automating the design of the dynamic mechanisms behind these movements. Their methods generate simulations that match the real-world behaviors of flexible devices at rates 70-times faster than previously possible and provide critical improvements in the accuracy of simulated collisions and rebounds. These methods are then both fast and accurate enough to be used to automate the design process used to create dynamic mechanisms for controlled jumping.

The team will present their methods and results from their paper, “Dynamics-Aware Numerical Coarsening for Fabrication Design,” at the SIGGRAPH 2017 conference in Los Angeles, 30 July to 3 August. SIGGRAPH spotlights the most innovative results in computer graphics research and interactive techniques worldwide.

“This research is pioneering work in applying computer graphics techniques to real physical objects with dynamic behavior and contact,” says lead author Desai Chen, a PhD candidate at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). “The techniques we’ve developed open the door to automating the design of highly dynamic, fast-moving objects.”

Chen’s co-authors include David I.W. Levin, assistant professor at the University of Toronto; Wojciech Matusik, associate professor of electrical engineering and computer science at MIT; and Danny M. Kaufman, senior research scientist at Adobe Research.

Major advances in computational design, physical modeling and rapid manufacturing have enabled the fabrication of objects with customized physical properties–such as tailored sneakers, complex prosthetics, and soft robots–while computer graphics research has seen rapid improvements and efficiencies in creating compelling animations of physics for games, virtual reality and film. In this new work, the team aims to combine efficiency and accuracy to enable simulation for design fabrication, and to accurately simulate objects in motion.

“The goal is to bring the physical rules of virtual reality much closer to those of actual reality,” says Levin.

In the research, the team addresses the challenge with simulating elastic objects as they collide – making things accurate enough to match reality and fast enough to automate that design process. Attempting to create such simulations in the presence of contact, impact or friction remains time-consuming and inaccurate.

“It is very important to get this part right, and, until now, our existing computer codes tend to break down here,” says Kaufman. “We realize that if we are doing design for the real world, we have to have code that correctly models things such as high-speed bouncing, collision and friction.”

The researchers demonstrate their new methods, Dynamics-Aware Coarsening (DAC) and Boundary Balanced Impact (BBI), by designing and fabricating mechanisms that flip, throw and jump over obstacles. Their methods perform simulations much faster than existing, state-of-the-art approaches and with greater accuracy when compared to real-world motions.

DAC works by reducing degrees of freedom, the number of values that encode motion, to speed up simulations while still capturing important motions for dynamic scenarios. It finds the roughest meshes that can correctly represent the key shapes that will be taken by dynamics and matches the material properties of these meshes directly to recorded video experiment. BBI is a method for modeling impact behavior of elastic objects. It uses material properties to smoothly project velocities near impact sites to model many real world impact situations such as the impact and rebound between a soft printed material and a table, for instance.

The team was inspired by the need for faster, more accurate design tools that can capture accurate simulations of elastic objects undergoing deformation and collision – especially at high-speeds. These new methods could, down the road, be applied to robotics design, developing robots as they increasingly take on human-like movements and characteristics.

“This project is really a first step for us in pushing methods for simulating reality,” says Kaufman. “We are focusing on pushing them for automatic design and exploring how to effectively use them in design. We can create beautiful images in computer graphics and in animation, let’s extend these methods to actual objects in the real world that are useful, beautiful and efficient.”

Adriana Schulz, an MIT PhD student in the Computer Science and Artificial Intelligence Laboratory, demonstrates the InstantCAD computer-aided-design-optimizing interface. Photo: Rachel Gordon/MIT CSAIL

Almost every object we use is developed with computer-aided design (CAD). Ironically, while CAD programs are good for creating designs, using them is actually very difficult and time-consuming if you’re trying to improve an existing design to make the most optimal product. Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) and Columbia University are trying to make the process faster and easier: In a new paper, they’ve developed InstantCAD, a tool that lets designers interactively edit, improve, and optimize CAD models using a more streamlined and intuitive workflow.

“From more ergonomic desks to higher-performance cars, this is really about creating better products in less time,” says Department of Electrical Engineering and Computer Science PhD student and lead author Adriana Schulz, who will be presenting the paper at this month’s SIGGRAPH computer-graphics conference in Los Angeles. “We think this could be a real game changer for automakers and other companies that want to be able to test and improve complex designs in a matter of seconds to minutes, instead of hours to days.”

The paper was co-written by Associate Professor Wojciech Matusik, PhD student Jie Xu, and postdoc Bo Zhu of CSAIL, as well as Associate Professor Eitan Grinspun and Assistant Professor Changxi Zheng of Columbia University.

Traditional CAD systems are “parametric,” which means that when engineers design models, they can change properties like shape and size (“parameters”) based on different priorities. For example, when designing a wind turbine you might have to make trade-offs between how much airflow you can get versus how much energy it will generate.

InstantCAD enables designers to interactively edit, improve, and optimize CAD models using a more streamlined and intuitive workflow. Photo: Rachel Gordon/MIT CSAIL

However, it can be difficult to determine the absolute best design for what you want your object to do, because there are many different options for modifying the design. On top of that, the process is time-consuming because changing a single property means having to wait to regenerate the new design, run a simulation, see the result, and then figure out what to do next.

With InstantCAD, the process of improving and optimizing the design can be done in real-time, saving engineers days or weeks. After an object is designed in a commercial CAD program, it is sent to a cloud platform where multiple geometric evaluations and simulations are run at the same time.

With this precomputed data, you can instantly improve and optimize the design in two ways. With “interactive exploration,” a user interface provides real-time feedback on how design changes will affect performance, like how the shape of a plane wing impacts air pressure distribution. With “automatic optimization,” you simply tell the system to give you a design with specific characteristics, like a drone that’s as lightweight as possible while still being able to carry the maximum amount of weight.

The reason it’s hard to optimize an object’s design is because of the massive size of the design space (the number of possible design options).

“It’s too data-intensive to compute every single point, so we have to come up with a way to predict any point in this space from just a small number of sampled data points,” says Schulz. “This is called ‘interpolation,’ and our key technical contribution is a new algorithm we developed to take these samples and estimate points in the space.”

Matusik says InstantCAD could be particularly helpful for more intricate designs for objects like cars, planes, and robots, particularly for industries like car manufacturing that care a lot about squeezing every little bit of performance out of a product.

“Our system doesn’t just save you time for changing designs, but has the potential to dramatically improve the quality of the products themselves,” says Matusik. “The more complex your design gets, the more important this kind of a tool can be.”

Because of the system’s productivity boosts and CAD integration, Schulz is confident that it will have immediate applications for industry. Down the line, she hopes that InstantCAD can also help lower the barrier for entry for casual users.

“In a world where 3-D printing and industrial robotics are making manufacturing more accessible, we need systems that make the actual design process more accessible, too,” Schulz says. “With systems like this that make it easier to customize objects to meet your specific needs, we hope to be paving the way to a new age of personal manufacturing and DIY design.”

The project was supported by the National Science Foundation.

In this episode, MeiXing Dong conducts interviews at the 2017 Midwest Speech and Language Days workshop in Chicago. She talks with Michael White of Ohio State University about question interpretation in a dialogue system; Dmitriy Dligach of Loyola University Chicago about extracting patient timelines from doctor’s notes; and Denis Newman-Griffiths of Ohio State University about connecting words and phrases to relevant medical topics.

Links

• Download mp3 (14.7 MB)
• Midwest Speech and Language Days 2017 Website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

MATLAB© is a programming language and environment designed for scientific computing. It is one of the best languages for developing robot control algorithms and is widely used in the research community. While it is often thought of as an offline programming language, there are several ways to interface with it to control robotic hardware ‘in the loop’. As part of our own development we surveyed a number of different projects that accomplish this by using a message passing system and we compared the approaches they took. This post focuses on bindings for the following message passing frameworks: LCM, ROS, DDS, and ZeroMQ.

The main motivation for using MATLAB to prototype directly on real hardware is to dramatically accelerate the development cycle by reducing the time it takes to find out out whether an algorithm can withstand ubiquitous real-world problems like noisy and poorly-calibrated sensors, imperfect actuator controls, and unmodeled robot dynamics. Additionally, a workflow that requires researchers to port prototype code to another language before being able to test on real hardware can often lead to weeks or months being lost in chasing down new technical bugs introduceed by the port. Finally, programming in a language like C++ can pose a significant barrier to controls engineers who often have a strong electro-mechanical background but are not as strong in computer science or software engineering.

We have also noticed that over the past few years several other groups in the robotics community also experience these problems and have started to develop ways to control hardware directly from MATLAB.

---

The Need for External Languages

The main limitation when trying to use MATLAB to interface with hardware stems from the fact that its scripting language is fundamentally single threaded. It has been designed to allow non-programmers to do complex math operations without needing to worry about programming concepts like multi-threading or synchronization. However, this poses a problem for real-time control of hardware because all communication is forced to happen synchronously in the main thread. For example, if a control loop runs at 100Hz and it takes a message ~8ms for a round-trip, the main thread ends up wasting 80% of the available time budget waiting for a response without doing any actual work.

A second hurdle is that while MATLAB is very efficient in the execution of math operations, it is not particularly well suited for byte manipulation. This makes it difficult to develop code that can efficiently create and parse binary message formats that the target hardware can understand. Thus, after having the main thread spend its time waiting for and parsing the incoming data, there may not be any time left for performing interesting math operations.

Figure 1. Communications overhead in the main MATLAB thread

Pure MATLAB implementations can work for simple applications, such as interfacing with an Arduino to gather temperature data or blink an LED, but it is not feasible to control complex robotic systems (e.g. a humanoid) at high rates (e.g. 100Hz-1KHz). Fortunately, MATLAB does have the ability to interface with other programming languages that allow users to create background threads that can offload the communications aspect from the main thread.

Figure 2. Communications overhead offloaded to other threads

Out of the box MATLAB provides two interfaces to other languages: MEX for calling C/C++ code, and the Java Interface for calling Java code. There are some differences between the two, but at the end of the day the choice effectively comes down to personal preference. Both provide enough capabilities for developing sophisticated interfaces and have orders of magnitude better performance than required. There are additional interfaces to other languages, but those require additional setup steps.

---

Message Passing Frameworks

Message passing frameworks such as Robot Operating System (ROS) and Lightweight Communication and Marshalling (LCM) have been widely adopted in the robotics research community. At the core they typically consist of two parts: a way to exchange data between processes (e.g. UDP/TCP), as well as a defined binary format for encoding and decoding the messages. They allow systems to be built with distributed components (e.g. processes) that run on different computers, different operating systems, and different programming languages.

The resulting systems are very extensible and provide convenient ways for prototyping. For example, a component communicating with a physical robot can be exchanged with a simulator without affecting the rest of the system. Similarly, a new walking controller could be implemented in MATLAB and communicate with external processes (e.g. robot comms) through the exchange of messages. With ROS and LCM in particular, their flexibility, wide-spread adoption, and support for different languages make them a nice starting point for a MATLAB-hardware interface.

---

Lightweight Communication and Marshalling (LCM)

LCM was developed in 2006 at MIT for their entry to DARPA’s Urban Challenge. In recent years it has become a popular alternative to ROS-messaging, and it was as far as we know the first message passing framework for robotics that supported MATLAB as a core language.

The snippet below shows how the MATLAB code for sending a command message could look like. The code creates a struct-like message, sets desired values, and publishes it on an appropriate channel.

%% MATLAB code for sending an LCM message
% Setup
lc = lcm.lcm.LCM.getSingleton();

% Fill message
cmd = types.command();
cmd.position = [1 2 3];
cmd.velocity = [1 2 3];

% Publish
lc.publish('COMMAND_CHANNEL', cmd);

Interestingly, the backing implementation of these bindings was done in pure Java and did not contain any actual MATLAB code. The exposed interface consisted of two Java classes as well as auto-generated message types.

• The LCM class provides a way to publish messages and subscribe to channels
• The generated Java messages handle the binary encoding and exposed fields that MATLAB can access
• The MessageAggregator class provides a way to receive messages on a background thread and queue them for MATLAB.

Thus, even though the snippet looks similar to MATLAB code, all variables are actually Java objects. For example, the struct-like command type is a Java object that exposes public fields as shown in the snippet below. Users can access them the same way as fields of a standard MATLAB struct (or class properties) resulting in nice syntax. The types are automatically converted according to the type mapping.

/**
 * Java class that behaves like a MATLAB struct
 */
public final class command implements lcm.lcm.LCMEncodable
{
    public double[] position;
    public double[] velocity;
    // etc. ...
}

Receiving messages is done by subscribing an aggregator to one or more channels. The aggregator receives messages from a background thread and stores them in a queue that MATLAB can access in a synchronous manner using aggregator.getNextMessage(). Each message contains the raw bytes as well as some meta data for selecting an appropriate type for decoding.

%% MATLAB code for receiving an LCM message
% Setup
lc = lcm.lcm.LCM.getSingleton();
aggregator = lcm.lcm.MessageAggregator();
lc.subscribe('FEEDBACK_CHANNEL', aggregator);

% Continuously check for new messages
timeoutMs = 1000;
while true

    % Receive raw message
    msg = aggregator.getNextMessage(timeoutMs);

    % Ignore timeouts
    if ~isempty(msg)

        % Select message type based on channel name
        if strcmp('FEEDBACK_CHANNEL', char(msg.channel))

            % Decode raw bytes to a usable type
            fbk = types.feedback(msg.data);

            % Use data
            position = fbk.position;
            velocity = fbk.velocity;

        end

    end
end

The snippet below shows a simplified version of the backing Java code for the aggregator class. Since Java is limited to a single return argument, the getNextMessage call returns a Java type that contains the received bytes as well as meta data to identify the type, i.e., the source channel name.

/**
 * Java class for receiving messages in the background
 */
public class MessageAggregator implements LCMSubscriber {

    /**
     * Value type that combines multiple return arguments
     */
    public static class Message {

        final public byte[] data; // raw bytes
        final public String channel; // source channel name

        public Message(String channel_, byte[] data_) {
            data = data_;
            channel = channel_;
        }
    }

    /**
     * Method that gets called from MATLAB to receive new messages
     */
    public synchronized Message getNextMessage(long timeout_ms) {

		if (!messages.isEmpty()) {
		    return messages.removeFirst();
        }

        if (timeout_ms == 0) { // non-blocking
            return null;
        }

        // Wait for new message until timeout ...
    }

}

Note that the getNextMessage method requires a timeout argument. In general it is important for blocking Java methods to have a timeout in order to prevent the main thread from getting stuck permanently. Being in a Java call prohibits users from aborting the execution (ctrl-c), so timeouts should be reasonably short, i.e., in the low seconds. Otherwise this could cause the UI to become unresponsive and users may be forced to close MATLAB without being able to save their workspace. Passing in a timeout of zero serves as a non-blocking interface that immediately returns empty if no messages are available. This is often useful for working with multiple aggregators or for integrating asynchronous messages with unknown timing, such as user input.

Overall, we thought that this was a well thought out API and a great example for a minimum viable interface that works well in practice. By receiving messages on a background thread and by moving the encoding and decoding steps to the Java language, the main thread is able to spend most of its time on actually working with the data. Its minimalistic implementation is comparatively simple and we would recommend it as a starting point for developing similar interfaces.

Some minor points for improvement that we found were:

• The decoding step fbk = types.feedback(msg.data) forces two unnecessary translations due to msg.data being a byte[], which automatically gets converted to and from int8. This could result in a noticeable performance hit when receiving larger messages (e.g. images) and could be avoided by adding an overload that accepts a non-primitive type that does not get translated, e.g., fbk = types.feedback(msg).
• The Java classes did not implement Serializable, which could become bothersome when trying to save the workspace.
• We would prefer to select the decoding type during the subscription step, e.g., lc.subscribe(‘FEEDBACK_CHANNEL’, aggregator, ‘types.feedback’), rather than requiring users to instantiate the type manually. This would clean up the parsing code a bit and allow for a less confusing error message if types are missing.

---

Robot Operating System (ROS)

ROS is by far the most widespread messaging framework in the robotics research community and has been officially supported by Mathworks’ Robotics System Toolbox since 2014. While the Simulink code generation uses ROS C++, the MATLAB implementation is built on the less common RosJava.

The API was designed such that each topic requires dedicated publishers and subscribers, which is different from LCM where each subscriber may listen to multiple channels/topics. While this may result in potentially more subscribers, the specification of the expected type at initialization removes much of the boiler plate code necessary for dealing with message types.

%% MATLAB code for publishing a ROS message
% Setup Publisher
chatpub = rospublisher('/chatter', 'std_msgs/String');

% Fill message
msg = rosmessage(chatpub);
msg.Data = 'Some test string';

% Publish
chatpub.send(msg);

Subscribers support three different styles to access messages: blocking calls, non-blocking calls, and callbacks.

%% MATLAB code for receiving a ROS message
% Setup Subscriber
laser = rossubscriber('/scan');

% (1) Blocking receive
scan = laser.receive(1); % timeout [s]

% (2) Non-blocking latest message (may not be new)
scan = laser.LatestMessage;

% (3) Callback
callback = @(msg) disp(msg);
subscriber = rossubscriber('/scan', @callback);

Contrary to LCM, all objects that are visible to users are actually MATLAB classes. Even though the implementation is using Java underneath, all exposed functionality is wrapped in MATLAB classes that hide all Java calls. For example, each message type is associated with a generated wrapper class. The code below shows a simplified example of a wrapper for a message that has a Name property.

%% MATLAB code for wrapping a Java message type
classdef WrappedMessage

    properties (Access = protected)
        % The underlying Java message object (hidden from user)
        JavaMessage
    end

    methods

        function name = get.Name(obj)
            % value = msg.Name;
            name = char(obj.JavaMessage.getName);
        end

        function set.Name(obj, name)
            % msg.Name = value;
            validateattributes(name, {'char'}, {}, 'WrappedMessage', 'Name');
            obj.JavaMessage.setName(name); % Forward to Java method
        end

        function out = doSomething(obj)
            % msg.doSomething() and doSomething(msg)
            try
                out = obj.JavaMessage.doSomething(); % Forward to Java method
            catch javaException
                throw(WrappedException(javaException)); % Hide Java exception
            end
        end

    end
end

Due to the implementation being closed-source, we were only able to look at the public toolbox files as well as the compiled Java bytecode. As far as we could tell they built a small Java library that wrapped RosJava functionality in order to provide an interface that is easier to call from MATLAB. Most of the actual logic seemed to be implemented in MATLAB code, but we also found several calls to various Java libraries for problems that would have been difficult to implement in pure MATLAB, e.g., listing networking interfaces or doing in-memory decompression of images.

Overall, we found that the ROS support toolbox looked very nice and was a great example of how seamless external languages could be integrated with MATLAB. We also really liked that they offered a way to load log files (rosbags).

One concern we had was that there did not seem to be a simple non-blocking way to check for new messages, e.g., a hasNewMessage() method or functionality equivalent to LCM’s getNextMessage(0). We often found this useful for applications that combined data from multiple topics that arrived at different rates (e.g. sensor feedback and joystick input events). We checked whether this behavior could be emulated by specifying a very small timeout in the receive method (shown in the snippet below), but any value below 0.1s seemed to never successfully return.

---

Data Distribution Service (DDS)

In 2014 Mathworks also added a support package for DDS, which is the messaging middleware that ROS 2.0 is based on. It supports MATLAB and Simulink, as
well as code generation. Unfortunately, we did not have all the requirements to get it setup, and we could not find much information about the underlying implementation. After looking at some of the intro videos, we believe that the resulting code should look as follows.

%% MATLAB code for sending and receiving DDS messages
% Setup
DDS.import('ShapeType.idl','matlab');
dp = DDS.DomainParticipant

% Create message
myTopic = ShapeType;
myTopic.x = int32(23);
myTopic.y = int32(35);

% Send Message
dp.addWriter('ShapeType', 'Square');
dp.write(myTopic);

% Receive message
dp.addReader('ShapeType', 'Square');
readTopic = dp.read();

---

ZeroMQ

ZeroMQ is another asynchonous messaging library that is popular for building distributed systems. It only handles the messaging aspect, so users need to supply their own wire format. ZeroMQ-matlab is a MATLAB interface to ZeroMQ that was developed at UPenn between 2013-2015. We were not able to find much documentation, but as far as we could tell the resulting code should look similar to following snippet.

%% MATLAB code for sending and receiving ZeroMQ data
% Setup
subscriber = zmq( 'subscribe', 'tcp', '127.0.0.1', 43210 );
publisher = zmq( 'publish', 'tcp', 43210 );

% Publish data
bytes = uint8(rand(100,1));
nbytes = zmq( 'send', publisher, bytes );

% Receive data
receiver = zmq('poll', 1000); // polls for next message
[recv_data, has_more] = zmq( 'receive', receiver );

disp(char(recv_data));

It was implemented as a single MEX function that selects appropriate sub-functions based on a string argument. State was maintained by using socket IDs that were passed in by the user at every call. The code below shows a simplified snippet of the send action.

// Parsing the selected ZeroMQ action behind the MEX barrier
// Grab command String
if ( !(command = mxArrayToString(prhs[0])) )
	mexErrMsgTxt("Could not read command string. (1st argument)");

// Match command String with desired action (e.g. 'send')
if (strcasecmp(command, "send") == 0){
	// ... (argument validation)

	// retrieve arguments
	socket_id = *( (uint8_t*)mxGetData(prhs[1]) );
	size_t n_el = mxGetNumberOfElements(prhs[2]);
	size_t el_sz = mxGetElementSize(prhs[2]);
	size_t msglen = n_el*el_sz;

	// send data
	void* msg = (void*)mxGetData(prhs[2]);
	int nbytes = zmq_send( sockets[ socket_id ], msg, msglen, 0 );

	// ... check outcome and return
}
// ... other actions

---

Other Frameworks

Below is a list of APIs to other frameworks that we looked at but could not cover in more detail.

Project	Notes
RabbitMQ-Matlab-Client	Simple Java wrapper for RabbitMQ with callbacks into MATLAB
URBI (tutorial)	Seems to be deprecated

Crops are key for a sustainable food production and we face several challenges in crop production. First, we need to feed a growing world population. Second, our society demands high-quality foods. Third, we have to reduce the amount agrochemicals that we apply to our fields as it directly affects our ecosystem. Precision farming techniques offer a great potential to address these challenges, but we have to acquire and provide the relevant information about the field status to the farmers such that specific actions can be taken.

This paper won the IEEE Robotics & Automation Best Automation Paper Award at ICRA 2017.

Modern approaches as agriculture system management and smart farming typically require detailed knowledge about the current field status. For successful on-field intervention, it is key to know the type and spatial distribution of the crops and weeds on the field at an early state. The earlier mechanical or chemical weeding actions are executed, the higher the chances for obtaining a high yield. A prerequisite to trigger weeding and intervention task is a detailed knowledge about the spread of weeds. Therefore, it is important to have an automatic perception pipeline that monitors and analyzes the crops automatically and provides report about the status on the field.

Fig. 1 illustrates the input and output of our approach. We use a comparably cheap, out-of-the-box UAV system to capture images of a field and compute a class label to each pixel, i.e., determine if that pixel belongs a crop or a weed. Through the use of GNSS and bundle adjustment procedures, we can obtain the spatial distribution of the detected crops and weeds in a global reference frame.

Use Case 1: Farmer Support: In order to reduce the investments on herbicides, pesticides, and fertilizers by keeping or even increasing the yield, farmers must meet the following key questions: Where are weeds in the field? as well as where do the crops grow abnormally? Given the knowledge about the plant specific spatial distribution as illustrated in Fig. 2, farmers can decide, which and how much agro-chemicals are needed to address their specific needs. Furthermore, they can perform individual actions at specific locations in the field.

Figure 1: Low-cost DJI Phantom 4 UAV used for field monitoring (left) as well as an example image analyzed by our approach (right). Here, green refers to detected sugar beets, which we consider as crop, and purple refers to weeds.

Use Case 2: Autonomous Robots: In the future, autonomous robotic systems are likely to be used to manage arable farm land. Robots have the potential to survey the current field status on a continuous time scale and perform specific tasks such as weed control on a per-plant basis only in regions on a field where an action is actually needed. This work is part of the FLOURISH project (flourishproject.eu) were we are developing teams of autonomously and collaboratively operating UAVs and UGVs. Here, the UAVs offer excellent survey capabilities at low cost and can be used to analyze the field status. The analysis of crop and weed growth on the field is an essential building block to infer target locations for a targeted ground intervention. Thus, the UAV can send an UGV to those locations to execute interventions task.

Approach

We address the problem of analyzing UAV imagery to inspect the status of a field in terms of weed types and spatial crop and weed distribution. We focus on a detection on a per-plant basis to estimate the amount of crops as well as various weed species to provide this information to the farmers or as a part of an autonomous robotic perception system. In our work, we propose a vision-based classification system for identifying crops and weeds in both, RGB-only as well as RGB combined with near infra-red (NIR) imagery of agricultural fields captured by an UAV. Our perception pipeline is capable of detecting plants in such images only based on their appearance. Our approach is also able to exploit the geometry of the plant arrangement without requiring a known spatial distribution to be specified explicitly. Optionally, we can utilize prior information regarding the crop arrangement like crop rows. In addition to that it can deal with vegetation, which is located within the intra-row space.

Figure 2: Analyzed field status by our approach. Based on these informations the farmers can schedule their next individual actions and estimate the needed amount of agro-chemicals for their field.

Processing Pipeline

The three key steps of our systems are the following: First, we apply a preprocessing in order to normalize intensity values on a global scale and detect the vegetation in each image. Second, we extract features only for regions, which correspond to vegetation, exploiting a combination of an object-based and a keypoint based approach. We compute around 550 visual features containing intensity statistics, texture and shape information. In addition to that, we exploit geometric features, which encode spatial relationships between the vegetation as well as optional crop row features, which consider the distance from the vegetation to estimated crop rows. Third, we apply a multi-class Random Forest classification and obtain a probability distribution for the predicted class labels.

Figure 3: Example image captured by a DJI Phantom 4 at low altitude at different levels of progress within the classification pipeline. From left to right: normalized RGB image, computed Excess Green Index (ExG), vegetation mask that reflect the spatial distribution of the vegetation and multi-class classification output of our plant/weed classification system

Fig. 3 illustrates the key steps of our pipeline and Fig. 4 depicts an example of the detected crop rows. The papers [1] and [2] provide a detailed description of the pipeline as well as the features we are using for the classification.

Results and Conclusion

Figure 4: Detected set of crop rows (red) estimated based on Hough transform followed by a search algorithm in order to find the best set of parallel lines, which is supported by the vegetation. Example result from a sugar beet field with high weed pressure.

We implemented and tested our approach using different types of cameras and UAVs, on real sugar beet fields. Our experiments suggest that our proposed system is able to perform a classification of sugar beets and different weed types in RGB images captured by a commercial low cost UAV system. We show that the classification system achieves good performance in terms of overall accuracy of up to 96% even in images where neither a row detection of the crops nor a regular pattern of the plantation can be exploited for geometry-based detection. Furthermore, we evaluate our system using RGB+NIR images captured with a comparable expensive camera system to those obtained using the standard equipped RGB camera mounted on a consumer quad-copter (DJI Phantom 4 and camera for around $1,500). Fig. 5 depicts example results achieved on the different datasets respectively.

With our approach, we enable farmers to analyze the field status in terms of the spatial distribution of the crops and weeds on their field by using a low-cost UAV system, which can be used out-of-the-box. Typically, these systems are easy to use without requiring expertise on flying or aerial dynamics. The farmers only has to provide their expertise in terms of labeling into the perception pipeline. The farmers are predestined to perform this task because nobody knows better, which plants and weed types commonly appear on their specific fields. Compared to the area of the field only a small amount of data has to be labeled for analyzing the whole field. In terms of the usage withing autonomous robotic systems in agriculture this approach can be used as part of the perception pipeline of the system. Based on the classification results of the images taken by the UAV a program for decision making or planning can execute different task to achieve the destined goals, e.g. to send the UGV to a target location.

Figure 5: Zoomed view of images analyzed by our plant/weed classification system (left column) and corresponding ground truth images (right column). Top row: Multi-class results for the dataset captured by the DJI Phantom 4 UAV. Here, the crops do not follow a row structure. Thus, the results are based on the visual as well as geometric features. Center row: Result for the dataset captured with an 4-channel RGB+NIR camera system. Bottom row: Result for the dataset captured with a DJI Matrice 100 UAV. Arrows point to weeds detected in intra-row space. Sugar beet (green), chamomile (blue), saltbush (yellow) and other weeds (red).

Spiders are truly amazing creatures. They have evolved over more than 200 million years and can be found in almost every corner of our planet. They are one of the most successful animals. Not less impressive are their webs, highly intricate structures that have been optimised through evolution over approximately 100 million years with the ultimate purpose of catching prey.

However, interestingly, the closer you look at spiders’ webs the more details you can observe and the structures are much more complicated than one would expect from a simple snare. They are made of a variety of different types of silks, use water droplets to keep the tension [see: citation 4], and the structure is highly dynamic [see: citation 4]. Spider’s webs have a great deal more morphological complexity than what you would need to simply catch flies.

Since nature typically does not spoil resources the question arises: why are spiders’ webs so complex? Might they have other functionalities besides being a simple trap? One of the most interesting answers to this question is that spiders might use their webs as computational devices.

How does the spider use the web as a computer?

Despite the fact that most spiders have a lot of eyes (the majority has 8, but some have even up to 12), a lot of the spiders have bad sight. In order to understand what is going on in their webs, they use mechanoreceptors in their legs (called lyriforms) to “listen” to vibrations in the web.  Different species of spiders have different preferred places to sit and observe. While some can be found right at the center, others prefer to sit outside the actual web and to listen to one single thread. It is quite remarkable that based only on the information that comes through this single thread the spider seems to be able to deduce what is going on in their web and where these events are taking place.

For example, they need to know if there is a prey, like a fly, entangled in their web. Or if the vibrations are coming from a dangerous insect like a wasp and they should stay away. The web is also used to communicate with potential mates and the spider even excites the web and listens to the echo. This might be a way for the spider to check if threads are broken or if the tension in the web has to be increased.

From a computational point of view, the spider needs to classify different vibration pattern (e.g., prey vs predator vs mate) and to locate its origin (i.e., where the vibration started).

One way to understand how a spider’s web could help to carry out this computational functionality is the concept of morphological computation. This is a term that describes the understanding that mechanical structures all over in nature are carrying out useful computations. For example, they help to stabilise running, facilitate sensory data processing, and helps animals and plants to interact with complex and unpredictable environments.

One could say computation is outsourced to the physical body (e.g., from the brain to another part of the body).

From this point of view, the spider’s web can be seen as a nonlinear, dynamic filter. It can be understood as some kind of pre-processing unit that makes it easier for the animal to interpret the vibration signals. The web’s dynamic properties and its complex morphological structure mix vibration signals in a nonlinear fashion. It even has some memory. This can be easily seen by pinching the web. It responds with vibrations for some moments after the impact echoing the original input.  The web can also damp unwanted frequencies, which is crucial to get rid of noise. On the other hand, it might even be able to highlight other signals at certain frequencies that carry more relevant information about the events taking place on the web.

These are all useful computations and they make it easier for the spider to “read” and understand the vibration patterns. As a result, the brain of the animal has to do less work and it can concentrate on other tasks. In effect, the spider seems to devolve computation to the web.  This might be also the reason why spiders tend to their webs so intensively. They constantly observe it and adapt the tension if it has changed, e.g. due to change in humidity, and repair it as soon a thread is broken.

From spider webs to sensors

People have speculated for a while that spider webs might have additional functionalities. A great article that discusses that is “The Thoughts of a Spiderweb“.

However, nobody so far has systematically looked into the actual computational capabilities of the web. This is about to change. We recently started a Leverhulme Trust Research project that will investigate naturally spun spider webs of different species to understand how and which computing might take place in these structures.  Moreover, the project does not only try to understand the underlying computational principles but will also develop morphological computation-based sensor technology to measure flow and vibrations.

The project combines our research expertise in Morphological Computation at the University of Bristol and the expertise on spider webs at the Silk Group in Oxford.

In experimental setups we will use solenoids and laser Doppler vibrometers to measure vibrations in the web with very high precision. The goal is to understand how computation is carried out. We will systematically investigate how filtering capabilities, memory, and signal integration can happen in such structures. In parallel, we will develop a general simulation environment for vibrating structures. We will use this to ask specific questions about how different shapes and materials others than spider webs and silk can help to carry out computations. In addition, we will develop real prototypes of vibration and flow sensors, which will be inspired by these findings. It’s very likely they will look different from spider webs and they will use various types of materials.

Such sensors can be used in various applications. For example, morphological computation based flow sensors could be used to detect anomalies in the flow in tubes. Or vibration sensors put at strategic places on buildings could be able to detect earthquakes or structural failure. Also highly dynamic machines, for example, a wind turbine, could be monitored by such sensors to predict failure.

Ultimately, the project will provide not only a new technology to build sensors, but we hope also to get a fundamental understanding how spiders use their webs for computation.

References

[1] Hauser, H.; Ijspeert, A.; Füchslin, R.; Pfeifer, R. & Maass, W.”Towards a theoretical foundation for morphological computation with compliant bodies.”Biological Cybernetics, Springer Berlin / Heidelberg, 2011, 105, 355-370

[2]  Hauser, H.; Ijspeert, A.; Füchslin, R.; Pfeifer, R. & Maass, W. “The role of feedback in morphological computation with compliant bodies”. Biological Cybernetics, Springer Berlin / Heidelberg, 2012, 106, 595-613

[3] Hauser, H.; Füchslin, R.M.; Nakajima, K.“Morphological Computation – The Physical Body as a Computational Resource” Opinions and Outlooks on Morphological Computation, editors Hauser, H.; Füchslin, R.M. and Pfeifer, R., Chapter 20, pp 226-244,  2014, ISBN 978-3-033-04515-6

[4] Mortimer, B., Gordon, S. D., Holland, C., Siviour, C. R., Vollrath, F. and Windmill, J. F. C. (2014), The Speed of Sound in Silk: Linking Material Performance to Biological Function. Adv. Mater., 26: 5179–5183. doi:10.1002/adma.201401027

Lecturer Steffen Pfiffner of University of Weingarten in Germany is teaching ROS to 26 students at the same time at a very fast pace. His students, all of them within the Master on Computer Science of University of Weingarten, use only a web browser. They connect to a web page containing the lessons, a ROS development environment and several ROS based simulated robots. Using the browser, Pfiffner and his colleague Benjamin Stähle, are able to teach how to program with ROS quickly and to many students. This is what Robot Ignite Academy is made for.

“With Ignite Academy our students can jump right into ROS without all the hardware and software setup problems. And the best: they can do this from everywhere,” says Pfiffner.

Robot Ignite Academy provides a web service which contains the teaching material in text and video format, the simulations of several ROS based robots that the students must learn to program, and the development environment required to build ROS programs and test them on the simulated robot.

Student’s point of view

Students bring their own laptops to the class and connect to the online platform. From that moment, their laptop becomes a ROS development machine, ready to develop programs for many simulated real robots.

The Academy provides the text, the videos and the examples that the student has to follow. Then, the student creates her own ROS program and makes the robot perform a specific action. The student develops the ROS programs as if she is in a typical ROS development computer.

The main advantage is that students can use a Windows, Linux or Mac machine to learn ROS. They don’t even have to install ROS in their computers. The only prerequisite of the laptop is to have a browser. So students do not mess with all the installation problems that frustrate them (and the teachers!), especially when they are starting.

After class, students can continue with their learning at home, library or even the beach if there is a wifi available! All their code, learning material and simulations are stored online so they can access them from anywhere, anytime using any computer.

Teacher’s point of view

The advantage of using the platform is not only for the students but also for the teachers. Teachers do not have to create the material and maintain it. They do not have to prepare the simulations or work on multiple different computers. They don’t even have to prepare the exams!! (which are already provided by the platform).

So what are the teachers for?

By making use of the provided material, the teacher can concentrate on guiding the students by explaining the most confusing parts, answer questions, suggest modifications according to the level of each student, and adapt the pace to the different types of students.

This new method of teaching ROS is exploding among the Universities and High Schools that want to provide the latest and most practical teachings to their students. The method, developed by Robot Ignite Academy, combines a new way of teaching based on practice and an online learning platform. Those two points combined make the teaching of ROS a smooth experience and can potentially see the students’ knowledge base skyrocket.

As user Walace Rosa indicates in his video comment about Robot Ignite Academy:

It is a game changer [in] teaching ROS!

The method is becoming very popular in the robotics circuits too, and many teachers are using it for younger students. For example, High School Mundet in Barcelona is using it to teach ROS to 15 years old students.

Additionally, the academy provides a free online certification exam with different levels of knowledge certification. Many Universities are using this exam to certify that their students did learn the material since the exam is quite demanding.

Some examples of past events

•  1 week ROS course in Barcelona for SMART-E project team members. This is a private course given by Robot Ignite Academy at Barcelona for 15 members of the SMART-E project that need to be up to speed with ROS fast. From 8th to 12nd of May 2017
•  1 day ROS course for the Col·legi d’Enginyers de Barcelona. The 17th of May 2017.
•  3 months course for University of La Salle in Barcelona within the Master on Automatics, Domotics and Robotics. From 10th of May to 29th of June 2017.
•  1 weekend ROS course for teenagers in Bilbao, Spain. The 20th and 21st of May 2017.
•  We can also organize a special event like these for you and your team.

Helpful ROS videos

Engineers at MIT have come up with an algorithm to tune a Dynamic Vision Sensor (DVS) camera, simplifying a scene to its most essential visual elements and potentially enabling the development of faster drones. Image: Jose-Luis Olivares/MIT

There’s a limit to how fast autonomous vehicles can fly while safely avoiding obstacles. That’s because the cameras used on today’s drones can only process images so fast, frame by individual frame. Beyond roughly 30 miles per hour, a drone is likely to crash simply because its cameras can’t keep up.

Recently, researchers in Zurich invented a new type of camera, known as the Dynamic Vision Sensor (DVS), that continuously visualizes a scene in terms of changes in brightness, at extremely short, microsecond intervals. But this deluge of data can overwhelm a system, making it difficult for a drone to distinguish an oncoming obstacle through the noise.

Now engineers at MIT have come up with an algorithm to tune a DVS camera to detect only specific changes in brightness that matter for a particular system, vastly simplifying a scene to its most essential visual elements.

The results, which they presented at the IEEE American Control Conference in Seattle, can be applied to any linear system that directs a robot to move from point A to point B as a response to high-speed visual data. Eventually, the results could also help to increase the speeds for more complex systems such as drones and other autonomous robots.

“There is a new family of vision sensors that has the capacity to bring high-speed autonomous flight to reality, but researchers have not developed algorithms that are suitable to process the output data,” says lead author Prince Singh, a graduate student in MIT’s Department of Aeronautics and Astronautics. “We present a first approach for making sense of the DVS’ ambiguous data, by reformulating the inherently noisy system into an amenable form.”

Singh’s co-authors are MIT visiting professor Emilio Frazzoli of the Swiss Federal Institute of Technology in Zurich, and Sze Zheng Yong of Arizona State University.

Taking a visual cue from biology

The DVS camera is the first commercially available “neuromorphic” sensor — a class of sensors that is modeled after the vision systems in animals and humans. In the very early stages of processing a scene, photosensitive cells in the human retina, for example, are activated in response to changes in luminosity, in real time.

Neuromorphic sensors are designed with multiple circuits arranged in parallel, similarly to photosensitive cells, that activate and produce blue or red pixels on a computer screen in response to either a drop or spike in brightness.

Instead of a typical video feed, a drone with a DVS camera would “see” a grainy depiction of pixels that switch between two colors, depending on whether that point in space has brightened or darkened at any given moment. The sensor requires no image processing and is designed to enable, among other applications, high-speed autonomous flight.

Researchers have used DVS cameras to enable simple linear systems to see and react to high-speed events, and they have designed controllers, or algorithms, to quickly translate DVS data and carry out appropriate responses. For example, engineers have designed controllers that interpret pixel changes in order to control the movements of a robotic goalie to block an incoming soccer ball, as well as to direct a motorized platform to keep a pencil standing upright.

But for any given DVS system, researchers have had to start from scratch in designing a controller to translate DVS data in a meaningful way for that particular system.

“The pencil and goalie examples are very geometrically constrained, meaning if you give me those specific scenarios, I can design a controller,” Singh says. “But the question becomes, what if I want to do something more complicated?”

Cutting through the noise

In the team’s new paper, the researchers report developing a sort of universal controller that can translate DVS data in a meaningful way for any simple linear, robotic system. The key to the controller is that it identifies the ideal value for a parameter Singh calls “H,” or the event-threshold value, signifying the minimum change in brightness that the system can detect.

Setting the H value for a particular system can essentially determine that system’s visual sensitivity: A system with a low H value would be programmed to take in and interpret changes in luminosity that range from very small to relatively large, while a high H value would exclude small changes, and only “see” and react to large variations in brightness.

The researchers formulated an algorithm first by taking into account the possibility that a change in brightness would occur for every “event,” or pixel activated in a particular system. They also estimated the probability for “spurious events,” such as a pixel randomly misfiring, creating false noise in the data.

Once they derived a formula with these variables in mind, they were able to work it into a well-known algorithm known as an H-infinity robust controller, to determine the H value for that system.

The team’s algorithm can now be used to set a DVS camera’s sensitivity to detect the most essential changes in brightness for any given linear system, while excluding extraneous signals. The researchers performed a numerical simulation to test the algorithm, identifying an H value for a theoretical linear system, which they found was able to remain stable and carry out its function without being disrupted by extraneous pixel events.

“We found that this H threshold serves as a ‘sweet-spot,’ so that a system doesn’t become overwhelmed with too many events,” Singh says. He adds that the new results “unify control of many systems,” and represent a first step toward faster, more stable autonomous flying robots, such as the Robobee, developed by researchers at Harvard University.

“We want to break that speed limit of 20 to 30 miles per hour, and go faster without colliding,” Singh says. “The next step may be to combine DVS with a regular camera, which can tell you, based on the DVS rendering, that an object is a couch versus a car, in real time.”

This research was supported in part by the Singapore National Research Foundation through the SMART Future Urban Mobility project.

In this episode, Audrow Nash and Christina Brester conduct interviews at the 2016 International Association of Science Parks and Areas of Innovation conference in Moscow, Russia. They speak with Roman Luchin, CEO of CyberTech Labs., about a robotics development platform called Trik. Trik is intended to be an intermediate step when learning about robotics between Lego Mindstorms and programming on an embedded platform. Trik allows users to program with a graphical interface by ordering blocks. These blocks contain code in several common programming languages (python, F#, Pascal, etc.) and the code can be modified directly.

This is the second of three interviews from the conference.

Roman Luchin

Roman Luchin is Founder of TRIK and CEO of CyberTech Labs in Russia. He started as a researcher in a laboratory of theoretical cybernetics in Saint Petersburg State University, where he developed multi-agent control systems. Faced with two challenges in Russian robotics, young professionals and outdated approaches, he decided to devote himself to the improvement of STEM education through robotics and the creation of new educational tools. As a result, he developed a unified cybernetic platform for education, research, and robotics projects called TRIK. There is a continuously growing community around TRIK, including universities, researchers, prototyping labs, and creative teams working to provide everyone with the opportunity to create modern intelligent robots that can “see”, “hear” and collaborate.

Links

• Download mp3 (16.5 MB)
• IASP 2016 conference website
• Skolkovo’s robotics page
• Trik’s homepage
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

D’Arpino Perez discusses her work teaching Optimus to pick up a bottle. Image: Jason Dorfman, MIT CSAIL

Most robots are programmed using one of two methods: learning from demonstration, in which they watch a task being done and then replicate it, or via motion-planning techniques like optimization or sampling, which require a programmer to explicitly specify a task’s goals and constraints.

Both methods have drawbacks. Robots that learn from demonstration can’t easily transfer one skill they’ve learned to another situation and remain accurate. On the other hand, motion planning systems that use sampling or optimization can adapt to these changes, but are time-consuming, since they usually have to be hand-coded by expert programmers.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have recently developed a system that aims to bridge the two techniques: C-LEARN, which allows non-coders to teach robots a wider range of tasks simply by providing some information about how objects are typically manipulated and then showing the robot a single demo of the task. Importantly, this enables users to teach robots skills that can be automatically transferred to other robots with different “kinematics” (ways of moving) – a key time- and cost-saving measure for companies that want a range of robots to perform similar actions.

“By combining the intuitiveness of learning from demonstration with the precision of motion-planning algorithms, this approach can help robots do new types of tasks that they haven’t been able to learn before, like multi-step assembly using both of their arms,” says Claudia Pérez-D’Arpino, a PhD student who wrote a paper on C-LEARN with MIT professor Julie Shah.

The team tested the system on Optimus, a new two-armed robot designed for bomb disposal that they programmed to perform tasks like opening doors, transporting objects and extracting objects from containers.

(The robot is made of components from multiple manufacturers, including a manipulation system by RE2, a Husky unmanned ground vehicle by Clearpath, 3-finger grippers by Robotiq and the multisense sensor from Carnegie Robotics that is used in the Atlas robot.)

In simulations they showed that Optimus’ learned skills could be seamlessly transferred to Atlas, CSAIL’s six-foot-tall, 400-pound humanoid robot.

The paper was recently accepted to the IEEE International Conference on Robotics and Automation (ICRA), which takes place May 29-June 3 in Singapore.

How it works

With C-LEARN the user first gives the robot a knowledge base of information on how to reach and grasp different objects that have different constraints. (The C in C-LEARN stands for “constraints.”) The operator then uses a 3D interface to show the robot a single demonstration of the specific task, which is represented by a sequence of relevant moments known as “keyframes.” By matching these keyframes with the knowledge base, the robot can automatically suggest motion plans for the operator to approve or edit as needed.

“This approach is actually very similar to how humans learn in terms of seeing how something’s done and connecting it to what we already know about the world,” says Pérez-D’Arpino. “We can’t magically learn from a single demonstration, so we take new information and match it to previous knowledge about our environment.”

One challenge was that existing constraints that could be learned from demonstrations weren’t accurate enough to enable robots to precisely manipulate objects. To overcome that, the researchers developed constraints inspired by computer-aided design (CAD) programs that can tell the robot if its hands should be parallel or perpendicular to the objects it is interacting with.

The team also showed that the robot performed significantly better in collaboration with humans. While the robot successfully executed tasks 87.5 percent of the time on its own, it did so 100 percent of the time when it had a operator that could correct minor sensory inaccuracies.

“Having a knowledge base is fairly common, but what’s not common is integrating it with learning from demonstration,” says Dmitry Berenson, an assistant professor of computer science at the University of Michigan who was not involved in the research. “That’s very helpful, because if you are dealing with the same objects over and over again, you don’t want to then have to start from scratch to teach the robot every new task.”

The system is part of a larger wave of research focused on making LfD approaches more adaptive. If you’re a robot that has learned to take an object out of a tube from a demonstration, you might not be able to do it if there’s an obstacle in the way that requires you to move your arm differently. However, C-LEARN can do this, because it does not learn one specific way to perform the action.

“It’s good for the field that we’re moving away from directly imitating motion, towards actually trying to infer the principles behind the motion,” says Berenson. “By using these learned constraints in a motion planner, we can make systems that are far more flexible than those which just try to mimic what’s being demonstrated”

Shah says that advanced LfD methods could prove important in time-sensitive scenarios like bomb disposal and disaster response, where robots are currently tele-operated at the level of individual joint movements.

“Something as simple as picking up a box could take 20-30 minutes, which is significant for an emergency situation,” says Pérez-D’Arpino.

C-LEARN can’t yet handle certain advanced tasks, like avoiding collisions or planning for different step sequences for a given task. But the team is hopeful that incorporating more insights from human learning will give robots an even wider range of physical capabilities.

“Traditional programming of robots in real-world scenarios is difficult, tedious and requires a lot of domain knowledge,” says Shah. “It would be much more effective if we could train them more like how we train people: by giving them some basic knowledge and a single demonstration. This is an exciting step towards teaching robots to perform complex multi-arm and multi-step tasks necessary for assembly manufacturing and ship or aircraft maintenance.”

Click here to access the full paper.

Would you like to make a robot to grasp something, but you think that is impossible to you just because you can’t buy a robot arm? I’m here to tell that you can definitely achieve this without buying a real robot. Let’s see how:

How is that possible?

The Smart Grasping Sandbox built by Shadow Robotics is now available for everybody on the ROS Development Studio – a system that allows you to create and test your robot programs through simulations using only a web browser.

Wait, what is the Smart Grasping Sandbox?

The Smart Grasping Sandbox is a public simulation for the Shadow’s Smart Grasping System with theUR10 robot from Universal Robots. It allows you to make a robot to grasp something without having to learn everything related to Machine Learning, and being available on the ROS Development Studio, it allows you to test it without the hassle of installing all the requirements.

As Ugo Cupcic said:

I don’t want to have to specify every aspect of a problem — I’d rather the system learn the best way to approach a given problem itself.

Using the Development Environment for Grasping

In order to user the Smart Grasping Sandbox on ROS Development Studio—also known as RDS—just go to http://rds.theconstructsim.com and Sign In. Once logged in, go to the Public Simulations, select the Smart Grasping Sandbox simulation and press the red Launch this simulation button.

After pressing the Launch this simulation, a new screen will appears asking you to select the launch file. Just keep the default main.launch and press the red run button.

After a few seconds, the simulation will be loaded and you will be able interact with the simulation.

Once the simulation is loaded, you can see the instructions about how to control it on the left side.

In the center you have the simulation itself, and on the right side you can see the Integrated Development Environment and the Web Shell that allows you to modify the code and send Linuxcommands to the simulation respectively.

You can modify, the simulation, the control programs or even create your own manipulation control programs, make the grasping system learn using deep learning or else. This is up to you as developer!

You can also see what the robot sees using the online ROS 3D Visualizer (RViz) typing the following command on the Web Shell:

$ rosrun rviz rviz

After running Rviz, you press the red Open ROS Graphic Tools button on the bottom left side of the system and a new tab should appear with Rviz.

Conclusion

Here we have seen how to use a simulation to programme a robot to grasp something.

The simulation uses ROS as the middleware to control the robot.

If you want to learn ROS or master your ROS Skills, we recommend you to give a try to Robot Ignite Academy.

If you are a robot builder and want to have your own simulation available for everybody on RDS, or you want to have a course specifically for your robot, contact us through the email info@theconstructsim.com.

Today we are looking at how to program your NAO Robot for Human Interaction. Watch the video and follow the steps below to get interactive with your robot pal!

1.  Create an animation of movements
   • Right click, select “Create a new box”
   • Select “Timeline”

We want the robot to do 3 things, a five high, a hello, and a goodbye move:

• On the Timeline Edit Box
  • Name: High Five
  • Image: Click “Edit”
    • On the Edit box image
    • Click “Browse” and select a file
  • Click “OK”

Now, as you can remember, if we take it to the box, we can get access to the NAO’s Timeline. Mine now is connected to the PC. So what I’m going to do is I’m going to set some points of the NAO to do some movements. As I have said before on one of my tutorial I’m going to make it set all the joints to the whole body.

We want to make him look like he is doing a high five:

• Right click the arm
  • Click “Stiffen chain on/off”
  • Raise his arm up
• Set joints and keyframe
  • Select “Arms”
• Click “Play” on the Motion Controller

That’s a quite quick high five actually. What we want to do is to actually give him a pause when he got his arm in the air. So, the way we do that is to copy and paste the keyframe.

Grab a few already built in animation, just to show you how much it works:

• Go to Motions>Animations
  • Drag a “Hello” box to the workspace
  • Drag a “Wipe Forehead” box for a goodbye move to the workspace

We want to hear him speak, so:

• Go to Voice
  • Drag a “Speech Recognition” box to the workspace
• Drag a “Switch Case”
  • Connect the “Speech Recognition” box to the “Switch Case” box
  • On the first case, write “Hello” and connect to the “Hello” box
  • On the second case, write “High Five” and connect it to the “High Five” box
  • On the third case, write “Goodbye” and connect it to the “Wipe Forehand” box
• Set parameters of Speech Recognition
  • Word list: Hi; High Five; Goodbye;
  • Play with the threshold to see what works better
  • Click “OK”

So there we have a speech recognition, he will listen to what I am saying and then we will act a motion. Now, what we need to do is to make sure that the robot doesn’t repeat it, so you know, it doesn’t loop back. Before we do that, we are going to:

• Drag 3 “Say” box to the workspace so he can  speak back to us
  • On the “Switch case”, connect the “Hello” case on the “Say” box
    • Localized Text
      • Change the language to English(United States)
      • Write “Hi” on the textbox
  • Connect the “High Five” case on the “Say (1)” box
    • Localized Text
      • Change the language to English(United States)
      • Write “High Five” on the textbox
  • Connect the “Goodbye” case on the “Say(2)” box
    • Localized Text
      • Change the language to English(United States)
      • Write “Goodbye” on the textbox

So, here’s what we are saying, he recognizes it, says a word and does an animation, now what we want to do is we don’t want him to loop, as in he keeps hearing the same sound and maybe he starts to hear himself and repeat himself, so to stop that we need to connect the end of the motion boxes back to the “Speech Recognition” box. A simple output to the input speech.

I am going to make a little change actually, instead of “Hello”, we should put “Hi” here and I’m going to put “Hi” in there which is already done because they have to match otherwise if it doesn’t match he won’t recognize it. So if he hit “Load”

Philip: High Five!

Nao: High Five!

Philip: Goodbye!

Nao: Goodbye!

Philip: Hi!

Nao: Hello!

Brilliant! So that looks likes he’s working very well, so again if we says “Hi”, “High Five” or “Good bye” then he will greet you. If you walk in to room and you say “Hi” to a robot, obviously, he will do his animation and say a speech. Here you can use your imagination and try different things, different animations.

If you liked this article, you may also enjoy these from Robo-Phil:

• Python programming your NAO robot
• Hard at work: A review of the Laevo Exoskeleton

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

A few decades ago, touchscreens were impressive, yet clunky pieces of technology reserved for applications that did little more than show off that touchscreens were possible. Today’s touchscreens are commonplace and readily accepted as an easy way to interact with an operating system; they’re accurate, support multi-touch, are comfortable to use, and can even operate in multiple dimensions (pressing hard).

We may not have perfected touchscreens yet, but we’re getting close. Accordingly, engineers and researchers are already speculating about the next phase of UI development, especially for robotics control. So far, the leading candidate is gesture-based control—the use of physical gestures to relay commands.

The idea

The major limitation for touchscreens is the fact that they operate only in two dimensions; the third dimension introduced with force touch is extremely limited. Comparatively, hand gestures and physical movements can operate in three dimensions, and depending on how they’re designed, could feel more natural than swiping and tapping a smartphone screen.

Demand for three-dimensional gestural control is increasing with the onset of virtual reality (VR) and augmented reality (AR) technology; because the digital world we experience will be moving from two dimensions to three dimensions, the controls we use to manage those experiences will also need to change. With enough sophistication, these systems could provide better feedback to the users in control; rather than merely responding with visual feedback like movement or lights, users could feasibly be immersed with physical feedback like vibration or resistance.

Where we stand

Currently, one of the most advanced gestural systems is the Real-Time 3D Gesture Analysis for Natural Interaction with Smart Devices, a project headed by researchers in Linnaeus University in Sweden. However, simpler forms of gesture-based control are already available.

For example, there are projects that use Arduino to create a robot that can respond to four different simple hand gestures (plus a neutral position). Of course, iPhone technology also makes use of some simple “gestures,” such as shaking the phone to undo typing or rotating the phone to experience an immersive 360-degree view of an image or video.

The main obstacles

There are a few obstacles preventing gestures from being solidified as the next landmark in robotics control, however:

• Gesture complexity. The biggest obstacle described by researchers at Linnaeus University is the sheer complexity of physical gestures; to be effective, their recognition systems need to be able to gather and recognize thousands of tiny data points and interpret complex patterns to “understand” what movement is being attempted. Moreover, this big data-based interpretation needs to happen in real-time—especially for applications like live robotics control, or engagement in a VR-based video game. That demands not only an incredibly intelligent system, but a processor that can operate quickly.
• Accessibility. Gesture recognition systems would likely be developed to accommodate a “standard” human model, such as a “standard” right human hand. How would the system accommodate somebody whose hand was missing, or who is missing a few fingers? What about left-handed people? And people with Parkinson’s disease, or who are unable to operate their hands with precision control?
• Applications. For gestures to be accepted as a mainstream way to interact with robotics and virtual environments, robotics and virtual environments need to be available. Virtual reality (VR) technology has been advancing strongly for the past several years and is poised to take a big step forward in user adoption by 2020, but until those user adoption numbers are reliable, the demand for gestural systems is relatively low. Using hand movements and other physical gestures to control a two-dimensional screen, for example, would be largely ineffective.

On paper, gestures seem like the best form of control for the digital devices, robotics, and VR systems of the future. However, there are still many obstacles to overcome before we’re ready for large-scale adoption. Fortunately, researchers are ahead of the curve, already preparing the intelligent gesture-based recognition programs we’ll need when touchscreens become obsolete.

Researchers including MIT professor Daniela Rus (left) and research scientist Stephanie Gil (right) have developed a technique for preventing malicious hackers from commandeering robot teams’ communication networks. To verify the theoretical predictions, the researchers implemented their system using a battery of distributed Wi-Fi transmitters and an autonomous helicopter. Image: M. Scott Brauer.

Distributed planning, communication, and control algorithms for autonomous robots make up a major area of research in computer science. But in the literature on multirobot systems, security has gotten relatively short shrift.

In the latest issue of the journal Autonomous Robots, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory and their colleagues present a new technique for preventing malicious hackers from commandeering robot teams’ communication networks. The technique could provide an added layer of security in systems that encrypt communications, or an alternative in circumstances in which encryption is impractical.

“The robotics community has focused on making multirobot systems autonomous and increasingly more capable by developing the science of autonomy. In some sense we have not done enough about systems-level issues like cybersecurity and privacy,” says Daniela Rus, an Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science at MIT and senior author on the new paper.

“But when we deploy multirobot systems in real applications, we expose them to all the issues that current computer systems are exposed to,” she adds. “If you take over a computer system, you can make it release private data — and you can do a lot of other bad things. A cybersecurity attack on a robot has all the perils of attacks on computer systems, plus the robot could be controlled to take potentially damaging action in the physical world. So in some sense there is even more urgency that we think about this problem.”

Identity theft

Most planning algorithms in multirobot systems rely on some kind of voting procedure to determine a course of action. Each robot makes a recommendation based on its own limited, local observations, and the recommendations are aggregated to yield a final decision.

A natural way for a hacker to infiltrate a multirobot system would be to impersonate a large number of robots on the network and cast enough spurious votes to tip the collective decision, a technique called “spoofing.” The researchers’ new system analyzes the distinctive ways in which robots’ wireless transmissions interact with the environment, to assign each of them its own radio “fingerprint.” If the system identifies multiple votes as coming from the same transmitter, it can discount them as probably fraudulent.

“There are two ways to think of it,” says Stephanie Gil, a research scientist in Rus’ Distributed Robotics Lab and a co-author on the new paper. “In some cases cryptography is too difficult to implement in a decentralized form. Perhaps you just don’t have that central key authority that you can secure, and you have agents continually entering or exiting the network, so that a key-passing scheme becomes much more challenging to implement. In that case, we can still provide protection.

“And in case you can implement a cryptographic scheme, then if one of the agents with the key gets compromised, we can still provide  protection by mitigating and even quantifying the maximum amount of damage that can be done by the adversary.”

Hold your ground

In their paper, the researchers consider a problem known as “coverage,” in which robots position themselves to distribute some service across a geographic area — communication links, monitoring, or the like. In this case, each robot’s “vote” is simply its report of its position, which the other robots use to determine their own.

The paper includes a theoretical analysis that compares the results of a common coverage algorithm under normal circumstances and the results produced when the new system is actively thwarting a spoofing attack. Even when 75 percent of the robots in the system have been infiltrated by such an attack, the robots’ positions are within 3 centimeters of what they should be. To verify the theoretical predictions, the researchers also implemented their system using a battery of distributed Wi-Fi transmitters and an autonomous helicopter.

“This generalizes naturally to other types of algorithms beyond coverage,” Rus says.

The new system grew out of an earlier project involving Rus, Gil, Dina Katabi — who is the other Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science at MIT — and Swarun Kumar, who earned master’s and doctoral degrees at MIT before moving to Carnegie Mellon University. That project sought to use Wi-Fi signals to determine transmitters’ locations and to repair ad hoc communication networks. On the new paper, the same quartet of researchers is joined by MIT Lincoln Laboratory’s Mark Mazumder.

Typically, radio-based location determination requires an array of receiving antennas. A radio signal traveling through the air reaches each of the antennas at a slightly different time, a difference that shows up in the phase of the received signals, or the alignment of the crests and troughs of their electromagnetic waves. From this phase information, it’s possible to determine the direction from which the signal arrived.

Space vs. time

A bank of antennas, however, is too bulky for an autonomous helicopter to ferry around. The MIT researchers found a way to make accurate location measurements using only two antennas, spaced about 8 inches apart. Those antennas must move through space in order to simulate measurements from multiple antennas. That’s a requirement that autonomous robots meet easily. In the experiments reported in the new paper, for instance, the autonomous helicopter hovered in place and rotated around its axis in order to make its measurements.

When a Wi-Fi transmitter broadcasts a signal, some of it travels in a direct path toward the receiver, but much of it bounces off of obstacles in the environment, arriving at the receiver from different directions. For location determination, that’s a problem, but for radio fingerprinting, it’s an advantage: The different energies of signals arriving from different directions give each transmitter a distinctive profile.

There’s still some room for error in the receiver’s measurements, however, so the researchers’ new system doesn’t completely ignore probably fraudulent transmissions. Instead, it discounts them in proportion to its certainty that they have the same source. The new paper’s theoretical analysis shows that, for a range of reasonable assumptions about measurement ambiguities, the system will thwart spoofing attacks without unduly punishing valid transmissions that happen to have similar fingerprints.

“The work has important implications, as many systems of this type are on the horizon — networked autonomous driving cars, Amazon delivery drones, et cetera,” says David Hsu, a professor of computer science at the National University of Singapore. “Security would be a major issue for such systems, even more so than today’s networked computers. This solution is creative and departs completely from traditional defense mechanisms.”

If you enjoyed this article from CSAIL, you might also be interested in:

• Mind control: Correcting robot mistakes using EEG brain signals
• MIT CSAIL’s 6-foot-tall NASA humanoid robot has landed
• Researchers add a splash of human intuition to planning algorithms
• Wearable AI that can detect the tone of a conversation
• Ingestible robots, glasses-free 3-D, and computers that explain themselves

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

You probably know the Sphero robot. It is a small robot with the shape of a ball. In case that you have one, you must know that it is possible to control it using ROS, by installing in your computer the Sphero ROS packages developed by Melonee Wise and connecting to the robot using the bluetooth of the computer.

Now, you can use the ROS Development Studio to create ROS control programs for that robot, testing as you go by using the integrated simulation.

The ROS Development Studio (RDS) provides off-the-shelf a simulation of Sphero with a maze environment. The simulation provides the same interface as the ROS module created by Melonee, so you can test your develop and test the programs on the environment, and once working properly, transfer it to the real robot.

We created the simulation to teach ROS to the students of the Robot Ignite Academy. They have to learn ROS enough to make the Sphero get out of the maze by using odometry and IMU.

Using the simulation

To use the Sphero simulation on RDS go to rds.theconstructsim.com and sign in. If you select the Public simulations, you will quickly identify the Sphero simulation.

Press the red Play button. A new screen will appear giving you details about the simulation and asking you which launch file you want to launch. The main.launch selected by default is the correct one, so just press Run.

After a few seconds the simulation will appear together with the development environment for creating the programs for Sphero and testing them.

On the left hand side you have a notebook containing information about the robot and how to program it with ROS. This notebook contains just some examples, but it can be completed and/or modified at your will. As you can see it is an iPython notebook and follows its standard. So it is up to you to modify it, add new information or else. Remember that any change you do to the notebook will be saved in a simulation in your private area of RDS, so you can come back later and launch it with your modifications.

You must know that the code included in the notebook is directly executable by selecting the cell of the code (do a single click on it) and pressing the small play button at the top of the notebook. This means that, once you press that button, the code will be executed and start controlling the Sphero simulated robot for a few time-steps (remember to have the simulation activated (Play button of the simulation activated) to see the robot move).

On the center area, you can see the IDE. It is the development environment for developing the code. You can browse there all the packages related to the simulation or any other packages that you may create.

On the right hand side, you can see the simulation and beneath it, the shell. The simulation shows the Sphero robot as well as the environment of the maze. On the shell, you can issue commands in the computer that contains the simulation of the robot. For instance, you can use the shell to launch the keyboard controller and move the Sphero around. Try typing the following:

• $ roslaunch sphero_gazebo keyboard_teleop.launch

Now you must be able to move the robot around the maze by pressing some keys of the keyboard (instructions provided on the screen).

You can also launch there Rviz, and then watch the robot, the frames and any other additional information you may want of the robot. Type the following:

• $ rosrun rviz rviz

Then press the Screen red icon located at the bottom-left of the screen (named the graphical tools). A new tab should appear, showing how the Rviz is loading. After a while, you can configure the Rviz to show the information you desire.

There are many ways you can configure the screen to provide more focus to what interests you the most.

To end this post, I would like to indicate that you can download the simulation to your computer at any time, by doing right-click on the directories and selecting Download. You can also clone the The Construct simulations repository to download it (among other simulations available).

If you liked this tutorial, you may also enjoy these:

• UgCS photogrammetry technique for UAV land surveying missions
• Programming for robotics: Introduction to ROS
• Python programming your NAO robot
• Robotics, maths, python: A fledgling computer scientist’s guide to inverse kinematics
• 3-D printing hydraulically-powered robots, no assembly required

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Level 3 automation, where the car handles all aspects of driving with the driver on standby, is being tested in Sweden. Image courtesy of Volvo cars

Today’s self-driving car isn’t exactly autonomous – the driver has to be able to take over in a pinch, and therein lies the roadblock researchers are trying to overcome. Automated cars are hurtling towards us at breakneck speed, with all-electric Teslas already running limited autopilot systems on roads worldwide and Google trialling its own autonomous pod cars.

However, before we can reply to emails while being driven to work, we have to have a foolproof way to determine when drivers can safely take control and when it should be left to the car.

‘Even in a limited number of tests, we have found that humans are not always performing as required,’ explained Dr Riender Happee, from Delft University of Technology in the Netherlands, who is coordinating the EU-funded HFAuto project to examine the problem and potential solutions.

‘We are close to concluding that the technology always has to be ready to resolve the situation if the driver doesn’t take back control.’

But in these car-to-human transitions, how can a computer decide whether it should hand back control?

‘Eye tracking can indicate driver state and attention,’ said Dr Happee. ‘We’re still to prove the practical usability, but if the car detects the driver is not in an adequate state, the car can stop in the safety lane instead of giving back control.’

Next level

It’s all a question of the level of automation. According to the scale of US-based standards organisation SAE International, Level 1 automation already exists in the form of automated braking and self-parking.

Level 4 & 5 automation, where you punch in the destination and sit back for a nap, is still on the horizon.

But we’ll soon reach Level 3 automation, where drivers can hand over control in situations like motorway driving and let their attention wander, as long as they can safely intervene when the car asks them to.

HFAuto’s 13 PhD students have been researching this human-machine transition challenge since 2013.

Backed with Marie Skłodowska-Curie action funding, the students have travelled Europe for secondments, to examine carmakers’ latest prototypes, and to carry out simulator and on-road tests of transition takeovers.

Alongside further trials of their transition interface, HFAuto partner Volvo has already started testing 100 highly automated Level 3 cars on Swedish public roads.

Another European research group is approaching the problem with a self-driving system that uses external sensors together with cameras inside the cab to monitor the driver’s attentiveness and actions.

Blink

‘Looking at what’s happening in the scene outside of the cars is nothing without the perspective of what’s happening inside the car,’ explained Dr Oihana Otaegui, head of the Vicomtech-IK4 applied research centre in San Sebastián, Spain.

She coordinates the work as part of the EU-funded VI-DAS project. The idea is to avoid high-risk transitions by monitoring factors like a driver’s gaze, blinking frequency and head pose — and combining this with real-time on-road factors to calculate how much time a driver needs to take the wheel.

Its self-driving system uses external cameras as affordable sensors, collecting data for the underlying artificial intelligence system, which tries to understand road situations like a human would.

VI-DAS is also studying real accidents to discern challenging situations where humans fail and using this to help train the system to detect and avoid such situations.

The group aims to have its first interface prototype working by September, with iterated prototypes appearing at the end of 2018 and 2019.

Dr Otaegui says the system could have potential security sector uses given its focus on creating artificial intelligence perception in any given environment, and hopes it could lead to fully automated driving.

‘It could even go down the path of Levels 4 and 5, depending on how well we can teach our system to react — and it will indeed be improving all the time we are working on this automation.’

The question of transitions is so important because it has an impact on liability – who is responsible in the case of an accident.

It’s clear that Level 2 drivers can be held liable if they cause a fender bender, while carmakers will take the rap once Level 4 is deployed. However, with Level 3 transitions, liability remains a burning question.

HFAuto’s Dr Happee believes the solution lies in specialist insurance options that will emerge.

‘Insurance solutions are expected (to emerge) where a car can be bought with risk insurance covering your own errors, and those which can be blamed on carmakers,’ he said.

Yet it goes further than that. Should a car choose to hit pedestrians in the road, or swerve into the path of an oncoming lorry, killing its occupants?

‘One thing coming out of our discussions is that no one would buy a car which will sacrifice its owner for the lives of others,’ said Dr Happee. ‘So it comes down to making these as safe as possible.’

The five levels of automation:

1. Driver Assistance: the car can either steer or regulate speed on its own.
2. Partial Automation: the vehicle can handle both steering and speed selection on its own in specific controlled situations, such as on a motorway.
3. Conditional Automation: the vehicle can be instructed to handle all aspects of driving, but the driver needs to be on standby to intervene if needed.
4. High Automation: the vehicle can be instructed to handle all aspects of driving, even if the driver is not available to intervene.
5. Level 5 – Full Automation: the vehicle handles all aspects of driving, all the time.

Figure 15: Adding 40m overshot to both ends of each survey line

UgCS is easy-to-use software for planning and flying UAV drone-survey missions. It supports almost any UAV platform, providing convenient tools for areal and linear surveys and enabling direct drone control. What’s more, UgCS enables professional land survey mission planning using photogrammetry techniques.

How to plan photogrammetry mission with UgCS

Standard land surveying photogrammetry mission planning with UgCS can be divided into following steps :

1. Obtain input data
2. Plan mission
3. Deploy ground control points
4. Fly mission
5. Image geotagging
6. Data processing
7. Map import to UgCS (optional)

https://www.youtube.com/watch?v=SalCQoG6WUw&feature=youtu.be

Step one: Obtain input data

Firstly, to reach the desired result, input settings have to be defined:

• Required GSD (ground sampling distance – size of single pixel on ground),
• Survey area boundaries,
• Required forward and side overlap.

GSD and area boundaries are usually defined by the customer’s requirements for output material parameters, for example by scale and resolution of digital map. Overlap should be chosen according to specific conditions of surveying area and requirements of data processing software.

Each data processing software (e.g., Pix4D, Agisoft Photoscan, Dronedeploy, Acute 3d) has specific requirements for side and forward overlaps for different surfaces. To choose correct values, please refer to documentation of chosen software. In general, 75% forward and 60% side overlap will be a good choice. Overlapping should be increased for areas with small amount of visual cues, for example for deserts or forests.

Often, aerial photogrammetry beginners are excited about the option to produce digital maps with extremely high resolution (1-2cm/pixel), and to use very small GSD for mission planning. This is very bad practice. Small GSD will result in longer flight time, hundreds of photos for each acre, tens of hours of processing and heavy output files. GSD should be set according to the output requirements of the digital map.

Other limitations can occur. For example, GSD of 10cm/pixel is required, but designed to use a Sony A6000 camera. Based on mentioned GSD and camera’s parameters, the flight altitude would be set to 510 meters. In most countries, maximum allowed altitude of UAV’s (without special permission) is limited to 120m/400ft AGL (above ground). Taking into account the maximum allowed altitude, the maximum possible GSD in this case could be no more than 2.3cm.

Step two: Plan your mission

Mission planning consists of two stages:

• Initial planning,
• Route optimisation.

-Initial planning:

The first step is to set surveying area using UgCS Photogrammetry tool. Area can be set using visual cues on underlying map or using exact coordinates of edges. The result – survey area is marked with yellow boundaries (Figure 1).

Figure 1: Setting the survey area

The next step is to set GSD and overlapping for the camera in Photogrammetry tool’s settings window (Figure 2).

Figure 2: Setting camera’s Ground Sampling Distance and overlapping

To take photos in Photogrammetry tool’s setting window, define the control action of the camera (Figure 3). Set camera by distance triggering action with default values.

Figure 3: Setting camera’s control action

At this point, initial route planning is completed. UgCS will automatically calculate photogrammetry route (see Figure 4).

Figure 4: Calculated photogrammetry survey route before optimisation

-Route optimisation

To optimise the route, it’s calculated parameters should be known: altitude, estimated flight time, number of shots, etc.

Part of the route’s calculated information can be found in the elevation profile window. To access the elevation profile window (if it is not visible on screen) click the parameters icon on the route card (lower-right corner, see Figure 5), and from the drop-down menu select show elevation:

Figure 5: Accessing elevation window from Route cards Parameters settings

The elevation profile window will present an estimated route length, duration, waypoint count and min/max altitude data:

Figure 6: Route values in elevation profile window

To get other calculated values, open route log by clicking on route status indicator: the green check-mark (upper-right corner, see Figure 7) of the route card:

Figure 7: Route card and status indicator, Route log

Using route parameters, it can be optimised to be more efficient and safe.

-Survey line direction

By default, UgCS will trace survey lines from south to north. But, in most cases, it will be more optimal to fly parallel to the longest boundary line of the survey area. To change survey line direction, edit direction angle field in the photogrammetry tool. In the example, by changing angle to 135 degrees, the number of passes is reduced from five (Figure 4) to four (Figure 8) and route length is 1km instead of 1.3km.

Figure 8: Changed survey line angle to be parallel to longest boundary

-Altitude type

UgCS Photogrammetry tool has the option to define how to trace the route according to altitude, with constant altitude above ground (AGL) or above mean sea level (AMSL). Please refer to your data processing software requirements as to which altitude tracking method it recommend.

In the UgCS team’s experience, the choice of altitude type depends on desired result. For orthophotomap (standard aerial land survey output format) it is better to choose AGL to ensure constant GSD for the entire map. If the aim is to produce DEM or 3D reconstruction, use AMSL so the data processing software has more data to correctly determine ground elevation by photos in order to provide more qualitative output.

Figure 9: Elevation profile with constant altitude above mean sea level (AMSL)

In this case, UgCS will calculate flight altitude based on the lowest point of the survey area.

If AGL is selected in photogrammetry tool’s settings, UgCS will calculate the altitude for each waypoint. But in this case, terrain following will be rough if no “additional waypoints” are added (see Figure 10).

Figure 10: Elevation profile with AGL without additional waypoints

Therefore, if AGL is used, add some “additional waypoints” flags and UgCS will calculate a flight plan with elevation profile accordingly (see Figure 11).

Figure 11: Elevation profile with AGL with additional waypoints

-Speed

In general, if flight speed is increased it will minimise flight time. But high speed in combination with large camera exposure can result in blurred images. In most cases 10m/s is the best choice.

-Camera control method

UgCS supports 3 camera control methods (actions):

1. Make a shot (trigger camera) in waypoint,
2. Make shot every N seconds,
3. Make shot every N meters.

Not all autopilots support all 3 camera control options. For example (quite old) DJI A2 does support all three options, but newer (starting from Phantom 3 and up to M600) cameras support only triggering in waypoints and by time. DJI promised to implement triggering by distance, but it’s not available yet.

Here are some benefits and drawbacks for all three methods:

Table 1: Benefits and Drawback for camera triggering methods

In conclusion:

• Trigger in waypoints should be preferred when possible
• Trigger by time should be used only if no other method is possible
• Trigger by distance should be used when triggering in waypoints is not possible to use

To select triggering method in UgCS Photogrammetry tool accordingly, use one of three available icons:

• Set camera mode
• Set camera by time
• Set camera by distance

-Glibal control

Drones, e.g., DJI Phantom 3, Phantom 4, Inspire, M100 or M600 with integrated gimbal, have the option to control camera position as part of an automatic route plan.

It is advisable to set camera to nadir position in the first waypoint, and in horizontal position before landing to prevent lenses from potential damage.

To set camera position, select the waypoint preceding the photogrammetry area and click set camera attitude/zoom (Figure 12) and enter “90” in the “Tilt” field (Figure 13).

Figure 12: Setting camera attitude

Figure 13: Setting camera position

As described previously, this waypoint should be a Stop&Turn type, otherwise the drone could skip this action.

To set camera to horizontal position, select last waypoint of survey route and click set camera attitude/zoom and enter “0” in the “Tilt” field.

-Turn types

Most autopilots or multirotor drones support different turn types in waypoints. Most popular DJI drones have three turn-types:

• Stop and Turn: drone flies to the fixed point accurately, stays at that fixed point and then flies to next fixed point.
• Bank Turn: the drone would fly with constant speed from one point to another without stopping.
• Adaptive Bank Turn: It is almost the same performance like Bank Turn mode (Figure 13), but the real flight routine will be more accurately than Bank Turn.

It is advisable not to use Bank Turn for photogrammetry missions. Drone interprets Bank Turns as “recommendation destination waypoint”. The drone will fly towards this direction but will almost never pass through the waypoint. Because drone will not pass the waypoint, no action will be executed, meaning the camera will not be triggered, etc.

Adaptive Bank Turn should be used with caution because a drone can miss waypoints and, again, no camera triggering will be initiated.

Figure 14: Illustration of typical DJI drone trajectories for Bank Turn and Adaptive Bank Turn types

Sometimes, adaptive bank turn type has to be used in order to have shorter flight time compared to stop and turn. When using adaptive bank turns, it is recommended to use overshot (see below) for the photogrammetry area.

-Overshot

Initially overshot was implemented for fixed wing (airplane) drones in order to have enough space for manoeuvring a U-turn.

Overshot can be set in photogrammetry tool to add an extra segment to both ends of each survey line.

Figure 15: Adding 40m overshot to both ends of each survey line

In the example (Figure 15) can be seen that UgCS added 40m additional segments to both ends of each survey line (comparing to Figure 8).

Adding overshot is useful for copter-UAVs in two situations:

1. When Adaptive Bank Turns are used (or similar method for non-DJI drones), adding overshot will increase the chance that drone will precisely enter survey line and camera control action will be triggered. UgCS Team recommends to specify overshot that is approximately equal to distance between the parallel survey lines.
2. When Stop and Turn type is in use in combination with action to trigger camera in waypoints, there is a possibility that before making the shot, drone will start rotation to next waypoint – it can result in having photos with wrong orientation or blurred. To avoid that, shorter overshot has to be set, for example 5m. Don’t specify too short value (< 3m) because some drones could ignore waypoints, that are too close.

Figure 16: Example of blurred image taken by drone in rotation to next waypoint

-Takeoff point

It is important to check the takeoff area at site before flying any mission! To better explain best practice on how to set takeoff point, first discuss an example of how it should not be done. Supposing that the takeoff point in our example mission (Figure 17) would be from the point marked with the airplane-icon, and drone pilot would upload the route on the ground with set automatic mission for automatic take-off.

Figure 17: Take-off point example

Most drones in automatic takeoff mode would climb to low altitude about 3-10meters and then fly straight towards the first waypoint. Other drones would fly towards first waypoint straight from ground. Looking closely at the example map (Figure 17), some trees between the takeoff point and the first waypoint can be noticed. In this example, the drone more likely will not reach a safe altitude and will hit the trees.

Not only the surroundings can affect takeoff planning. Drone manufacturers can change drones elevation behavior in drone firmware, therefore after firmware updates it is recommended that you check drones automatic takeoff mode.

Also, a very important consideration is that most small UAVs use relative altitude for mission planing. Altitude counted relatively according to first waypoint is a second reason why an actual takeoff point should be near the first waypoint, and on the same terrain level.

UgCS Team recommends placing the first waypoint as close as possible to actual takeoff point and specifying a safe takeoff altitude (≈30m in most situations will be above any trees, see Figure 18). This is the only method that warrants safe takeoff for any mission. It also protects from any weird drone behaviour and unpredictable firmware updates, etc.

Figure 18: Route with safe take-off

-Entry point to the survey grid

In the previous example, (see Figure 18), it can be noticed, that after adding the takeoff point, the route’s survey grid entry point was changed. This is because if additional waypoint is added subsequently to the photogrammetry area, UgCS will plan to fly the survey grid starting from nearest corner to the previous waypoint.

To change the entry point to survey grid, set additional waypoint close to the desired starting corner (see Figure 19).

Figure 19: Changing survey grid entry point by adding additional waypoint

-Landing point

If no landing point will be added outside the photogrammetry area after the survey mission, the drone will fly and hover in the last waypoint. There are two options for landing:

1. Take manual control over the drone and fly to landing point manually,
2. Activate the Return Home command in UgCS or from Remote Controller (RC).

In situations when the radio link with the drone is lost, for example if the survey area is large or there are problems with the remote controller, depending on the drone and it’s settings, one of these actions can occur:

• Drone will return to home location automatically if lost radio link with ground station,
• Drone will fly to last waypoint of survey area and hover as long as battery capacity will enable that, then: drone will perform emergency landing, or it will try to fly to home location.

The recommendation is to add an explicit landing points to the route in order to avoid relying on unpredictable drone behavior or settings.

If the drone doesn’t support automatic landing, or the pilot prefers to land manually, place the route’s last waypoint over the planned landing point with an altitude for comfortable manual drone descending and landing above any obstacles in the surrounding area. In general 30m is best choice.

-Action execution

Photogrammetry tool has a magic parameter “Action Execution” with three possible values:

• Every point
• At start
• Forward passes

This parameter defines how and where camera actions specified for photogrammetry tool will be executed.

The most useful option for photogrammetry/survey missions is to set forward passes, the drone will make photos only on survey lines, but will not make excess photos on perpendicular lines.

-Complex survey areas

UgCS enables photogrammetry/survey mission planning for irregular areas, having functionality to combine any number of photogrammetry areas in one route, avoiding splitting the area in separate routes.

For example, if a mission has to be planned for two fields connected in a T-shape, and if these two fields are marked as one photogrammetry area, the whole route will not be optimal, regardless any direction of survey lines.

Figure 20: Complex survey area before optimisation

If the survey area is marked as two photogrammetry areas within one route, survey lines for each area can be optimised individually (see Figure 21).

Figure 21: Optimised survey flight passes for each part of a complex photogrammetry area

Step three: deploy ground control points

Ground control points are mandatory if the survey output map has to be precisely aligned to coordinates on Earth.

There are lots of discussions about the necessity of ground control points in cases when a drone is equipped with Real Time Kinematics (RTK) GPS receivers with centimeter-level accuracy. This is useful, but the drone coordinates are not in themselves sufficient because, for precise map aligning, image center coordinates are necessary.

Data processing softwares like Agisoft Photoscan, Dronedeplay, Pix4d, Icarus OneButton and others will produce very accurate maps using geotagged images, but the real precision of the map will not be known without ground control points.

Conclusion: ground control points have to be used to create survey-grade result. For a map with approximate precision, it is sufficient to rely just on RTK GPS and the capabilities of data processing software.

Step four: fly your mission

For carefully planned missions, flying it is the most straightforward step. Mission execution differs according to the type of UAV and equipment used, therefore it will not be described in detail in this article (please refer to equipment’s and UgCS documentation).

Important issues before flying:

• In most countries there are strict regulations for UAV usage. Always comply with the regulations! Usually these rules can be found on web-site of local aviation authority.
• In some countries special permission for any kind of aerial photo/video shooting is needed. Please check local regulations.
• In most cases missions are planned before arriving to flying location (e.g., in office, at home) using satellite imaginary from Google maps, Bing, etc. Before flying always check actual circumstances at the location. There could be a need to adjust take-off/landing points, for example, to avoid tall obstacles (e.g., trees, masts, power lines) in your survey area.

Step five: image geotagging

Image geotagging is optional if ground control points were used, but almost any data processing software will require less time to process geotagged images.

Some of the latest and professional drones with integrated cameras can geotag images automatically during flight. In other cases images can be geotagged in UgCS after flight.

Very important: UgCS uses the telemetry log from drone, that is received via radio channel, to extract the drone’s altitude for any given moment (when pictures were taken). To geotag pictures using UgCS, assure robust telemetry reception during flight.

For detailed information how to geotag images using UgCS refer to UgCS User Manual.

Step six: data processing

For data processing, use third party software or services available on the market.

From UgCS Team experience, the most powerful and flexible software is Agisoft Photoscan (http://www.agisoft.com/), but sometimes too much user input is required to get necessary results. The most uncomplicated solution for users is online service Dronedeploy (https://www.dronedeploy.com/). All other software packages and services will fit somewhere between these two in terms of complexity and power.

Step seven (optional): import created map to UgCS

Should the need arise for the mission to be repeated in the future, UgCS enables importing the GeoTiff file as a map layer and using it for mission planning. More detailed instructions can be found in UgCS User Manual. See the result of an imported map created using UgCS photogrammetry tool imported as GeoTiff file in Figure 22.

Figure 22: Imported GeoTiff map as layer. The map is output of a Photogrammetry survey mission panned with UgCS

Visit the UgCS homepage

Download this tutorial as a PDF

If you liked this tutorial, you may also enjoy these:

• Programming for robotics: Introduction to ROS
• Python programming your NAO robot
• Robotics, maths, python: A fledgling computer scientist’s guide to inverse kinematics
• Adjust how you learn and quickly pick up robotics programming
• 3-D printing hydraulically-powered robots, no assembly required

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

This handy video-tutorial course gives an introduction to the Robot Operating System (ROS), including many of the available tools that are commonly used in robotics. With the help of different examples, the tutorials offer a great starting point to learn programming robots. You will learn how to create software including simulation, to interface sensors and actuators, and to integrate control algorithms.

The course consists of a guided tutorial and exercises with increasing level of difficulty working with an autonomous robot. We provide recordings of the lectures and give an introduction to the exercises. From the course website, you can download all the material including exercise sheets and templates, and use the provided Virtual Machine (VM) image to start programming right away.

Objectives:

• ROS architecture: Master nodes, topics, messages, services, parameters and actions
• Console commands: Navigating and analyzing the ROS system and the catkin workspace
• Creating ROS packages: Structure, launch-files, and best practices
• ROS C++ client library (roscpp): Creating your own ROS C++ programs
• Simulating with ROS: Gazebo simulator, robot models (URDF) and simulation environments (SDF)
• Working with visualizations (RViz) and user interface tools (rqt)
• Inside ROS: TF transformation system, time, bags

Course 1:

Course 2:

Course 3:

Course 4:

Download the full presentation here.

P. Fankhauser, D. Jud, M. Wermelinger, M. Hutter, “Programming for Robotics – Introduction to ROS”, ETH Zurich, 2017. DOI: 10.13140/RG.2.2.14140.44161

If you liked this tutorial, you may also enjoy these:

• Python programming your NAO robot
• Robotics, maths, python: A fledgling computer scientist’s guide to inverse kinematics
• Sticky business: Five adhesives tested for 3D printing
• Adjust how you learn and quickly pick up robotics programming
• 3-D printing hydraulically-powered robots, no assembly required

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Samsung Simband. Image: Jason Dorfman, MIT CSAIL

It’s a fact of nature that a single conversation can be interpreted in very different ways. For people with anxiety or conditions like Asperger’s, this can make social situations extremely stressful. But what if there was a more objective way to measure and understand our interactions?

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) say that they’ve gotten closer to a potential solution: an artificially intelligent wearable system that can predict if a conversation is happy, sad or neutral based on a person’s speech patterns and vitals.

“Imagine if, at the end of a conversation, you could rewind it and see the moments when the people around you felt the most anxious,” says graduate student Tuka Alhanai, who co-authored a related paper with PhD candidate Mohammad Ghassemi that they will present at next week’s Association for the Advancement of Artificial Intelligence (AAAI) conference in San Francisco. “Our work is a step in this direction, suggesting that we may not be that far away from a world where people can have an AI social coach right in their pocket.”

As a participant tells a story, the system can analyze audio, text transcriptions and physiological signals to determine the overall tone of the story with 83 percent accuracy. Using deep-learning techniques, the system can also provide a specific sentiment score for five-second intervals within a conversation.

“As far as we know, this is the first experiment that collects both physical data and speech data in a passive but robust way, even while subjects are having natural, unstructured interactions,” says Ghassemi. “Our results show that it’s possible to classify the emotional tone of conversations in real-time.”

The team is keen to point out that they developed the system with privacy strongly in mind: the algorithm runs locally on a user’s device as a way of protecting personal information. (Alhanai says that a consumer version would obviously need clear protocols for getting consent from the people involved in the conversations.)

How it works

Past studies in this area would often show participants “happy” and “sad” videos, or ask them to artificially act out specific emotive states. But in an effort to elicit more organic emotions, the team instead asked subjects to tell a happy or sad story of their own choosing.

Subjects wore a Samsung Simband, a device that captures high-resolution physiological waveforms to measure features like movement, heart rate, blood pressure, blood flow and skin temperature,. The system also captured audio data and text transcripts to analyze the speaker’s tone, pitch, energy, and vocabulary.

Mohammad Ghassemi and Tuka Alhanai converse with the wearable. Image: Jason Dorfman MIT CSAIL

“The team’s usage of consumer market devices for collecting physiological data and speech data shows how close we are to having such tools in everyday devices,” says Björn Schuller, professor and chair of Complex and Intelligent Systems at the University of Passau in Germany who was not involved in the research. “Technology could soon feel much more emotionally intelligent, or even ‘emotional’ itself.”

After capturing 31 different conversations of several minutes each, the team trained two algorithms on the data: one classified the overall nature of a conversation as either happy or sad, while the second classified each five-second block of every conversation as either positive, negative or neutral.

Alhanai notes that, in traditional neural networks, all features about the data are provided to the algorithm at the base of the network. In contrast, her team found that they could improve performance by organizing different features at the various layers of the network.

“The system picks up on how, for example, the sentiment in the text transcription was more abstract than the raw accelerometer data, says Alhanai. “It’s quite remarkable that a machine could approximate how we humans perceive these interactions, without significant input from us as researchers.”

Results

Indeed, the algorithm’s findings align well with what we humans might expect to observe. For instance, long pauses and a monotonous vocal tones were associated with sadder stories, while more energetic, varied speech patterns were associated with happier ones. In terms of body language, sadder stories were also strongly associated with increased fidgeting and cardiovascular activity, as well as certain postures like putting one’s hands on one’s face.

Graph showing real-time emotion detection. Image: MIT CSAIL

On average, the model could classify the mood of each five-second interval with an accuracy that was approximately 18 percent above chance, and a full 7.5 percent better than existing approaches. In future work, the team hopes to collect data on a much larger scale, potentially using commercial devices like the Apple Watch that would allow them to more easily deploy the system out in the world.

“Our next step is to improve the algorithm’s emotional granularity so it can call out boring, tense, and excited moments with greater accuracy instead of just labeling interactions as ‘positive’ or ‘negative’,” says Alhani. “Developing technology that can take the pulse of human emotions has the potential to dramatically improve how we communicate with each other.”

Click here to download the research paper.

You might also enjoy the following articles:

• Artificial intelligence and ethics: Who does the thinking?
• Soft exosuit economies: Understanding the costs of lightening the load
• MIT Media Lab to participate in $27 million initiative on AI ethics and governance
• How artificial intelligence is changing our Christmas shop
• White House report: Artificial intelligence, automation, and the economy

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Jussi Aaltonen explaining some robotics-specific topics.

The UNEXMIN (Underwater Explorer for Flooded Mines) project is almost one year old. After a busy first year of work, UNEXMIN is on-schedule to deliver the first mechanical UX-1 prototype. Jussi Aaltonen, from TUT (Tampere University of Technology), leaders of WP1 – Robotic Functions Validations, talks about what has been done over the past year concerning his team’s work in UNEXMIN’s development scene.

Luís Lopes, LPRC: What is WP1 (Robotic Functions Validations) and what has been developed from it?

Jussi Aaltonen: The purpose of WP1 is to develop and validate a robotic platform, i.e. all mechanical and mechatronic structures, systems and components of UX-1 (the name of the robot). Also, low level (directional) control principles are developed and validated in this work package. WP1 has successfully established a robotic platform which fulfils all necessary requirements.

LL: How important is the work developed in WP1 for the future UX-1 robots

JA: Work in WP1 is fundamental for the successful completion of the project. All later development stages rely on this work.

LL: The systems/subsystems that the robot will employ were defined in this WP. What can you tell about them? How crucial are they for the robot’s overall functionality?

JA: All systems are needed for the robot to achieve full functionality. However, survivability, i.e. the robot’s capability to return from its missions, is more heavily affected by some systems than others. It is, for example, possible to return with a broken multi-spectral camera, but losing a sonar may cause the robot to be lost.

LL: What kinds of tests were developed to test the hardware/software systems? How about the results obtained, were they positive?

JA: Tests are done as so-called hardware-in-the-loop (HWIL) tests in laboratory. In HWIL testing, computer simulations and real hardware components interact, making it possible to do laboratory testing more closely to real operating conditions. Components and systems have also been tested in the pool by using a plastic mock-up of UX-1. Tests have been successful, which, unfortunately, does not always mean positive results. But weaknesses were found and lessons learned.

LL: In what conditions was the prototype tested?

JA: Tests have been done in shallow depths (below 5 m) and moderate temperatures. Pressure testing at 60 bars (equivalent of 600 m depth) will start soon.

LL: The robot is designed considering stakeholders’ wants. How hard was it to define the requirements and specifications for the UX-1 robots?

JA: I have to say that requirements and specifications are the easy part. The hardest is to design a robot complying them all.

LL: Being leader of such an important WP of the UNEXMIN project must be challenging. Has it worked out as you imagined? Do you feel that everything has been accomplished?

JA: Over this past year of cooperation, UNEXMIN teams from the various partners have gotten to know each other and each other’s ways of working. The possibilities to meet the others in person and to really collaborate in real life conditions, such as in workshops, have been a great benefit and a very important thing, bonding people from all around Europe together and working towards  a common goal. Of course there are always some challenges. The biggest challenge has been finding a common and efficient enough platform for information exchange.

LL: TUT will organise a UNEXMIN workshop in the beginning of February (2017) in Tampere, Finland. What can be expected to come out of this?

JA: This event will conclude the work of the first year in the UNEXMIN project. Most of the work packages have already started or about to start (exception goes to WP7: Demonstration/Pilots), and thus we will hear a lot about results and developments in this meeting. We will also hear about the plans for a couple of essential work packages starting right after the meeting. I expect that we will have a good and fruitful meeting with the same good spirit and warm atmosphere that we have always had. There will probably be some heated discussions, but that is only a sign of people being passionate about their work.

LL: You are a specialist in the robotics field. What possible impact do you think the UNEXMIN project will have on the robotics field?

JA: UX-1 [UNEXMIN robot design-prototypes] will have a very different structure and layout than what autonomous underwater vehicles usually have. This will most probably have some effects on the field of robotics and will open new doors to further miniaturised underwater robots – the future starts here. Also, the environment where UX-1 will operate is probably the most challenging after Mars and asteroid landers and that will also give tiger leaps in the field of robotics, developing new and better robotic systems.

LL: After one year, how do you feel the UNEXMIN project has progressed?

JA: We are roughly in the place where we should be according to our plans. Work has been harder than what was expected. On the other hand, we learned a lot more, while still managing to keep pace.

This post was originally published on UNEXMIN.

If you enjoyed this article, you might also be interested in:

• Surveying flooded mines with UNEXMIN: Interview with Norbert Zajzon
• When robots become art: Interview with Yi-Wei Keng
• A European perspective on robot law: Interview with Mady Delavaux
• Art meets robotics in Patrick Tresset’s OPLINE Prize entry
• Robots Podcast #171: Robotics in theatre, film and television, with Grant Imahara and Richard McKenna

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this episode, Ron Vanderkley spoke to Dr Moritz Tenorth, head of software development at Magazino, a Munich-based startup developing mobile pick-and-place robots for item-specific logistics. They discussed his work on the Toru robot and what it means to the warehouse industry today and in the future.

Dr Moritz Tenorth

Dr. Moritz Tenorth

After obtaining his PhD in robotics from TU Munich Dr Moritz Tenorth spent several years as a post-doc and freelance robotics consultant, with stays at the CMU Robotics Institute and the ATR labs in Japan. His research focuses on knowledge representation methods that can help autonomous robots make smarter decisions, and on methods for task coordination in robotics.

 Links

• Magazino (website)
• Toru Cube – Pick by robot (Youtube)
• Pick-by-Robot | technical features of TORU Cube (YouTube)
• Magazino TORU Cube – DHL Logistik-Center (YouTube)
• Download mp3 (10.1 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

This year marks the occasion of ROS turning 9 years old! Over the years, ROS has grown into a strong world-wide community. It’s a community with a large variety of interests: from academic researchers to robotic product developers, as well as the many robot users. Academic use of ROS continues to grow. Citations of the first ROS paper “ROS: An Open-Source Robot Operating System” has grown to 2,871.

To get a better sense of what’s happening in the ROS community, if you have not already done so, I highly recommend reviewing the ROSCon 2016 program. You can also find all the video recordings in this gallery. ROSCon 2016 was another great event bringing ROS community members together to share how they’re using ROS to solve their challenges. As the goal of ROSCon is to share information between the entire community, we record the talks and make them available online. We’ve sold out our venues the last two years and are looking forward to another ROSCon next fall!

Part of understanding our growing community is to try to measure it. For the last 6 years we’ve been generating metrics reports. These reports can give a sense of aggregate concerning what’s happening in the ROS community. Our most recent report is from July 2016. David Lu has put together plots of several of the metrics across the last 6 years which can be quite informative.

This year we wanted to dig a little deeper into the code metrics, so we downloaded the source of all of packages listed in the Indigo Igloo rosdistro and ran some analysis.

• The total line count is over 14 million lines of code
• There have been 2477 authors
• And 181509 commits
• Averaging 73.3 commits per author

You can see the commits as a function of month in this graph.

Our committers are active around the world as evidenced by the commits coming in at all hours of the day.

And the git commits record 24 different time zones (out of 39 possible).

Analyzing the repository for significant lines of code using [SLOCCount](http://www.dwheeler.com/sloccount/) shows:

• 4,077,199 significant lines of code.
• This represents an estimated 1,236 person-years of development.
• For a sense of scale, that is an average of 137 developers contributing full time over the last 9 years!

For those of you curious about the breakdown by language lines of code, it is as follows:

• cpp: 2608592 (63.98%)
• python: 553332 (13.57%)
• ansic: 297629 (7.30%)
• xml: 280615 (6.88%)
• lisp: 149439 (3.67%)
• java: 135343 (3.32%)
• ruby: 26484 (0.65%)
• sh: 21120 (0.52%)

This only represents the packages publicly released into the Indigo rosdistro index.

Note that the tools only worked on Git repos, so code from other source control systems was excluded. There are also a few projects which predate ROS but have ported to use ROS and their history is included.

We’re looking forward to continuing growth through 2017 leading up to the 10-year anniversary of ROS. With the Beta 1 version of ROS 2.0 out, there will be space for new development. We’re looking forward to our next release, Lunar Loggerhead, to coincide with Ubuntu’s next release, Zesty Zapus. With both of these, the ROS community can continue to rely on the many libraries, tools, and capabilities they have come to know and enjoy, as well as begin to experiment with the new features in ROS 2.0

Another exciting project to watch is the upcoming TurtleBot 3! The TurtleBot and TurtleBot 2 have been great platforms for learning and prototyping. However by packing that same capability into a smaller platform with more punch, we look forward to it providing another avenue to grow the ROS community.

We write these anniversary posts to help give you a sense of how ROS has been doing over the past year, but we’d certainly encourage you to find out for yourself. Get involved. Write or edit a wiki page. Answer a question on ROS Answers. Come to ROSCon. And, when you’re ready, think about helping to maintain ROS itself, or even contributing a brand new ROS package.

OSRF is doing great, but the long-term success of ROS depends on every member of the incredibly awesome ROS community. If you’re already an active part of the ROS community, we can’t thank you enough; and if you’re not, think about how you can help ROS grow and thrive for the next nine years, and beyond.

If you enjoyed this article, you may also also enjoy:

• Introducing H-ROS: the Hardware Robot Operating System
• #IROS2016 in pictures
• A practical look at latency in robotics: The importance of metrics and operating systems

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this episode, Audrow Nash interviews Dieter Fox, Professor in the Department of Computer Science and Engineering at the University of Washington, about the 100/100 Computer Vision Tracking Challenge. This is a self-imposed challenge to understand 100% of the pixels in an image 100% of the time in video footage; this includes understanding semantic information. Such understanding would allow robots to assist humans more naturally in environments like a home kitchen, wet lab, or in disaster response. To accomplish this challenge, Fox discusses challenges which include modeling, tracking, and detecting articulated objects.

Fox’s Keynote at ICRA 2016 on the 100/100 tracking challenge:

Dieter Fox
Dieter Fox is a German roboticist and a Professor in the Department of Computer Science & Engineering at the University of Washington, Seattle. He is most notable for his contributions to several fields including robotics, artificial intelligence, machine learning, and ubiquitous computing. Together with Wolfram Burgard and Sebastian Thrun he is a co-author of the book Probabilistic Robotics.

Links

• Download mp3 (13.5 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Research and development of robotic assistive technologies has gained tremendous momentum in the last decade due to several factors such as the maturity level reached by several technologies, the advances in robotics and AI and the fact that more than 700 million of persons have some kind of disability or handicap. For many people with mobility impairments, essential and simple tasks, such as dressing or feeding, require the assistance of dedicated people. Thus, the use of devices providing independent mobility can have a large impact on their quality of life.

A team of researchers from the University of Cassino, under the direction of Professor Filippo Arrichiello, is working on the development of a control architecture for assistive robotic systems operated via a Brain-Computer Interface (BCI) to allow a user to operate a robot using his thoughts. The proposed software architecture relies on widely used frameworks to operate BCIs and robots (namely, BCI2000 for the operation of the BCI and ROS for the control of the robot) integrating control, perception and communication modules developed for the application at hand.

In particular, using the BCI2000 framework, they developed a Graphical User Interface (GUI) based on P300 paradigm to allow the user to select options via a BCI Epoch+. The user can select objects from the scene and the action to perform by focusing attention on the corresponding flashing icons on a screen. By counting how many times the icon he wants to select flashes, the user generates a P300 potential that is detected via the BCI and translated in a selection operation on the GUI.

Messages related to the object and actions selected via the GUI are automatically sent to a 7DOF lightweight robot manipulator Jaco2 that has to operate in the environment. The robot control architecture relies on a perception module that allows to detection and localization of objects, and the face/mouth of the user, using a RGB-D sensor (Kinect one). The motion of the manipulator is then controlled relying on a closed loop inverse kinematic algorithm that simultaneously manages multiple setbased and equality-based tasks. Preliminary experiments have been performed where the user commands the robot via the BCI in order to move objects on a table or to select a bottle from where to drink.

Team members: prof. Filippo Arrichiello, Paolo Di Lillo (PhD student), Daniele Di Vito (PhD student), prof. Gianluca Antonelli, prof. Stefano Chiaverini.

Click here for more information on the Robotics Research Group of the DIEI.

If you enjoyed this article, you might also be interested in:

• Virtual race: Competing in Brain Computer Interface at Cybathlon
• Brain Computer Interface used to control the movement and actions of an android robot
• Can a brain-computer interface convert your thoughts to text?

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

But what does it mean to describe systems as complex? How do these complex systems differ from the more easily understood ‘modular’ systems that we are familiar with? Vocabulary in this area is often dangerously inconsistent. For example, the terms ’emergence’, ‘complex’, and ‘complicated’ are used differently by different disciplines, and often differently even within the same discipline. This makes it very difficult to understand whether people are really talking about the same thing, and whether the systems being described are different in superficial or profound ways. On the one hand, failing to identify the underlying similarities between different systems (whether modular or complex) results in missed opportunities for sharing knowledge, best practices and methods. On the other hand, failing to identify the underlying differences between different systems results in practices and methods being misapplied. More broadly, many of today’s real-world problems require engineers, designers and policy-makers across all domains to think in terms of complex systems.
To address problems with translating between disciplines, Chih-Chun Chen and Nathan Crilly at the University of Cambridge have produced ‘A primer on the design and science of complex systems’. This introduces complex systems constructs by building them up from basic concepts, and contrasting them with more familiar constructs that are associated with modularity. For example, ’emergence’ can be understood with respect to a breakdown in how a system’s functions are mapped to the structures that perform those functions. Abstract diagrams that are independent of any particular domain are used to represent the constructs that are discussed. These are illustrated with worked examples that make the explanations accessible for those who have no experience with ‘complexity’. The primer is intended to provide both an introduction to complex systems constructs for those new to the topics discussed, and also a basis for cross-domain translations for researchers and practitioners wishing to engage with other fields when addressing the systems problems they are working on.

Being able to communicate unambiguously across disciplines and application areas would greatly expand the space of solutions available to all domains in solving problems currently deemed to be too ‘complex’. As a mature complex systems engineering discipline which is by its nature interdisciplinary, Swarm Robotics will no doubt have much to contribute to – and take from – this endeavour.

If you liked this article, you may also want to read these other articles on swarm robotics:

• mROBerTO: The modular millirobot for swarm behavior studies
• Swarms of precision agriculture robots could help put food on the table
• Machines can learn by simply observing, without being told what to look for
• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• Scaling up underwater swarmbot research from tabletop ‘aquarium’ to the Venice Lagoon (CoCoRo Video #50/52)

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Frame of the tutoring experience of the student with low motivation and high competence. Reproduced with permission from the parents of the children. Courtesy of Dr Imbernon Cuadrado

By: Ian Salter

The use of robotic tutors in primary school classrooms is one step closer according to research recently published in the open access journal Frontiers in Computational Neuroscience.

Dr Imbernòn Cuadrado and his co-workers at the Department of Artificial Intelligence in Madrid have developed an integrated computational architecture (ARTIE) for use with software applications in schools.

“The main goal of our work was to design a system that can detect the emotional state of primary school children interacting with educational software and make pedagogic interventions with a robot tutor that can ultimately improve the learning experience,” says Luis Imbernòn Cuadrado.

Online educational resources are becoming increasingly common in the classroom, although they have not taken into sufficient account that the learning ability of primary school children is particularly sensitive to their emotional state. This is perhaps where robot tutors can step in to assist teachers.

Rather than focusing on specific emotions, the researchers first identified three cognitive states (concentrating, distracted and inactive) known to influence the course of learning.  Keyboard strokes and mouse actions of children using educational software were used to predict which of these cognitive states the child is experiencing and subsequently linked to an algorithm that chooses the correct form of pedagogic intervention. These could be words and gestures of encouragement or attempts to raise interest and motivation for a specific learning objective, all of which can be delivered by a robot tutor.

In the Frontiers research article, Dr. Imbernón Cuadrado and co-workers describe an example of a robotic tutor platform, MONICA, which integrates the educational software Scratch to a commercially available robot through the computational architecture developed in their paper.

Reproduced with permission from the parents of the children. Courtesy of Dr Imbernon Cuadrado

MONICA was tested out on two primary school volunteers known to have different cognitive learning states. Whilst the children enjoyed having a robot guide their learning, and preferred it to working alone, they both felt they would have learned more with their normal teacher, despite feeling more relaxed in the presence of a robot tutor.

“Our first prototype was designed to demonstrate that the architecture works in detecting simplified emotional states,” says Dr. Imbernón Cuadrado. “The next step will be to implement methods for detecting a more complex range of emotions with cameras and microphones and to test the longer term impact of robot tutors on children’s learning curves.”

It seems however that robot tutors are not ready to replace school teachers just yet.  “We consider that robot tutors could have an effective support role to play in the primary school classroom in helping children reach their specific learning objectives,” concludes Dr. Imbernón Cuadrado.

Click here to read the research paper

If you liked this article, you may also want to read these other articles by Frontiers:

• Living with a prosthesis that learns: A case-study in translational medicine
• Putting humanoid robots in contact with their environment
• Robotic research: Are we applying the scientific method?
• Robots can successfully imitate human motions in the operating room

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this episode, Abate De Mey interviews Jeff Sprenger, founder of the startup Xemory in Vermont, USA. At Xemory they are developing a robot simulation game called Xemo, where players learn to animate virtual robots.

Complex motion at each joint is broken down into its more fundamental components, called degrees of freedom. Players are faced with the challenge of controlling the several degrees of freedom to recreate lifelike motions such as crawling, walking, jumping, and even dancing. These challenges are similar to the ones faced by roboticists trying to develop lifelike, robust and balanced motions for legged robots.

Sprenger discusses the unique ways different age groups and genders interact with the software. Through incorporating feedback from the students, Sprenger adds new activities and challenges to keep students engaged and challenged, improving their understanding of robot control.

Jeff Sprenger
Jeff Sprenger is the founder and CEO of Xemory Software, creator of the Xemo Robot Simulation game. Jeff founded Xemory in 2014 when he received an NSF grant to develop interactive robotics software for STEM education. The company received a second, follow up grant in 2015 to further develop and bring Xemo to market. Jeff has worked in scientific and business software for over 30 years, with eleven years as VP of Technology at MBF Bioscience. Jeff is also the co-organizer for Vermont’s monthly game developer meetup.

Links:

• Collaborator Dr Josh Bongard from The Morpholology, Evolution and Cognition lab advises the Xemo team and heads the MECL robotics lab at the University of Vermont
• Email robots@xemory.com for questions or suggestions about Xemo
• Download mp3 (11.1 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Raheeb Muzaffar, an information technology specialist, has developed an application-layer framework that improves the transmission of videos between moving drones and mobile devices located at ground level. His work within the Interactive and Cognitive Environments (ICE) doctoral programme will be completed soon. Raheeb explains what makes this technology innovative and talks about his plans for the future in a conversation with Romy Mueller.

Raheeb Muzaffar was already aware of what to expect when he arrived in Klagenfurt as a PhD student, travelling from Islamabad, the capital of Pakistan, December 2012: He’s travelled well before, had experience working with various universities and professional organizations, and was acquainted both with the European culture and the research environment. The doctoral programme “Interactive and Cognitive Environments (ICE)” suited him perfectly, allowing him to conduct research in an international environment. Doctoral students accepted into the programme are funded for a period of three years. During this time, they work at two universities. The research teams at the Alpen-Adria-Universität and the Queen Mary University in London offered Muzaffar, who wanted to work on the communication between drones, an environment that allowed him insight into various laboratories and research approaches. By combining the key research areas he experienced in Austria and Great Britain, he ultimately arrived at his very own topic: video transmission in drone networks.

He spotted a problematic issue in this context, which he has studied very closely over the past few years. “In a situation where several drones are airborne and are transmitting videos to several units located at ground level, we cannot currently guarantee that the footage will be reliably transmitted.” The problem is that the receiving devices do not have a feedback mechanism for the video packets being transmitted from the drones. The missing feedback mechanism causes unreliability rendering a distorted video reception. Also, when the transmitting devices and the receiving devices are in motion, the wireless transmission conditions are subject to continuous change, introducing an additional factor of uncertainty.

To address this problem, Raheeb Muzaffar has developed a so-called “video multicast streaming framework”, which allows feedback from multiple ground devices to the drones to add reliability allowing a smooth video reception. However, the framework is even more capable: “The drone does not only learn whether the video has arrived, but thanks to the feedback mechanism, the drone can adjust the transmission rate and the video quality to match the prevailing wireless conditions. Thus, the transmitting drones can re-send lost video packets following the transmission and video rate adaption.” Video communication between drones and receiving devices is useful for various applications such as search and rescue, surveillance, and disaster management.

When we asked him, if he can make profitable use of his technology, Raheeb Muzaffar laughed. At various conferences, he talked to the developers from the industry, who confirmed to him that modules endowed with similar functionality are expensive and that the WLAN protocol requires modification. However, his application is freely available. Anyone wishing to commercialise the technology as a product, is able to do so. In any case, over the course of numerous simulations and experiments, he has demonstrated that the streaming framework works and uses the conventional WLAN protocol without modifications.

Muzaffar’s research in Klagenfurt is coming to an end. By November, he will complete his doctoral thesis, and will pull up stakes in Klagenfurt, following his hopes and dreams to work in the research-based industry. The researcher was accompanied by his wife when he came to Klagenfurt. In the meantime, they celebrated the birth of their daughter. The Muzaffar family intends to travel more: “I am not tied to one location. I would enjoy working anywhere in Europe, but we also like the idea of moving to New Zealand.”

A few words with Raheeb Muzaffar

What career would you have chosen, if you had not become a scientist?

If I would not have become a scientist, I would have become a social worker. Before turning to research, I worked with Non-governmental organizations including iMMAP and UNICEF thinking that my technical expertise could help expedite the social work that the organizations are involved in. Besides the work I do plan to involve myself in some sort of social work may it be technical, educational, financial or emotional.

Do your parents understand what it is you are working on?

Yes, especially my father, he is quite interested in technology. He may not have the technical details but he is aware of research and the work I am involved in.

What is the first thing you do when you arrive at the office in the morning?

First thing in the morning is to check and answer my emails. Thereafter, I work by the plan. I always have a plan for the week/month regarding the tasks I need to do. Although sometimes things don’t go by the plan and I have to readjust the plan to meet deadlines.

Do you have proper holidays? Without thinking about your work?

Yes, at least once a year.

Why are so many people afraid of technical sciences?

In my opinion, every individual has a different aptitude to his/her work domain and one should do what one thinks he/she is interested in and can perform best. Having said that, working in the domain of technical science is quite challenging. The field has advanced and it is getting increasingly complex to even gain the basic understanding of different domains. To work in this field, one has to stay up to date. Developing something new involves a minimal margin of error. I think, people are afraid due to the complex nature of the studies that involve an extra bit of hard work, understanding, and patience.

Written by: Romy Müller, University of Klagenfurt

Originally published in German language in AAU News of the University of Klagenfurt, Austria. Also published at Medium.com.

In this episode, Audrow Nash interviews several researchers presenting their work at the Robotics Science and Systems (RSS) 2016 conference in Ann Arbor, Michigan.

Audrow speaks with Gangyuan Jing from Cornell University about modular robotics; Rico Jonschkowski from Technical University of Berlin about lessons from last year’s Amazon picking challenge, which his team won; Matthew Gombolay from MIT about software to help nurses coordinate; and Dorsa Sadigh from UC Berkeley about developing human-like behavior for autonomous cars.

Links:

• Download mp3 (9.7 MB)
• Link to RSS 2016 homepage
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

In this episode, Audrow Nash interviews Fredrik Gustafsson, Professor in Sensor Informatics at Department of Electrical Engineering in Linköping University, about an initiative to reduce poaching in a rhino sanctuary in Ngulia, Kenya. Gustafsson discusses how he first became involved in this project, how he has worked with the rangers to develop solutions, and the future of this work.

Fredrik Gustafsson

Fredrik Gustafsson is professor in Sensor Informatics at the Department of Electrical Engineering, Linköping University, since 2005. He received the M.Sc. degree in electrical engineering 1988 and the Ph.D. degree in Automatic Control, 1992, both from Linköping University. His research interests are in stochastic signal processing, adaptive filtering and change detection, with applications to communication, vehicular, airborne, and audio systems. His work in the sensor fusion area involves design and implementation of nonlinear filtering algorithms for localization, navigation and tracking of all kind of platforms, including cars, aircraft, spacecraft, UAV’s, surface and underwater vessels, cell phones and film cameras for augmented reality. He is a co-founder of the companies NIRA Dynamics (automotive safety, including tire pressure monitoring systems found in more than 20 million, here), Softube (plug-ins for music studios and software solutions for Marshall and Fender), and Senionlab (leading manufacturer of indoor navigation solutions for sm, here).

Links:

• Download mp3 (16.9 MB)
• Project Ngulia website
• A PDF overview of Project Ngulia
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

In this episode, Audrow Nash interviews Emo Todorov, Director of Movement Control Laboratory at the University of Washington, about a physics-based optimization method for controlling robots. Todorov describes how his physics-based method can be used to solve problems and discusses results in simulation and on hardware.

Emo Todorov

Emo Todorov obtained his PhD in Cognitive Neuroscience from MIT in 1998. Since then he has worked as Postdoctoral Felow in the Gatsby Computational Neuroscience Unit at UCL, Research Scientist in Biomedical Engineering at USC, Assistant Professor in Cognitive Science at UCSD, and is now Associate Professor in Applied Mathematics and Computer Science & Engineering at UW. He is generally interested in intelligent control of complex systems, in both engineering and biology. His current focus is autonomous robot control through high-performance numerical optimization. He is also the founder of Roboti LLC and developer of the MuJoCo physics simulator.

Links:

• Download mp3 (15.8 MB)
• Todorov’s lab website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

This is the second of two episodes where Audrow Nash interviews several companies at the International Conference for Robotics and Automation (ICRA). ICRA is the IEEE Robotics and Automation Society’s biggest conference and one of the leading international forums for robotics researchers to present their work. The 2016 conference was May 16-21 in Stockholm, Sweden.

Interviews include the following:

Greg Burman, Engineer at iniLabs, explains a novel, high frequency camera that senses differences in illumination at each pixel.

Dave Rollinson, Co-Founder of HEBI robotics, talks about on a modular system for constructing robots.

Tony Prescott, Director of Research at Consequential Robotics, discusses work on a companion robot.

Xavier Carpentier, Phoenix Technologies, on motion tracker used on the International Space Station.

Links:

• Download mp3 (17.4 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Code review of a C++ program with an error found.

I have been part of many software teams where we desired to do code reviews. In most of those cases the code reviews did not take place, or were pointless and a waste of time. So the question is: how do you effectively conduct peer reviews in order to improve the quality of your systems?

I found this book, Peer Reviews in Software: A Practical Guide by Karl E. Wiegers. This book was recommended to me, and having “practical guide” in the title caught my attention —  I have reviewed other books that claimed practical, but were not. Hopefully this book will help provide me (and you) with tools for conducting valuable code reviews.

Peer Reviews in Software: A Practical Guide. Photo: Amazon

As a human, I will make mistakes when programming; finding my mistakes is often difficult since I am very close to my work. How many times have you spent hours trying to find a bug in your code only to realize you had a bad semi-colon or parentheses? Another person who has not worked on the code for hours might have been able to spot the problem right away. When I first started programming it could be embarrassing to have somebody review my code and point out the problems. However now that I am more senior, I do not view it as an embarrassment, but as a learning opportunity since everybody has a different set of experiences that influences their code. I would encourage other developers to view it as a learning experience and not be bashful about reviews. Remember the person is critiquing the work, not you; this is how to become a better developer.

According to Wiegers, there are many types of peer reviews including: inspections, team reviews, walkthroughs, pair programming, peer desk check passaround, and finally ad hoc review.

This book is divided into three sections:

1. Cultural & social aspects
2. Different types of reviews (with a strong focus on inspection)
3. How to implement review process within your projects

Cultural & Social Aspects

In this first section of the book, the author makes the argument that quality work is not free and that “paying” the extra cost of peer reviews is a good investment in the future. By having peer review you can reduce failures before the product is released out into the world and any applicable reworks. Shifting the defect detection to the early stages of a product has huge potential payoff, due to high costs of fixing defects found late in the release cycle, or after release. The space shuttle program found the relative cost for fixing a defect is: $1, if found during initial inspection; $13, if found during a system test; and $92, to fix after delivery! In the book, the author documents various companies who saved substantial amounts of time and money all by having code inspection programs.

One thing I like is the reference to IEEE 1999, which talks about other items that are good to review. People don’t think about it but other things, such as marketing brochures, requirement specifications, user guides, test plans and many other things, are good candidates for peer review.

I have seen many project teams try to do error tracking and/or code reviews but fail, due to team culture. I saw one case where peer review actually worked: when a dedicated person’s only job was to manage reliability in the project. He was great at hounding people to track bugs and review code. This book discussed how team culture must be developed to value “quality”. If you are the type of person that does not want to waste time reviewing another’s code, you must remember you will want the other person to “waste” time looking at your code. In this manner, we must all learn to scratch each other’s back. There are also two traps to watch out for:

1. Becoming lazy and submitting bad code for review since somebody else will find/fix it, or
2. Trying to perfect your code before sharing it, in order to protect your ego from getting bruised, and to only show your best work.

We also cannot forget managers. Managers need to value quality and provide time and resources for employees to develop good practices. Managers need to understand the point of these exercises are to find flaws and people should not be punished based on those flaws. I have often seen managers not putting time in the schedule for good code reviews.

Types of reviews

Before discussing the types of reviews there is a good discussion on the guiding principles for reviews. Some of the principles are:

• Check your egos at the door
• Keep the review team small
• Find problems during review, but don’t try to fix them at the review. Give up to 1 minute for discussion of fixes.
• Limit review meeting to 2 hours max
• Require advanced preparation

There are several types of peer reviews discussed in this book. This list starts with the least formal approach and develops until the most formal approach (the book uses the opposite order which I found non intuitive).

1. Ad Hoc – These are the spur of the moment meetings where you call a coworker to your desk to help with a small problem. Usually, this just solves an immediate problem. (This is super useful when trying to work out various coordinate transforms)
2. Peer passaround/deskcheck, – In this approach a copy of the work is sent to multiple reviewers, after which you can then collate all of the reviews. This allows multiple people to look at the code/item and also lets you get something if one person does not respond. In the peer deskcheck version, only one person looks at it instead of passing it around for multiple reviews.
3. Pair Programming – This is the idea that two people program together. So while there is no official review two sets of eyes see each line of code being typed. This has an added bonus that now two people will understand the code. The downside is that often one of the coders can “doze-off” and not be effective at watching for flaws. Also, many coders might not like this.
4. Walkthrough – This is where the author of the code walks through the code to a group of reviewers. This is often unstructured and heavily dependent on how good of a job the author prepared. In my experience this is good for helping people understand the code and finding large logic flaws, but not so much for finding smalls flaws/bugs.
5. Team Review – This is similar to the walkthrough however reviewers are provided with documentation/code in advance to review and their results are collated.
6. Inspection – Finally, we have the most formal approach which the author appears to favor. In this approach the author of the code does not lead the review, rather, a moderator, often with the help of checklists, will lead the meeting and read out the various sections. After the moderator reads a section, the reviewers discuss it. The author of the code can answer questions and learn how to improve various sections. Often the author might identify other instances of the same problem that the reviewers did not point out. An issue log should be maintained as a formal way of providing feedback and a list to verify fixes against.

Suggested review methods from “Peer Reviews in Software”

The book then spends the next few chapters detailing the inspection method of peer review. Here are just a few notes. As always, read the book to find out more.

• In most situation 3-7 is a good size group for the inspection. The number of people can be based on the item being reviewed.
• The review needs to be planned in advance and have time to prepare content to distribute to reviewers.
• After the meeting the author should address each item in the issue log that was created and submit it to the moderator (or other such person) to verify that the solutions are good.
• Perform an inspection when that module is ready to pass to the next development stage. Waiting too long can leave a person with a lot of bad code that is now too hard to fix.
• You can (sort of) measure the ROI by looking at the bugs found and how long they took to find. There are many other metrics detailed in the book.
• Keep spelling and grammar mistakes on a separate paper and not on the main issue list.

How to implement review processes within your projects

Getting a software team and management to change can be difficult. The last part of this book is dedicated to how you can get reviews started, and how to let them naturally grow within the company. One significant thing identified is to have a key senior person act as a coordinator for building a culture of peer review and to provide training to developers. There is a nice long table in the book of the various pitfalls that an organization may encounter and how to deal with them.

This book also discusses special challenges and how it can affect your review process. Some of the situations addressed are:

• Large work products
• Geographic or time separation
• Distributed reviewers
• Asynchronous review
• Generated and non-procedural code
• To many participants
• No qualified reviewers available

At the end of this book, there is a link to supplemental material online. I was excited to see this. However when I went to the site, I saw it was all for sale and not free (most things were around $5). That kind of burst my bubble of excitement for the supplemental material. There is a second website for the book that is referenced but does not seem to be valid anymore.

Throughout the book the idea is getting proper training for people on how to conduct inspection reviews. Towards the end of the book, the idea of hiring the book’s author as a trainer to help with this training is suggested.

Overall, I think this is a good book. It introduces people on how to do software reviews. The use of graphics and tables in the book are pretty good. It is practical and easy to read. I also like how this book addresses management and makes the business case for peer reviews. I give this book 4.5 out of 5 stars. The missing 0.5 stars is due to the supplemental material not being free and for not providing those forms with the book.

Disclaimer: I do not know the book author. I purchased this book myself from Amazon.

In this episode, Audrow Nash interviews Todd Hylton, Senior Vice President at Brain Corporation, about neuromorphic computers. They discuss the robotics development board bStem, which approximates a neuromorphic computer, as well as the eyeRover: a small balancing robot that demonstrates how the bStem can be used in mobile robots.

youtu.be/uGcUB5MAVsY

Todd Hylton 

As Senior Vice President of Brain Corporation, Dr. Todd Hylton leads the development of business and technical strategies within the company. A scientist and co founder of a small semi-conductor equipment manufacturer, Hylton brings 25 years of experience in the semiconductor, optical communications, data storage and defense industries alongside a broad technical entrepreneurial background in research and development, small business, marketing and government programs.

Links:

• Download mp3 (16.4 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Transcript

Audrow Nash: Hi, welcome to robot’s podcast. Can you introduce yourself?

Todd Hylton: I’m the senior VP of Strategy at Brain Corporation. Brain Corporation is a technology startup located in San Diego, California working on computing hardware and software technology for robotics.

Audrow Nash: Can you tell me the goal and motivation behind the company?

Todd Hylton: The company’s mission is to provide computing technology – hardware, software and cloud-based or web-based services – for people who want to build robots, so that robots can become more a part of everyday life than they currently are.

I often ask people, “How many robots did you see today?” And usually the answer is “none” unless they have a [robot] vacuum cleaner at home. Our mission is to make it possible for many more robots to be created by providing some of the foundational technologies that are, frankly, quite challenging to produce if you’re building a robot from scratch.

Audrow Nash: To begin, can you give me an overview of neuromorphic computing and its advantages over using a CPU?

Todd Hylton: This is a topic near and dear to my heart, and certainly part of what Brain Corporation does.

Before working at Brain Corporation, I was at DARPA where I sponsored a project in neuromorphic computing called SyNAPSE. The goal for neuromorphic computing in general was to see if it was possible to build a different kind of computing architecture, one in which computing was massively distributed, and memory was embedded very near the computing elements. To draw inspiration from biology: very simple processing units (which you might think of as neurons) sending messages to other processing units (via things you might think of as spikes) with connections between them, which are the memory around the processors (what you might that of as synapses).

The goal for that project was to see if it was possible to build a large-scale neuromorphic computer and also to push boundaries in memory technology. You need a lot of memory on board to make a good neuromorphic computer, and [we wanted] to push our understanding of what we would be able to teach these chips to do, and how we would teach them by having synaptic modifications and neuronal dynamics built into the chips.

Though Brain Corportation is not building neuromorphic chips, many of the learning algorithms that we write for our robots have ideas like neurons and synapses underneath them, so we would very much like to be able to buy these neuromorphic chips when they become available because we think they will be a great enabler for our robotics packages.

To come back to your original question about CPU versus neuromorphic processor, CPU’s are great for general purpose computing. You can give them gigantic lists of instructions, they’re pretty simple to use, and that’s why we like them. But if you need to do a whole bunch of things in parallel – if you need to coordinate many different things in time and space – they’re not very efficient. It’s just too much energy wasted moving stuff back and forth in the chip.

Audrow Nash: An example of that is vision processing … ?

Todd Hylton: Yes, a good example of that is vision processing, which goes right back to robotics. One of the key things that robots need that they mostly don’t have are vision systems. There are different levels of vision system, and some are quite simple, but we envision at some point in the not too distant future a vision system that actually begins to learn the dynamics and the statistic of its environment in a spontaneous way. And that really needs a different kind of computer to do it well.

Audrow Nash: You guys have created bStem. Can you tell me a bit about it?

Todd Hylton: bStem does not include a neuromorphic processor – those neuromorphic processors are still in their very early phases of being built and the tools needed to make them are not commercially available yet. But bStem is the closest thing we can get right now.

We wanted a processor that was low power, low cost, but still with a lot of computational capacity, because you don’t have a lot of power and you need a lot of computation if you’ve got a mobile robot. bStem’s basic technology is derived from the mobile phone industry. Chips that are available for mobile phones provide lots of different computing capabilities: GPUs, CPUs, DSPs, various codecs for vision – all of these things are part of what we need. So we have taken state-of-the-art cell phone chipsets made by Qualcomm and repurposed them for robotics on this little board we called bStem. It also comes with sensors, drivers for various motors, and breakout boards for various motors that the robots may have.

Audrow Nash: Are there similar development boards out there?

Todd Hylton: I don’t know of any similar development boards that use state-of-the-art mobile phone technology for robotics. But there certainly are development boards that use mobile phone technology. If you want to build a phone or tablet you can get one of these boards. Unfortunately they’re a long way from what a roboticist really wants. With bStem, we closed the gap between those development boards built for phones and development boards built for robots. If get a bStem, it’s going to look and feel just like what you would expect it to be as a roboticist and not as a mobile phone developer.

Audrow Nash: What kind of reactions have you gotten from people who are using bStem?

Todd Hylton: We’ve given bStem to a handful of developers. The basic reaction is, “Wow I can’t believe it. I’ve got a whole Ubuntu package on this tiny little board, it takes about two watts to run it, I can develop my whole robotic controller on this board, I can do Gmail on it if I feel like it, plus it’s got a whole bunch of other tools and goodies that make it easy for me to write the controllers that I want.”

Audrow Nash: Can you tell me a bit about eyeRover?

Todd Hylton: As I said in the beginning we aspire to build the computing technology for people that want to build robots. We don’t plan to build and sell robots, but we built the eyeRover because we had to integrate the whole technology stack: the computer hardware, the software, put it on a robot, put the sensors on it and we wanted to be able to train the robots so it’s got a learning system that we put on it as well. It basically forced us to build the whole technology piece so that we could begin to show our potential partners what it is we can do.

Audrow Nash: Can you describe some of the sensors and actuators that eyeRover has?

Todd Hylton: eyeRover is a two-wheel balancing robot, sort of a Segway-type robot. It’s got two cameras and a relatively simple vision system with which it can see its environment. It has a bStem board, onboard IMUs, and magnetometers (which are used in some of the algorithms).

There are a lot of things on bStem that it doesn’t use because we didn’t need them for this particular robot, but other roboticists would probably like them. bStem also includes a user interface, which you can use to both remote control the robot and to train the robot in a behavior. That remote controller looks like a game pad/iPad combination. It’s sort of like a video game controller for the robot.

Audrow Nash: It looks to me like a PSP, the older kind.

Todd Hylton: Yeah, yeah.

Audrow Nash: Can you talk about eyeRover’s modes of learning and how you’ve been able to train him?

Todd Hylton: All of the learning modes on eyeRover are what we call ‘supervised learning’. Supervised learning is when you essentially show the robot what it is supposed to do; it learns to associate its sensory input with whatever it is you told it to do. For example, if the robot is supposed to turn right whenever it sees a green block on the left, I just show it that behavior a few times and it makes that association. It’s a well-known way of doing training.

We’ve built many different supervised learning algorithms, and the ones that we do on the eyeRover we’re doing primarily because we wanted to show navigation behaviors; we basically built our own special supervised algorithm that we thought was particularly effective and also didn’t require gigantic computational resources.

In the future, as we build out the learning piece – which we call the Brain Operating System, that’s the software stack that allows the robot to learn – it will also have what people generally think of as ‘unsupervised learning,’ where by exploring its environment the robot begins to learn the structure of its world. It will likely have some reinforcement learning as well, which is what I kind of think of as an impoverished supervised learning where instead of telling the robot exactly what it’s supposed to do, you just tell it “that was good” or “that was bad.”

Audrow Nash: Can you talk about some of your examples of actually training eyeRover?

Todd Hylton: We’ve trained it to do simple navigation around objects that may be on the floor, like around your desk, around your chair, around your trash can, do some loops and figure eights, go from point A to Point B. The way we train it is to show this behavior by remote control a few times, we show it a few loops around the path it’s supposed to do, and then we let it go. And if it’s correctly learned it, it does the same behavior. You can also train it incrementally, where you train it, you let it go, and you see if it’s doing what you want and you just give small corrections when it makes mistakes. Over time it does better at those areas where it was making mistakes. For the kinds of things that we’re doing now, it usually only takes a couple of minutes to train a path like the ones I just described.

We’ve also given it some simple gestures and it can learn to come to you or go away from you. On other robots we’ve trained them to play fetch and we can show them an object that they’re supposed to find: put the object out somewhere, it’ll go get the object and bring it back to a base, and it’s all trained.

The cool thing about that is it’s hard to write a piece of code that would just do that generically. You can’t foresee every environment even if they’re pretty simple, and you can’t really code it. So one of the key things we think robots need to be able to do to be part of everyday life is that they need to be able to adapt or be trained in the environments in which they’re going to perform because no coder can anticipate every single situation a robot’s going to see.

Audrow Nash: Will this bring the development of robots and their behaviors to the masses … because you can train the robot to behave as you want?

Todd Hylton: Exactly.

Audrow Nash: Is that the goal of Brain Corp?

Todd Hylton: That’s one of the goals, and I think it’ll go in stages. There will be some pre-training by the people who build the robots so that the thing doesn’t have to learn from scratch when you buy it from Best Buy and stick in your home. That way, when you get it in the home, it will be much more capable and much more useful, [especially] if you can give it additional training.

One of the challenges for us, though, is to come up with user interfaces so that people who aren’t roboticists or technologists can train a robot. There’s a big stack of things that we worry about, such as what can people actually do, how can we actually control the robot, how can we make it clear, how do you communicate the task to the robot … the AI pieces that try to put together what the robot senses and what it’s being told to do … all the computing hardware and then a whole cloud infrastructure so that you can get updates, you can upload brains, you can upload what the robot’s done so we can diagnose it if something bad has happened and it’s not performing the way it should be. We are doing all of those things.

Audrow Nash: Now that you’re using bStem, what are some of the other applications that you anticipate seeing from the user community?

Todd Hylton: That’s what we’re just now beginning to learn in detail. We were in hibernation for many years getting the technology together, and it’s only in the last month or so we’ve actually been telling people. We didn’t really want to start saying much until we had a platform on which we can demonstrate it, which is why the eyeRover exists. 

What I see in the robotics industry in general, like at this show – we’re at RoboBusiness now –it’s mostly very small companies in narrow niches. [Of course this is a] silly exaggeration, but it seems that most of the companies bring in gears and motors and sheet metal and plastic and a few chips and a loading dock, and a robot comes out the other end. Basically, they have to do the whole technology piece, which is really challenging.

What we’re doing took a gigantic amount of resources to actually accomplish. Our business proposition is that it doesn’t make sense for every company to do that; we’ll do it for you. Pretty much all the robots that I’ve seen at this meeting are simple robots that roll around on four wheels and see stuff, there are some that have grippers, there are some humanoid robots or some combinations of those things. I think all of those robots could benefit from the technologies that we’re developing.

Another thing that we would like to see happen in the robotics industry is for it to be much easier for new companies to form around building robots. There’s a huge economic barrier to get it going because the computing systems that you need mostly don’t exist and you have to build them from scratch. The people that really know robotics or who have some deep domain expertise in whatever the problem is they’re trying to solve – whether it’s drilling holes or sweeping floors or cleaning windows or whatever. There’s no reason they should have to know the entire software stack on a modern system on a chip – it’s crazy.

If we can get these technology pieces together and available – and that’s our plan of course – then I think there will be an explosion of new robotics companies. Two guys in a garage out of engineering school can build a robot, rapidly prototype it, show that it works, take it to an investor and validate that they can do what they say they want to do with very little investment. That will enable an explosion of robotic applications and different niches, and that’s the world as we see it in the future. Sort of what we have now, except it would be vastly larger.

And of course there will some big consumer plays where everybody’s got the robot that cleans the table, or whatever the killer application is, and we’re still talking to people about that.

Audrow Nash: In developing this technology, something like neuromorphic computing, what are some of the major challenges you’ve encountered and what is the lesson to learn from them?

Todd Hylton: The major challenges now and for the foreseeable future will be getting sufficient computational capacity on board small mobile robots. One of the ways we are working to mitigate is to try to shift part of the computation off into the cloud. That turns out to be a very complicated problem, too. It’s not as simple as saying, “There’s big computers in the cloud and we’ll just do it all there,” because there are all sorts of issues about moving data around and latency that it may make sense for some applications but not for others.

From my perspective, the biggest technical challenge is the computing capacity. If the neuromorphic computing technology matures rapidly, it would alleviate a great fraction of it. Most of the computation that we do now, even on eyeRover, is vision – vision algorithms of kind or another – and that’s a real sweet spot for neuromorphic computers.

I think the other challenge for us is that the industry structure needs to change, and I’m not sure how long that will take. As a business, you’ve got only so much capital to work with, and ou’ve got to become self-sustaining and profitable before you run out of money. That depends on the ability of the industry to adopt what we’re doing even if they maybe don’t think about it the way we do yet. So that’s a certainly a risk for us.

In terms of lessons learned, I would say it’s really challenging to take a state-of-the-art system on a chip built for a mobile phone and make it useful on a robotics platform. When we first started, we thought it was going to be a whole lot easier than it was. A lot of people talk about how mobile phone technology is going to enable robotics, and it will, but actually getting there from where we are now was a huge investment.

An additional lesson that we learned is in doing the learning algorithms; that there’s not one way to do them, there are many ways. There is no general-purpose learning algorithm out there – at least not yet. So you have to do some experimentation to figure out what the best ones are. I like to tell people that the work is the process of eliminating all the bad complicated ideas in favor of the simple ones that work. Sometimes, though, when you find a simple solution you kick yourself and wonder, “Why didn’t I think of that a long time ago?” but that’s just the way it is. The goal is to find a simple solution, and that takes a lot of work. Brain Corporation’s got some very capable people working for it, but we’re just a small group of people. There are all kinds of people who can write these kinds of algorithms, and there’s no reason we shouldn’t make it possible for them to do that.

So part of our strategy going forward, hopefully sometime next year, is to make our Brain OS system available with APIs so that if you are into AI, or a neural nets, or machine learning, you could start writing your own learning algorithms. You already have the whole robotic platform, all the infrastructure and so forth, and you can focus on that piece, because that’s a lot of work too. If I can multiplex that out into the world then I think there will be a much faster spread of the technology, and I think it’s a much better business strategy for us as well.

I think the other lesson I’ve learned is that it’s very challenging to try to build a new technology and build a new business model and into a new market at the same time. It forces us to make some educated guesses when it’s still quite unknown what the result will be. It takes a real leap of faith to say, “We’re going to build an eyeRover,” or “We’re going to spend a year building this little plastic robot that runs around so you can train it,” because if I don’t have it, I can’t show anybody what I’ve got. And then you say, “What if the eyeRover is the wrong sort of thing? What if nobody likes it?” or “What’s the application?” and so forth. You have a constant chicken and egg problem, but that’s also what makes it so exciting.

I have all these great people that want to work the hard technology problems, the business problems, and fortunately there’s an appetite for it now, so investors are interested and that enables us to do it.

Audrow Nash: For those that have less expertise, what would be some ways that they can get involved, learn more and eventually contribute to this technology?

Todd Hylton: Currently the eyeRover is in a closed beta program – we’ve just got a handful of people using it so we can get feedback on what we’ve got and shake out the bugs that we’ve missed. But sometime next year it’ll be possible to buy that computing system from us, the boards and the software and so forth. The eyeRover is mostly 3D printed, so we’ll just make those files and the parts list available, and you can buy the board and build your own. We’ll probably put a couple more robotic designs that are 3D printed like that. If people are enthusiastic about it and have access to some resources like a 3D printer (and of course you order 3D printed parts online), then that’s one easy way to get going. You still need some degree of expertise in computers and to use the board and change things on it. But it’s a Linux operating system; you basically turn it on, you plug the robot into your monitor and keyboard, and the screens come up. It looks just like your desktop, so it’s not nearly as intimidating as it used to be.

Audrow Nash: Fpr Brain Corp, what is your future direction and some of your future goals?

Todd Hylton: It’s pretty clear on the technical side what we’re doing for the next year, and it’s pretty ambitious. We’ve got lots of developments going on and new hardware, new learning algorithms, APIs for developers. On the business side we are just breaking out box, so we are talking to a lot of people, and we need to refine our business model. We were a technology vendor essentially, but there are different ways to do that. How do you capture it in such a way that you provide the best value to the customer, but you also make money on it? In order to do what we really want, we need to be someday a really big tech company. So we’ve got to figure out a business model that makes it possible for that to happen.

Audrow Nash: Wrapping up, what do you think is the future of robotics?

Todd Hylton: Robotics is always the technology that’s just around the corner and never quite gets here. That says to me that it’s hard. You can ask yourself whether now is the time or not, but there are certain things that suggest that maybe now is the time, and of course if I didn’t believe it I wouldn’t be doing what I’m doing. On the one hand there is the explosion in low cost computing hardware – it’s gotten really cheap thanks to Moore’s Law. And that’s going to continue for a while so we’re going to get better and cheaper computers, and maybe the neuromorphic stuff will come along, too, and that will be a big tech enabler.

There has been a huge amount of work in neural nets, AI, and machine learning over the past three decades. It really hasn’t had much of an impact on robotics yet, but a lot of people know how to do that now. There are a lot of tools out there, and people are trained in it so you can hire them. That’s a definite tech enabler for robotics, too, that in the past hasn’t been as prominent as it is now.

On the economics side are the big tech companies, the Intels and Qualcomms and Googles of the world. They’re looking for the next big thing and they can’t avoid robotics. Everybody says we’ve got to have robots to take care of old people, and I will be one soon enough. And it’s true. We really do need them. There is going to be a huge demand if we can get the technology and the industry structured appropriately in time. That’s why I’m bullish on it and I think it is a good time for robotics. Another important thing is that we’ve done a lot of work studying the brain lately, and although we are never going to put a mouse brain on a robot, knowing something about how brains work is really important for giving us inspiration for how to do it in a different kind of technology, like a computing technology. So that’s also why I think it’s a great time.

Audrow Nash: Thank you.

Todd Hylton: Thank you.

All audio interviews are transcribed and edited for clarity with great care, however, we cannot assume responsibility for their accuracy.

I have often thought about what the proper software methodology should be for the various robots that I build. My thoughts have evolved over time as I have seen these tool work. While I do not have any formal software engineering training, these are the common principles that I have seen, heard, read, etc. that I believe in (at 2am while I write this).

Disclaimer: Most of the suggestions here have come from other sources. Please check out the links attached for more information.

I will start with how to write code, since that is the basis of everything else. After that we will move into the software engineering infrastructure needed to make the software work properly.

General Programming Rules

Without having basic rules that you follow when coding, your code is doomed to fail and be unreliable. Everything else is built upon this. Many people start programming and want to use crazy template magic where everything inherits something else, however you must STOP and write simple code that is easy to understand and test.

• Keep control structures simple – Limit your branching, don’t have long nested if, case, and loop statements, no recursive magic.
• Endless loops – Do not use boundless loops unless you really want to be stuck there forever (such as the main loop). Everything else must have an upper/lower bound for exiting the loop. Think more about for loops than while loops.
• No dynamic memory – Forget that malloc exists. It can cause programs to act differently at each run time, and can also add memory leaks that can be hard to detect.
• asserts – Asserts can help you to enforce that the proper thing is helping in each function. I generally do not like the default assert functions that simply cause an exit; sometimes that is appropriate, but in many cases you should try recovering from the error instead of just exiting. You need to evaluate what is the safer approach in your application.
• Keep scope – Try to assign and initialize variables, structures, etc. in as small a scope as possible. This includes limiting your usage of global variables.
• Pointers – Limit your use of pointers when this is easily possible. You should really avoid multiple levels of pointers. They can be very confusing to follow and lead to all sorts of problems if you reference memory addresses that you should not be touching.
• Check return value from all (non void) functions – If a function can (and should be able to) return an error value, make sure to catch it and act on it. It can also be good to check the bounds of a value returned instead of blindly trusting it.
• Turn on all compiler warnings – So you catch things early. This is especially important when formatting text, working with memory, and when bit banging.
• Use parentheses to control order of operations – I have seen many people assume or forget about the proper order of operations. Using parentheses can clarify and enforce the desired result.
• Hard coded number – Please use variables for any value that might need to be changed by a programmer. If you have an even slightly non-obvious number, include a comment about the source of that number and how to derive it.
• Preprocesor – Limit the preprocessor to “simple” things, such as #define and #if. Using complex macros can lead to hard-to-understand code, poor portability, and a increased probability of bugs.
• Standards – Keep your code consistent. This includes naming, white space, and where your curly braces go. This makes it easier to follow the code. If you are editing existing code, use the format of the existing code.
• Sleep with noop’s or empty for loops – These should be avoided since they will run differently on different systems. This is primarily for embedded systems.
• Document your code – Need I say more?

Helpful links:
Rules for defensive C programming
The Power of Ten – Rules for Developing Safety Critical Code

Software Repositories

Having a code repo should be a given for every programmer and programming team. There are many free tools available, so there is no excuse to not use them.

Having a software repository lets you save your code at various points so that you can roll back to prior working versions. Repositories also let you log what was changed in each file and by whom. These tools are a great way to share code with a team – there is no need to email code back and forth. You can also put documentation in the repo so that is is always available.

There are many options to choose, from cvs, svn, mercurial, git, bitkeeper, etc. Just pick one!

Lately I have been using a hosted repo service called github.com – I like all of the tools that are packaged with git on the website. bitbucket.org is another similar option.

I generally like to create repositories with the following directories in the root directory:

• bin – A place to put all of the binaries after they are compiled. By default this directory should have nothing checked into the repository.
• config – If I have lots of config files I will put them all in this directory.
• doc – Documentation for the system (datasheets, diagrams, notes, etc..)
• externals – Source code and libraries from outside parties (ie. you did not write them)
• includes – Header files that other multiple programs want access to at compile time
• libs – A place to put all of the libraries after they are compiled. By default this directory should have nothing checked into the repository.
• src – This is where all of the code we write should live.
• tools – This is for random “tools” that do not fit anywhere else.

In addition to the directories above, I will usually put the main make file for the project and a README in the root directory.

Links:
Software repositories or just plain repo

Bug/Issue tracking

Having a central database to track bugs is very important for developing great code that runs bug free. This lets anyone who identifies a bug to report it in such a way that it does not get lost. Just telling the developer or emailing them is not sufficient. People forget things easily. Ideally, you do not want the developer to close a bug; you want somebody to report the bug, the developer to fix it, and then the original person who identified the bug verify that it was properly fixed.

A bug report needs three things:

1. What you expected to happen
2. What did happen
3. Steps to reproduce observed behavior (ie bug). Try to minimize the number of steps needed to reproduce error. Include versions of code that you are using.

Links:
Painless Bug Tracking

Automated Testing

Each of your processes/functions should have a set of tests (often called “unit tests”) performed on them to verify functionality. These tests make sure that functions work with various input extremes and combinations. Ideally you should have a test for each possible control path within a function.

For example if you are testing a “sum(number1, number2)” function, you can have the following test file:

int main () {
assert( 2 = sum(1,1));
assert( 0 = sum(-1,1));
assert( NULL = sum(1,NULL)); // or whatever you want it to be doing…
assert( 0 = sum(0,0));
assert( 1999998 = sum(999999,999999));
assert( 5 = sum(4.2,1)); // assuming integers and that floats are rounded.

RETURN (TRUE);
}

In this example assert will print an error message about where it failed and exit. (This is just my quick example, and can definitely be expanded and improved. It can also be changed to fail at compile time so that you do not need to “run” it.)

Automatic (Daily) build system

Having a build server that builds every time somebody commits code, or at minimum, every night, is valuable for identifying inadvertent bugs introduced into your system and automatically running your tests.

Often you will make a change to one module that will cause another module to fail. But as developers we get focused on our modules and tend to not try recompiling all of the code to make sure that we did not break anything. It is always better to find out you broke something earlier than later.

Certain things tend to cause other modules to break more than others. For example making a change to a widely used header file or message definition file can affect other items.

This also makes programmers only commit code that compiles to the repo and not broken code. If they commit broken code the build tools can/should send out emails to notify people that the build failed (and whose fault it was).

You can either just make a quick script to check out the repo, build it, parse the output and email the results; or you can get a proper build system such as Jenkins. Generally using somebody else’s product will be more reliable and a better use of your time.

Links:
Daily Builds Are Your Friend

Code Reviews

Code reviews are tough, but they are also one of the best ways to reduce the number of bugs in a piece of software. The traditional code review – where a person gets up and talks others in the room through the code (that the people have never seen before) – DOES NOT WORK! This might be good for getting people familiar with the program’s operation, but it is not the best way to do a code review and find problems.

To do a code review you should have the programmer send a review request to 1-3 other software engineers. The review request should have an overview of the code, a flowchart, and directions to access the code. The number of engineers and the experience level of those engineers should be based on how critical the code it. Each reviewer should:

1. Review the code (obviously) – Don’t dwell on style, you have better things to do with your time.
2. Develop a list of questions about the code
3. Write some tests and make sure they pass (these can be committed to always run in the nightly build).

Some of things reviewers should watch out for are when reviewing code before a large review:

• Does the code do what it is supposed to do (and not do what it is not supposed to do)?
• Should the code/function be written in a more efficient manner?
• Is the control flow simple/clean/good (if, else, return, etc..)?
• Do switch statements have a default catch all?
• Is the code commented adequately?
• Are variables declared at the proper scope (minimizing globals)?
• Are there memory leaks?
• What happens if things return NULL?
• Error/Exception handling
• Buffer overflows (doing raw things with char’s, int’s, etc.. memory locations)
• Pointer and pointer based math
• Array indexing
• Closing all handles (files, ports, etc..)
• Macro usage
• 64bit vs 32 bit modifications
• Multithreading (is it needed?, number of threads, race points, deadlocks, data passing, priority inversion, etc.)
• Operators (precedence and proper usage [ & vs &&, = vs == ])
• If input items from users and functions is checked before being used

(Did you notice how this list is similar to the good programming practices above?)

After doing the above steps it is now OK to hold the big meeting where people sit down and walk through and review the code; although this is not strictly necessary.

Reviewing code can be hard. Getting your programmers used to reading code and reviewing it will help strengthen their own coding. It can also be useful to have dedicated QA people who review code.

Links:
Embedded System Code Review Checklist
Microsoft: Best Practices: Code Reviews

Build System

You should be using a build tool so that you are not manually compiling each file. Probably the most common one is make.

Each make file should have at minimum the following directives:

• all – Build all of the production code
• clean – Deletes the binary and object files. This lets the next build be completely clean.
• install – Installs the binaries and libraries into the proper location. This can be a process such as:
  1. Copy old binary to <binaryName>_<archivedDateTime>
  2. Copy new binary to location with active processes
• test – Builds a set of unit/system tests that verify operation and make sure bugs were not introduced.

I believe you should be able to build, install, test, or clean your entire system with one command. As such I will normally create a make file for each process, and then create one master make file that calls all of the individual make files. Having a single make file that can build everything is good for installing a new system and makes setting up the nightly build easier.

Summary

• Write good code.
• Use a repository service (assuming it is not super sensitive data) such as github or bitbucket. That gives you the repo, the issue tracking, and the ability to easily do code reviews.
• Setup a server that builds your repository and runs your tests, at least once a day.

Do you have other favorite tools to make your software more reliable? Please leave comments below.

Main image based on sample code from wikipedia and hammer from pixabay.com/.

In this episode, Audrow Nash interviews Christina Brester, from the Siberian State Aerospace University, about her research on a method to identify emotional state from speech. This method performs speech analysis with a self-adaptive, multi-objective, genetic algorithm for feature selection and uses a neural network to classify those features. In this interview, we’ll discuss exactly what that means, as well as the implications and future of this research.

Christina Brester

Christina Brester completed her bachelor’s (2012) and master’s degree (2014) at the Siberian State Aerospace University (SibSAU) in Krasnoyarsk, Russia. Her master’s thesis was on Speech-based Emotion Recognition.

Currently, Brester is a PhD student in Computer Science and Engineering at SibSAU. Her research interests include evolutionary computation, neuro-evolutionary algorithms, machine learning, and speech analysis.

Links:

• Download mp3 (10.5 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Link to One of Brester’s Related Publications

UPDATE 04/03/2014:

In this video update, we show that a quadrocopter can be safely piloted by hand after a motor fails, without the aid of a motion capture system. This follows our previous video, where we demonstrated how a complete propeller failure can be automatically detected, and that a quadrocopter can still maintain stable flight despite the complete loss of a propeller. 

In the earlier video, we relied on an external motion capture system to measure the quadrocopter’s position and orientation.  By moving more of the algorithm onto the vehicle, the quadrocopter can now be piloted by hand after the failure. The algorithm is executed on the quadrocopter’s onboard micro-controller, and the only sensors required are the quadrocopter’s angular rate gyroscopes. We use blinking LEDs, mounted on the quadrocopter’s arms, to indicate a virtual yaw angle, so that the pilot can control the vehicle with the same remote control commands after the failure. As an alternative to the LED system, an onboard magnetometer could be used to track the vehicle’s yaw angle. Alternatively, by using more sophisticated algorithms, the system could be made to work using only the rate gyroscopes.

ORIGINAL STORY 02/12/2013

The video in this article shows an automatic failsafe algorithm that allows a quadrocopter to gracefully cope with the loss of a propeller. The propeller was mounted without a nut, and thus eventually vibrates itself loose. The failure is detected automatically by the system, after which the vehicle recovers and returns to its original position. The vehicle finally executes a controlled, soft landing, on a user’s command.

The failsafe controller uses only hardware that is readily available on a standard quadrocopter, and could thus be implemented as an algorithmic-only upgrade to existing systems. Until now, the only way a multicopter could survive the loss of a propeller (or motor), is by having redundancy (e.g. hexacopters, octocopters). However, this redundancy comes at the cost of additional structural weight, reducing the vehicle’s useful payload. Using this technology, (more efficient) quadrocopters can be used in safety critical applications, because they still have the ability to gracefully recover from a motor/propeller failure.

The key functionality of the failsafe controller is a novel algorithm that I developed as part of my doctoral research at the Institute for Dynamic Systems and Control at ETH Zurich. This new approach allows such a vehicle to remain in flight despite the loss of one, two, or even three propellers. Having lost one (or more) propellers, the vehicle enters a continuous rotation — we then control the direction of this axis of rotation, and the total thrust that the vehicle produces, allowing us to control the vehicle’s acceleration and thus position.

Even if the vehicle can no longer produce sufficient thrust to support its own weight, this technology would still be useful: one could, for example, try to minimize the multicopter’s velocity when it hits the ground, or steer the multicopter away from dangerous situations such as water, or people on the ground.

This control approach can also be applied to design novel flying vehicles — we will be releasing some related results soon.

This technology is patent pending.

For more information, have a look at the Flying Machine Arena website, the IDSC research page, or just post your question in the comments below.

If you liked this article, you may also be interested in:

• Three new quadrotor videos demonstrate agile control and the power of machine learning
• Live at TED Global 2013: Tweets, photos and video for quadrotor fans
• Behind the scenes at TED Global: Raffaello D’Andrea and team demo amazing quadrotor “athletes”
• Quadrocopters learn from prior experience to improve slalom flying

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

No, this is not about shapeshifting robots, come to save or destroy Earth. It is about transforming the contexts within which robotic technologies are applied, and about practicing robotics with the intention of bringing about transformational results. In some cases this means finding better ways of accomplishing the same ends as before. In other cases it means pursuing ends that were previously unachievable. It hinges on the recognition that robotics is a revolutionary development, on the order of fire or writing, with the potential to transform everything it touches.

We can already see some early examples of how transformational robotics can be. In medicine, automated lab equipment long ago made the comprehensive metabolic panel screening test affordable, and surgical assistant robots have shown considerable promise. In aviation, modern airliners are capable of flying themselves from takeoff to touchdown, and remotely piloted or autonomous aircraft remove biological constraints on maneuvers. In space, were it not for robotics, most of what we know about the solar system, including our nearest planetary neighbors, Venus and Mars, would be limited to what we could learn through the use of telescopes, and there would be no serious talk of mining asteroids.

Before jumping in to how this applies to agriculture, allow me to make one additional point about transformational robotics in general, which is that you can ordinarily expect a transformational application to make more intensive use of technology than an application that simply automates some familiar process or procedure – being mechatronically more complex, or expending more cpu cycles per unit of physical energy used, or both. This is because, in transformational applications, the machines will typically be taking more into account and choosing from a larger, more flexible repertoire of deft actions (precise in context, although not necessarily especially precise in the sense of repeatability).

The key to both understanding and making a case for transformational robotics (in pursuit of private investment or government funding) is to focus on the goals to be served, rather than on current approaches to achieving them. (‘People have to eat, but we also need to preserve and improve fertile land and provide space and habitat for native species.’ versus ‘How can farming as we know it be made more efficient?’) Those current approaches may be ruts, out of which the transformational application of robotic technologies can lift us. In this regard, it is important to include goals we wish to serve which are currently being ignored or given short shrift, whether because no business case can be made for them at present, or because the cost would be prohibitive. Much of the added value of transformational robotics stems from serving ends to which we have previously given little more than lip-service, often for essentially zero additional cost.

Now let’s consider the application of robotics to agriculture, beginning with the current state of the art. The most common use of automation in agribusiness has traditionally been in processing plants, after the food has left the farm, although more and more such operations are being installed on the farm. For plant products and eggs this begins with cleaning and sorting, and any sort of product may be automatically packaged or bottled and packed for shipment.

On dairy farms, robotic milking machines are becoming increasingly common, essentially allowing the cows to milk themselves. In the fields, new tractors now commonly come with GPS-based automated steering, allowing them to make more efficient use of the fuel they burn by reducing overlap from one pass to the next and to preserve soil conservation measures, such as terraces, by following contours more accurately. Combine harvesters may have both automated steering and automated ground speed, to keep them working near their peak capacity without pushing material through so quickly that they become clogged. Spraying equipment, whether tractor-mounted or self-propelled, can now meter out fertilizer, pesticides, herbicides, and fungicides, applying more on parts of the field where the need is greater. A recent development is planters that place seeds more deliberately than in the past, achieving a more even stand using less seed.

While perhaps not yet commercially available, automated pruners for vineyards and automated fruit harvesters are in an advanced stage of development, and should soon be commonplace. And last, but far from least, UAVs, commonly known as drones, can make an abundance of valuable information available to farmers, and the business of providing such services is set to take off like a rocket just as soon as the FAA determines that the time is right.

Next, it’s time to engage our imaginations, as we consider other ways in which robotics might be applied to agriculture, beginning with the main obstacles. One huge challenge, demanding a great deal of further research, is empowering machines to make sense of complex biological environments. As stated in a recent news item from Washington State University:

When apples are in clusters or obscured by leaves and branches, a robot requires complex algorithms and long computational time to identify them.

If this is true of fruit hanging from trees in orchards, more or less in plain sight, it must be even more true of ground-hugging herbaceous plants, the main stems of which may be hidden in deep shade, and still more true when those herbaceous plants occur in polyculture mixtures.

Another such challenge is enabling machines to nimbly reach through foliage without damaging it. For any device that isn’t itself ground-hugging, this is essential for weeding and for planting a second or third crop under an existing canopy of leaves. While much of this work might be accomplished by small devices, moving across the soil surface, there are tasks for which (pruning) and circumstances in which (soggy soil) devices suspended from above the leaf canopy would be more effective. The word ‘nimbly’ suggests both speed and a light, deft touch (sensory-driven precision), and both are intended here. Machines of a scale to support one or more suspended robotic arms, even very light arms, while less massive than most conventional equipment, would still represent a significant investment, and would need to operate quickly to be cost-effective. Quick motion means even quicker electronics to plot paths, gauge the forces needed to produce them, updating these with corrections as deviations are detected, and compensating for environmental factors such as wind-load, slope, the weight of a fruit being picked, and so forth. Quick motion also means being quite certain about the nature of the objects in one’s immediate vicinity, since they might include livestock, cherished pets, or human beings.

Given solutions to these primary challenges – and autonomous operation – in addition to planting, weeding, and harvesting, robotic machines might track the development of individual plants; detect nutrient deficiencies, toxic conditions, disease, and infestation; selectively prune affected leaves or branches, or selectively remove whole plants; collect plant material for shredding, anaerobic digestion, and/or composting; lay out soil sensors or tubing for drip irrigation; attach vines to elevated supports; employ mechanical methods of pest control and deploy biological pest controls. Beyond this they might also engage in crop development, by collecting pollen from certain plants and applying it to others, and through the collection of seed from plants with preferred characteristics, selecting from those based upon nondestructive testing of individual seeds. They could even monitor and to some extent manage populations of wild animals, such as mice, rabbits, and deer. In principle, there is no farming task which machines could not be made capable of performing.

Patch together enough such capabilities, at reasonable cost, and sooner or later you arrive at a tipping point, beyond which it makes far more sense to think about what you would like to see done, and how robots might be made capable of accomplishing it, than to take the more familiar approach of automating conventional methods. On the other side of this tipping point, agriculture may look very different from how it appears today. If the machines operate autonomously, then one farmer can conceivably have many of them all working at the same time, even working around the clock through the peak workload of the season. If there are many of them, working long hours, they can be smaller, lighter, and slower than conventional equipment, and, partly due to economies of scale in their manufacture, far less costly. If they are smaller, slower, and more numerous, they can take a more detailed approach to manipulating their environments. Given sufficiently detailed manipulations, it becomes possible to adapt the best practices of gardening, applied on a grand scale. Also, given this data/detail-intensive approach, production can be closely linked to changing market conditions, helping to keep the operation profitable.

Besides the needed research mentioned above, the ground floor opportunities in this scenario run the gamut of robotic technologies, from making equipment safely mobile through densely planted fields to sensory hardware, to processors, coprocessors, and algorithms for sorting out the sensory data, to end effector design and capability modeling, to force-control motion planning, actuation, result monitoring, and heuristics, and on and on.

The singular reality that makes such a grand effort more than a fool’s errand is the scale of agriculture, and its potential to jumpstart a larger, more robust robotics industry, underwriting investments for developments that will enable further transformational robotic applications in other contexts.

If you liked this article, you may also be interested in:

• Harvey: A working robot for container crops
• Robots Podcast: FutureDairy
• Air, water, energy and food in a nutshell: Space exploration as driver for sustainable robotic agriculture
• Robotics and the disruptive transformation of agriculture

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

This is the third tutorial in the Up and flying with the AR.Drone and ROS series.

In this tutorial we will:

1. Learn about the AR.Drone’s state feedback (and how it is handled by ROS)
2. Learn about the AR.Drone’s tag detection
3. Program our first ROS nodes: A subscriber and a publisher

In the previous tutorials we:

1. Installed ROS, the AR.Drone driver and AR.Drone keyboard controller then flew the AR.Drone using the provided keyboard controller (link)
2. Learned about ROS communication, setup and then flew the AR.Drone using a joystick (link)

In the next tutorials, we will:

• Write a controller to enable tag following
• Write a controller for the drone which allows us to control drone velocity, rather than body angle

What you’ll need for the tutorial

1. An AR.Drone!
2. A computer or laptop with WiFi
3. Linux, either the ARDroneUbuntu virtual machine, or a custom installation
4. Either the keyboard or joystick controller from the previous tutorials

I also assume that you’ve completed either the keyboard or joystick tutorial, which detail how to setup the software, connect to the drone, etc, and provide a method of controlling the AR.Drone from your computer.

Updating

If you have not completed the joystick tutorial:

Please go back and update your installation according to the instructions listed there.

If you downloaded the ARDroneUbuntu virtual machine:

To update your installation for this tutorial, click the “Update Tutorials” button in the ARDroneUbuntu Virtual Machine

If you use a custom installation:

At the terminal, enter the following:

$ roscd ardrone_autonomy
$ git pull
$ roscd ardrone_tutorials
$ git pull
$ roscd
$ rosmake ardrone_tutorials

AR.Drone State Feedback – A First Look

Preparation:

1. Find the Orange-[Green|Yellow|Blue]-Orange tag that came with the AR.Drone.
2. In your ARDroneUbunutu virtual machine (or custom installation), open a new terminal window and type:

   $ roscd ardrone_tutorials/launch

3. If you intend to fly with the keyboard, type:

   $ geany keyboard_controller_with_tags.launch

   Otherwise, if you intend to fly with the joystick:

   $ geany joystick_controller.launch

4. Now find the line:

   <param name="enemy_colors" value="3" />

5. And change the value to “1”, “2”, or “3”, depending on whether your tags are Orange-Green-Orange, Orange-Yellow-Orange or Orange-Blue-Orange respectively.

Tag Detection and Feedback Data

1. Run the system (but don’t takeoff with the drone!) using either the keyboard_controller:

   $ roslaunch ardrone_tutorials keyboard_controller_with_tags.launch

   or the joystick_controller:

   $ roslaunch ardrone_tutorials joystick_controller.launch

2. Open a new terminal window and type:

   $ rostopic list

   • This list shows all the active ROS “topics” used for communication to and from the AR.Drone
3. In the same terminal window, type:

   $ rostopic echo /ardrone/navdata

   • This echoes the AR.Drone’s feedback data into the terminal window.
   • Note that many items will be zero because the AR.Drone is not yet flying
4. Move the Orange-Color-Orange tag in front of the AR.Drone’s front camera. It should be detected, and the console output (from the previous step), should update to reflect this detection.
   • Note the tags_xc and tags_yc values. These represent the x/y location of the tag within the image.
   • The video display window will show a dot at these coordinates, and the distance measurement above the dot
5. Now place the tag somewhere that can be seen by the flying drone. After this is done, take off with the drone.
6. Watch the console output. What happens if you fly forwards? sideways? What variable changes if you rotate the drone?

Plotting the AR.Drone’s State

1. A more intuitive way of representing this feedback data is using a plot. PressCtrl-C to terminate the echo process from the last stage, and then type:

   $ rxplot /ardrone/navdata/rotX,/ardrone/navdata/rotY

   • This launches a plotting window showing the AR.Drone’s estimated roll (rotX) and pitch (rotY).
   • Similar plots could, for example, be generated for yaw (rotZ), x and y velocity (vx and vy), altitude (altd), etc.
2. Similarly to displaying the AR.Drone’s feedback data, we can use the plotting method to investigate how we are controlling the drone:

   $ rxplot /cmd_vel/linear/y,/cmd_vel/linear/x

   • This plots the roll (linear/y) and pitch (linear/x) commands.
   • Note that these are scaled with a value of $\pm1$ being the maximum commandable angle (set by the euler_angle_max in the launch file).
   • Note also that the direction of AR.Drone movement is commanded (ie. move along the x-axis = /cmd_vel/linear/x = commanded pitch), whereas the AR.Drone’s feedback data is named with reference to axis of rotation (ie. tilt around y-axis = /ardrone/navdata/rotY = feedback pitch).

Logging Data

1. To log data, we can use the inbuilt ROS command rosbag:

   $ rosbag record -O ARDroneFlight.bag /ardrone/navdata /cmd_vel

   • This will record the topics /ardrone/navdata and /cmd_vel to a “bag-file”, stamped with the current date and time.
   • Leave this process running for a few seconds, fly the AR.Drone around, and when finished press Ctrl-C to terminate.
   • Recording all topics is possible with rosbag record -a, however this also records the AR.Drone’s HD video stream, thus bag-files will quickly become very large
   • The advantage of using rosbag to log data, is that bag-files can be replayed at a later time, and interact very nicely with the existing ROS infrastructure.
2. Land the AR.Drone and close any windows / terminate any processes relating to the controller. Now, into the terminal window that ran the AR.Drone controller, type:

   $ roscore

   Now open a new terminal window and type:

   $ rxplot /cmd_vel/linear/y,/cmd_vel/linear/x

   Now finally (ignoring the warning from the previous step), switch back to the terminal window that you used to run the rosbag command and type:

   $ rosbag play ARDroneFlight.bag

   Does the flight look familiar? It should! This is the flight you recorded in the last step! Use Ctrl-C to terminate the above processes.

Custom Data Handling

In the last section, we investigated the AR.Drone’s feedback data by using ROS’s inbuilt plotting (rxplot) and logging (rosbag) commands. In this section, you will learn how we get this feedback data into your own application by writing a custom ROS Node. This node will be used to compare the commanded roll and pitch angles with the AR.Drone’s roll and pitch estimates, thus laying the foundations for the next tutorial, where we will use this feedback data to programmatically control the AR.Drone.

Messages, Topics and Nodes

In ROS terminology, a program is known as a Node. To interact with other nodes (for example the AR.Drone driver), each node will have some subscribers, which listen for information messages on a specific topic (communications channel); and some publishers, which send messages to a specific topic.

These messages can be anything. An example that we saw in the previous section was the Navdata message, which is sent over the /ardrone/navdata topic and contains feedback data from the AR.Drone.

We will now investigate this message a little further. Open up a new terminal window and type

rosmsg show ardrone_autonomy/Navdata

You should be presented with the following output:

$ rosmsg show ardrone_autonomy/Navdata
std_msgs/Header header
  uint32 seq
  time stamp
  string frame_id
float32 batteryPercent
uint32 state
int32 magX
int32 magY
int32 magZ
int32 pressure
int32 temp
float32 wind_speed
float32 wind_angle
float32 wind_comp_angle
float32 rotX
float32 rotY
float32 rotZ
int32 altd
float32 vx
float32 vy
float32 vz
float32 ax
float32 ay
float32 az
uint32 tags_count
uint32[] tags_type
uint32[] tags_xc
uint32[] tags_yc
uint32[] tags_width
uint32[] tags_height
float32[] tags_orientation
float32[] tags_distance
float32 tm

This listing shows us the various members of the Navdata message, including their datatypes. Note that the Navdata message has a header, which includes the time the message is sent.

Minimum Viable Subscriber

Open up a new terminal window and type:

$ roscd ardrone_tutorials
$ geany src/minimum_viable_subscriber.py

Up will pop a geany window, ready for your to edit the new file. Enter the following into the new geany window, then save and close the file:

import roslib
import rospy

roslib.load_manifest('ardrone_tutorials')

from ardrone_autonomy.msg import Navdata

def ReceiveData(data):
  print '[{0:.3f}] Pitch: {1:.3f}'.format(data.header.stamp.to_sec(), data.rotY)

rospy.init_node('minimum_viable_subscriber')
sub_Navdata = rospy.Subscriber('/ardrone/navdata', Navdata, ReceiveData)

while not rospy.is_shutdown():
  pass

We will now use the previously recorded bag-file to test the node that we’ve just written.

1. In one terminal window, enter:

   $ roscore

   Which begins the roscore process, which coordinates the interactions between nodes.

2. Now in a new terminal window type:

   $ roscd ardrone_tutorials
   $ python src/minimum_viable_subscriber.py

   This will launch the node that we’ve just programmed.

3. Finally, in a third terminal window, begin the bag-file playback with:

   $ rosbag play ARDroneFlight.bag

   • Note you must be in the same directory as the bag-file for the above command to work
4. If you switch back to the terminal window in which our new node is running, you should see a stream of time-stamped printouts with the drone’s pitch, for example:

   [1353596639.491] Pitch: 4.375
   [1353596639.511] Pitch: 4.128
   [1353596639.531] Pitch: 3.750
   [1353596639.552] Pitch: 3.304
   [1353596639.571] Pitch: 3.063

Minimum Viable Subscriber: A Closer Look

If you made it through the previous section, congratulations, you have successfully written and tested a ROS node, which listens for Navdata measurements from the AR.Drone (or equivalently: a recorded flight) and outputs a time-stamped pitch measurement. But how does it all work?

Lets take a look at the source-code from the previous section, line by line:

import roslib
import rospy

With these two lines, we import the standard ROS packages, required for every ROS node.

roslib.load_manifest('ardrone_tutorials')

Once the required packages are imported, load the manifest file. This file details the dependencies of the current project (for example, that ardrone_tutorials is dependent on the AR.Drone driver, ardrone_autonomy), which through the above command are loaded into the path.

from ardrone_autonomy.msg import Navdata

Once the project dependencies have been loaded, we can import those which are needed by the current node. In the above case, we load the Navdata message definition from the ardrone_autonomy package.

def ReceiveData(data):
  print '[{0:.3f}] Pitch: {1:.3f}'.format(data.header.stamp.to_sec(), data.rotY)

Here we define a callback function to handle the Navdata messages. This function prints a formatted string containing the message’s timestamp and the AR.Drone’s pitch angle. This callback function will be used later in the program.

rospy.init_node('minimum_viable_subscriber')

Before issuing any commands to the rospy package, we need to initialize a ROS node.

sub_Navdata = rospy.Subscriber('/ardrone/navdata', Navdata, ReceiveData)

We can now subscribe the node to various topics. Above we subscribe the node to the /ardrone/navdata topic, which carries Navdata messages. Received messages should then be handled by the previously-defined ReceiveData function.

while not rospy.is_shutdown():
  pass

Because our program is relying on callbacks for data processing, we need to stop the program from terminating. The above defines a loop that runs until the node is shut down (for example with Ctrl-C). In most situations the above loop would perform some sort of processing, however in our example, we simply pass (do nothing).

Publishing Data

In this section we will introduce the concept of publishing data to a topic (the opposite of subscribing). We will be using this more extensively in the next tutorial, where we will be programmatically controlling the AR.Drone by publishing messages to the /cmd_vel topic.

Minimum Viable Publisher

The goal of this section is to create a minimum_viable_publisher.py file such that Navdata messages are averaged over a second and the average pitch published to the (new) /ardrone/average_pitch topic. Modify the file as follows:

import roslib
import rospy

roslib.load_manifest('ardrone_tutorials')

from ardrone_autonomy.msg import Navdata
from std_msgs.msg import String

messages = []

def ReceiveData(data):
  messages.append(data)

rospy.init_node('minimum_viable_publisher')
sub_Navdata = rospy.Subscriber('/ardrone/navdata', Navdata, ReceiveData)
pub_Average = rospy.Publisher('/ardrone/average_pitch', String)

r = rospy.Rate(1)
while not rospy.is_shutdown():
  if len(messages)>0:
    avg = sum([m.rotY for m in messages])/len(messages)
    messages = []

    avgmsg = String()
    avgmsg.data = 'Average Pitch: {0:.3f}'.format(avg)
    pub_Average.publish(avgmsg)
  r.sleep()

You can run the program using the following series of commands:

1. In one terminal, run:

   $ roscore

2. In the next terminal, run the new program

   $ roscd ardrone_tutorials
   $ python src/minimum_viable_publisher.py

3. We now echo the published data to the topic we’ve created:

   $ rostopic echo /ardrone/average_pitch

4. And finally, like we’ve done twice before, we begin the playback of our recorded flight

   $ rosbag play ARDroneFlight.bag

5. Switching back to the terminal in which we ranrostopic echo, you should see a new message with the average pitch appearing once a second:

   data: Average Pitch: 0.649
   ---
   data: Average Pitch: 1.078
   ---
   data: Average Pitch: 3.781
   ---
   data: Average Pitch: 3.285
   ---

Minimum Viable Publisher: A Closer Look

Comparing minimum_viable_publisher.py to minimum_viable_subscriber.py from the last section, we can see many similarities:

from ardrone_autonomy.msg import Navdata
from std_msgs.msg import String

We additionally import the String message, required for our new topic.

messages = []
def ReceiveData(data):
  messages.append(data)

We have updated the navdata callback function to save the messages in a queue, rather than process them straight away.

Note: because we’re accessing the messages list in both a callback function and the main loop, we should be using Python’s threading.Lock to control access to the list and prevent race conditions. See ardrone_tutorials/src/drone_video_display.py for an example.

pub_Average = rospy.Publisher('/ardrone/average_pitch', String)

We create a new publisher, responsible for publishing messages of the String type to the /ardrone/average_pitch topic. Note that we do not need to explicitly initialize this topic, it will be initialized automatically on first usage.

The biggest change is in the program’s main loop, which I will now cover in a little more detail:

r = rospy.Rate(1)

Firstly we define a Rate object. We use this to control the frequency of the main processing loop. In our case, we want to print a message every second, thus we want a frequency of $1 \text{Hz}$.

while not rospy.is_shutdown():
  if len(messages)>0:
    avg = sum([m.rotY for m in messages])/len(messages)
    messages = []

    avgmsg = String()
    avgmsg.data = 'Average Pitch: {0:.3f}'.format(avg)
    pub_Average.publish(avgmsg)

If messages have been received, we calculate the average pitch using a python list comprehension to extract the pitch (.rotY) from every Navdata message. We then empty the messages queue.

We then construct a new String message and set its data property, before publishing it to the topic we previously created.

r.sleep()

Finally, we tell the rate object to sleep. This command will calculate how long it needs to sleep, such that the desired loop frequency is achieved.

Conclusions

I hope you’ve enjoyed working through this tutorial and learning about data flow in the ROS environment. In the next tutorial, which I will release in a month’s time, we will use what we have learned today to programmatically control the AR.Drone, while at the same time learning about PID control.

If you have any questions about the tutorial, please leave me a comment below and I will get back to you as soon as possible!

You can see all the tutorials in here: Up and flying with the AR.Drone and ROS.

If you liked this article, you may also be interested in:

• Parrot AR.Drone app harnesses crowd power to fast-track vision learning in robotic spacecraft
• Taking your first steps in robotics with the Thymio II
• Discovering social robotics with the Aisoy1
• Classroom robotics: Motivating independent learning and discovery

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Link to audio file (31:43)

In this episode we hear how the Spanish robotic startup Adele is creating a marketplace for robotics software. Through their platform, robot developers can buy software components for their robots, and software developers can sell their code, in a practical way. Examples of the software components, Adele calls them sparks, are speech recognition, synthetic speech, vision systems and user interface components. Their flagship project FIONA (Framework for Interactive-services Over Natural-conversational Agents) allows users to create intelligent and interactive virtual avatars.

Celestino Alvarez Martinez
Celestino Alvarez Martinez is an Industrial Engineer, specialized in Electronics and Automation from the University of Oviedo and with a Diploma of Advanced Studies in Robotics and Artificial Intelligence from Polytechnic University of Madrid. In 2002 he began his PhD in Robotics and Artificial Intelligence, under the supervision of Professor Antonio Barrientos (DISAM, UPM) in the field of human-robot interaction.

Lucia Fernandez Cossio
Lucia Fernandez Cossio is a Computer Science Engineer from the University of Oviedo. She is specialized in artificial intelligence and social robotics and currently holds the position of Project Manager at Adele Robots. She has experience in management and direction of R & D related to different topics within the field of Information Technology and Communication.

Links:

• Download mp3 (20.2MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Fiona website
• Adele’s website

What about Cloud Robotics?

The following video briefly explains the general motivation:

Generally, a robotics application interacts with its environment using sensors and actuators. This implies that the data produced by the sensors has to be processed and is normally used as feedback to determine the next action. For actions without hard real-time requirements, processing can be moved to the cloud. This reduces the necessary computing power on the robot, which can improve operation duration for mobile systems and may decrease costs.

Simplified cloud model: Each robot is connected to its own clone in the cloud (rectangular boxes), which are tightly interconnected with each other and with the databases (circular disks).

Many common high-level robotics tasks, such as planning or mapping, don’t require hard real-time. Mapping, for example, involves storing a reusable description of the environment to allow the robot to localize a landmark or itself. Additionally, maps can be used on a large scale to plan trajectories for moving to a landmark or on a small scale to grasp an object, which in turn might include querying an object database for grasp points to get the best position to grab the object. Because planning can be done ahead of time (i.e. act instead of react), it has no hard real-time constraints.

The advantages of offloading computation to the cloud can be even more pronounced when considering collaborative robot scenarios. For example, sharing data when creating a common map, or negotiating a cooperative plan when performing a common task can be more easily achieved with a centralized architecture with powerful computational capabilities.

RoboEarth allows robots to share knowledge in a centralized knowledge base. This centralized approach has the advantage of avoiding duplicate information and the necessity to constantly exchange newly generated knowledge to avoid duplicate learning. However, to benefit from the full potential of this approach not only requires a centralized storage of knowledge, but also centralized reasoning, i.e. a framework for cloud computation for robots.

The missing piece

Until now, RoboEarth only provided data storage for robots. With the introduction of Rapyuta, the RoboEarth Cloud Engine, the missing piece in the cloud infrastructure for robots is added. Rapyuta provides a platform where robots can create computing environments to offload there heavy computation into the cloud. These computing environments can either function as a personal cloud for individual robots or be shared by multiple robots. Additionally, computing environments have high-bandwidth connections to the RoboEarth database, allowing their processes to quickly access knowledge.

Technical aspects

Simplified overview of Rapyuta: The robots are connected to Rapyuta’s communications module in the cloud. Similarly, the cloud applications running inside the computing environments (LXC) are also connected to the communications module. The module manages communications from and to all connected elements.

To simplify the deployment and interaction of the cloud applications Rapyuta, similar to the Google App Engine, defines a protocol for cloud applications. However, in contrast to Google’s model where only web requests are used to communicate directly with the cloud application, Rapyuta introduces an additional layer. This means, a robot does not communicate directly with a cloud application, but uses Rapyuta as a medium. This model gives more flexibility and control over the system. For example, Rapyuta can be used as a relay to communicate directly between robots without the creation of a computing environment. Furthermore, the additional layer implies that Rapyuta allows to run multiple processes using sockets to communicate with each other. Google’s App Engine, however, prevents this, because web applications have no need for communicating processes to service web requests.

On the robot side Rapyuta uses common web protocols that allow access via a web browser. Like rosbridge and ros.js, Rapyuta uses WebSockets for persistent connections and JavaScript Object Notation (JSON) compatible strings for messages. However, because rosbridge provides only a subset of the functionality of Rapyuta the two protocols are not compatible. The main reason for this is that Rapyuta allows the client to interact with more than one ROS environment, i.e. computing environments, over the same connection, which leads to a more complex protocol for Rapyuta.

On the cloud application side the message system defined by the Robot Operating System (ROS) is used. This allows to deploy and run most of the more than 3’000 existing ROS packages (i.e., ROS applications) with minimal to no changes in the RoboEarth Cloud Engine.

To separate the ROS nodes, i.e. the cloud applications, for different users, Rapyuta provides individual computing environments. A robot can have many computing environments and a computing environment can be used by many robots, which allows a variety of deployment models. Linux Containers (LXC) were chosen for the implementation of computing environment to securely control all operational aspects such as communication as well as CPU, RAM, and disk usage. Additionally, Rapyuta allows different computing environments to communicate with each other using the ROS messaging system.

Rapyuta is open source. The Rapyuta Alpha release is available on github under the Apache License, Version 2.0.

For more information have a look at the Rapyuta documentation.

Further reading:

• Analysis: Robot learning in the cloud