What was the topic of your PhD research and why was it an interesting area?

My topic of research was developing an optical tactile sensor to track head motion during radiotherapy. I worked on both the hardware and software development of this sensor, though my focus was mostly on the software side. Its importance comes from the fact that during radiotherapy, patients undergoing head and neck cancer treatment are typically immobilised. This is usually done using a thermoplastic mask, which can feel very claustrophobic, or a stereotactic frame. Frames are more common for brain cancers, but they have to be surgically inserted into the patient’s skull using pins. Either of these immobilisation tools may be used depending on the situation. When patients are uncomfortable, they are more likely to move, which affects the accuracy of treatment, especially with thermoplastic masks.

Another major issue is that current systems use ceiling-mounted cameras to record patient motion. These cameras cannot be placed too close to the patient because of the electromagnetic environment around the equipment. Their view is also frequently occluded because the patient moves into a tunnel to receive the ionising beams, which makes it difficult to capture rotational motion.

One alternative is an infrared camera with a nose marker, but this only captures translational motion. Currently, when a nose tracker detects movement beyond a certain threshold, treatment is paused, the patient is repositioned, and treatment resumes. It is difficult to adapt this system to reliably measure the rotational motion of the patient’s head in the radiotherapy environment.

This is where the Motion Capture Pillow (MCP) comes in, which contains the optical tactile sensor I developed. The goal with this system is similar to the nose tracker, but with more accurate rotational feedback for the radiographer. It can be placed beneath the patient’s head and attached to the treatment bed. It estimates how much the patient’s neck is rotating and improves patient comfort. Radiographers can receive real-time feedback on both translational and rotational movement. The advantages of this system are that there are no occlusions, because the pillow is in direct contact with the patient’s head, and it is more compatible with radiotherapy environments because the sensor is non-ferromagnetic. Its premise is to maintain patient comfort, stay compact and easy to integrate into the pre-existing systems for radiotherapy, whilst improving the accuracy of the treatment through real-time head tracking.

Labelled diagram of the Motion Capture Pillow – Optical tactile sensor for head tracking during radiotherapy. The pneumatic pillow is a deformable rubber-like sheet with embedded white markers, held in its convex shape using air pressure. The fibrescope represents a non-ferromagnetic fibre optic bundle used as a lens extension to an area scan camera. The camera is ferromagnetic and will require safe positioning and fixation.

What were the main contributions of your work?

There were four main contributions to my work. The first contribution focused on making the system more non-ferromagnetic and improving the imaging and tracking approach. Previous work used a webcam and binary image processing within the optical tactile sensor to track marker displacement. I ultimately decided to use a fibrescope, optical flow tracking algorithm, and grayscale imaging instead, which improved the sensor’s tracking ability.

The second contribution focused on optimising marker density. The optical tactile sensor consists of an array of markers on a deformable rubber-like sheet, resembling a pillow. The deformation of these markers is captured by the camera. I investigated how dense the marker array needed to be by adjusting the spacing between markers to determine what worked best for this application.

The third contribution involved sensor fusion to improve reliability and robustness. To do this, I integrated a gyroscope and used Kalman filtering to fuse data from the gyroscope and the MCP. This was important for Gamma Knife systems, which are radiosurgery platforms used for brain cancers. They tend to have higher accuracy requirements than linear accelerators, which are commonly used for head and neck cancers, and lower constraints on the use of ferromagnetic components.

The final contribution was a participatory design study conducted in collaboration with clinicians and the social sciences department. We explored how the MCP could be integrated into hospital workflows and assessed its feasibility.

How feasible is it to integrate this sensor into hospital workflows?

Clinicians did seem to be very on board with it, but the study was more qualitative than quantitative. While they felt the idea had merit, there were reservations about adopting new technology and the associated learning curve.

They were also concerned about accuracy. Improving accuracy and reliability is essential for clinicians to feel confident using the system. At present, further development is needed before it can be widely implemented.

What future work is planned in this area?

One area to investigate is the differences between the mannequin and participant data. The pillow shape is controlled by a pneumatic system with a pressure sensor and air pump. When the patient or mannequin moves, pressure changes occur. The system compensates to maintain a set pressure, but this introduces errors in the motion readings. The mannequin produced more errors than the participant data. It may not accurately simulate human motion on the pillow, and the testing setup may introduce discrepancies that do not reflect real-world behaviour.
So, future work includes stabilising and refining the pressure control system to improve reliability. If necessary, reconsidering the use of gel on the sensors could be an option. Gel had been used previously but was abandoned due to clinician concerns about attenuation of ionising beams. However, if avoiding gel significantly compromises sensor performance, revisiting this approach may be worthwhile.

In addition, more participant data collection is needed. Not all previously collected data could be used due to ground-truth measurements being partially occluded in the experimental setup. Additional participant studies would provide a clearer understanding of performance across different individuals. Another priority is improving the fibrescope’s resolution and angle to better visualise high-density marker arrays. Hardware upgrades would help ensure a clearer field of view and improve overall system performance.

About Bhoomika

Bhoomika Gandhi is a recent PhD graduate from the University of Sheffield Medical Robotics group. Her undergraduate degree was in Bioengineering – Medical Devices and Instruments, with control engineering and robotics being the key themes.

Brandon Gilles, Founder and CEO of Luxonis, tells us his story about how Luxonis designed one of the most versatile perception platforms on the market.

Brandon took the lessons learned from his time at Ubiquiti, which transformed networking with network-on-a-chip architectures, and applied the mastery of embedded hardware and software to the OAK-D camera and the broader OAK line of products.

To refer to the OAK-D as a stereovision camera tells only part of the story. Aside from depth sensing, the OAK-D leverages the Intel Myriad X to perform perception computations directly on the camera in a highly power-efficient architecture.

Customers can also instantly leverage a wide array of open-source computer vision and AI packages that are pre-calibrated to the optics system.

Additionally, by leveraging a system-on-a-module design, the Luxonis team easily churns out a multitude of variations of the hardware platform to fit the wide variety of customer use cases. Tune in for more.

Brandon Gilles

Brandon Gilles is the Founder and CEO of Luxonis, maker of the OAK-D line of cameras. Brandon comes from a background in Electrical and RF Engineering. He spent his early career as a UniFi Lead at Ubiquiti, where his team helped bring Ubiquiti’s highly performant and power-efficient Unifi products to market.

Links

• Download mp3
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

Dr. Carolyn Matl, Research Scientist at Toyota Research Institute, explains why Interactive Perception and soft tactile sensors are critical for manipulating challenging objects such as liquids, grains, and dough. She also dives into “StRETcH” a Soft to Resistive Elastic Tactile Hand, a variable stiffness soft tactile end-effector, presented by her research group.

Carolyn Matl

Carolyn Matl is a research scientist at the Toyota Research Institute, where she works on robotic perception and manipulation with the Mobile Manipulation Team. She received her B.S.E in Electrical Engineering from Princeton University in 2016, and her Ph.D. in Electrical Engineering and Computer Sciences at the University of California, Berkeley in 2021. At Berkeley, she was awarded the NSF Graduate Research Fellowship and was advised by Ruzena Bajcsy. Her dissertation work focused on developing and leveraging non-traditional sensors for robotic manipulation of complicated objects and substances like liquids and doughs.

Carolyn Matl’s Related Research Videos

Links

• StRETcH: a Soft to Resistive Elastic Tactile Hand
• Carolyn Matl’s personal website
• Download mp3 (48.7 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

Over the past decade, camera technology has made gradual, and significant improvements thanks to the mobile phone industry. This has accelerated multiple industries, including Robotics. Today, Davide Scaramuzza discusses a step-change in camera innovation that has the potential to dramatically accelerate vision-based robotics applications.

Davide Scaramuzza deep dives on Event Cameras, which operate fundamentally different from traditional cameras. Instead of sampling every pixel on an imaging sensor at a fixed frequency, the “pixels” on an event camera all operate independently, and each responds to changes in illumination. This technology unlocks a multitude of benefits, including extremely highspeed imaging, removal of the concept of “framerate”, removal of data corruption due to having the sun in the sensor, reduced data throughput, and low power consumption. Tune in for more.

Davide Scaramuzza

Davide Scaramuzza is a Professor of Robotics and Perception at both departments of Informatics (University of Zurich) and Neuroinformatics (joint between the University of Zurich and ETH Zurich), where he directs the Robotics and Perception Group. His research lies at the intersection of robotics, computer vision, and machine learning, using standard cameras and event cameras, and aims to enable autonomous, agile, navigation of micro drones in search-and-rescue applications.

Links

• Download mp3 (49.3 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

In this episode, Audrow Nash speaks to Tim Chung, Program Manager in the Tactical Technology Office at the Defense Advanced Research Projects Agency (DARPA), on the DARPA Subterranean (SubT) Challenge. The SubT Challenge is a robotics challenge that aims to develop innovative technologies that would augment operations underground. In this conversation, they talk about the motivation of the SubT Challenge, its systems (hardware) and virtual (simulation) challenges, how the resulting technology has been shared with the world, on benchmarking the robotics behaviors in simulation, on the challenge of scoping the SubT Challenge, and on building the final environment for the systems challenge.

Episode Links

• Download the episode
• The SubT Challenge’s Website
• Tim’s LinkedIn
• DARPAtv’s YouTube Channel
• SubT’s Repository on OSRF

Podcast info

• Podcast website
• Apple Podcasts
• Spotify
• Podcast RSS feed
• Full episodes
• Clips

Oleg Shipitko, Chief Technical Director of Evocargo, an integrated logistics service company using autonomous vehicles speaks with Kate. Oleg talks about the need for automating operational logistics inside enclosed facilities centers and how their autonomous vehicles and other operational services can greatly improve the current way we transport goods within facilities such as ports, warehouses and factories. Oleg also discusses the sensing choices for the specific use cases of these autonomous vehicles.

Oleg Shipitko

Oleg Shipitko is the Chief Technical Director of Evocargo. He has a bachelors and masters degree in autonomous information and control systems (bachelors: 4.96/5.0, masters: 5.0/5.0), and a PhD in Space Science and Technology with focus in Mathematical modeling, numerical methods and program complexes.

Oleg has received numerous awards including: Best paper awarded at 32nd European Conference on Modeling and Simulation (ECMS-2018): Ground Vehicle Localization With Particle Filter Based On Simulated Road Marking Image and Best paper awarded at IV International Conference on Information Technology and Nanotechnology (ITNT-2018): Gaussian filtering for FPGA based image processing with High-Level Synthesis tools.

Links

• Evocargo
• Evocargo Youtube
• Download mp3
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

Torrey Smith, Co-Founder of Endiatx, is changing the reputation endoscopies have for being uncomfortable. At Endiatx, they are developing a pill-sized robot that you swallow, which will then livestream your digestive system for a doctor to view. Our interviewer Abate dives in.

Torrey Smith
Torrey Smith is the Co-Founder & CEO of Endiatx, a medical robotics company that manufactures tiny robotic pills capable of active movement inside the human stomach with control over internet protocol. Prior to launching Endiatx, he developed medical devices in the areas of endometrial ablation, atherectomy, therapeutic hypothermia, sleep apnea, and vascular closure.

An aerospace engineer by training, he takes a keen interest in the deep tech sector and is a proud mentor of up-and-coming founders at the Founder Institute. He is also the principal founder of the international arts collective known as Sextant, and he has had his art featured in the Smithsonian.

Links

• Download mp3 (49.3 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

Kate speaks with Brad Bogolea, CEO and Co-founder of Simbe Robotics. Simbe Robotics developed a mobile robot named Tally, which is bringing advanced shelf insights to improve the retail shopping experience.

Tally provides a state-of-the-art sensing system on a robust, scalable platform that collects analytics in real-time.

Brad Bogolea

Brad Bogolea is the CEO and Co-Founder of Simbe Robotics, where he is responsible for the company’s vision and execution of its leading retail intelligence solution. In November 2015, Brad brought to market the Tally robot, the world’s first autonomous shelf auditing and analytics solution to help retailers ensure merchandise is always stocked, in the right place, and correctly priced. The National Retail Federation Foundation has named Brad to its list of “People Shaping Retail’s Future.

Prior to Simbe, Brad spent 10 years in the energy and wireless sensor industry. Most recently Brad worked at Silver Spring Networks, where he led global product management and business development efforts for their energy management and data analytics platform for energy utilities. Products under Brad’s leadership allowed the world’s largest utility companies and their energy consumers alike to gain efficiency and visibility of their energy usage through smart sensors and data.

Brad holds a B.S. in Computer Science and Engineering from Pennsylvania State University where he also returned to serve as an Entrepreneur-in-Residence.

Links

• Simbe Robotics
• Download mp3 (34.8 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

In this episode, Audrow Nash speaks to Tobias Holmes, Quality Assurance Manager at Blue River Technologies. Blue River uses computer vision and robotics in agriculture and was acquired by John Deere in 2017. Tobias speaks about herbicide resistance, spraying weeds, quality assurance and testing on hardware, and on encouraging kids to learn robotics.

Episode Links

• Download the episode
• Tobias’ LinkedIn
• Blue River Technology
• FIRST Robotics
• Crystal Ray High School (East Bay)

Podcast info

• Podcast website
• Apple Podcasts
• Spotify
• RSS
• Full episodes on YouTube

In this episode, Audrow Nash interviews Erik Schluntz, co-founder and CTO of Cobalt Robotics, which makes a security guard robot. Erik speaks about how their robot handles elevators, how they have humans-in-the-loop to help their robot make decisions, robot body language, and gives advice for entrepreneurs.

Episode Links

• Cobalt’s website
• Erik’s website
• Video introducing the Cobalt robot
• Video of the Cobalt robot using an elevator

Podcast info

• Podcast website
• Apple Podcasts
• Spotify
• RSS
• Full episodes playlist
• Clips playlist

Anthony DiMare and Charles Chiau deep dive into how Bedrock Ocean is innovating in the world of Marine Surveys. At Bedrock Ocean, they are developing an Autonomous Underwater Vehicle (AUV) that is able to map the seafloor autonomously and at a high resolution. They are also developing a data platform to access, process, and visualize data captured from other companies at the seafloor.

Bedrock Ocean is solving two problems in the industry of Marine Surveying.
1. The vast majority of the seafloor is completely unmapped
2. The data that is captured from the seafloor is not standardized or centralized.

Seafloor data conducted by two different companies with the same or different hardware to capture the data can vary significantly in the calculated seafloor profile

Anthony DiMare
Anthony previously founded Nautilus Labs, a leading maritime technology company advancing the efficiency of ocean commerce through artificial intelligence. While at Nautilus, Anthony helped global companies solve challenges with distributed, siloed maritime data systems and built the early team that launched Nautilus Platform into large publicly listed shipping companies.

Charles Chiau
Charles, Bedrock’s CTO, was previously at SpaceX where he helped design the avionics systems for Crew Dragon. He also was a system integration engineer at Reliable Robotics working on their autonomous aviation system and was the CTO of DeepFlight where he worked on manned submersibles including ones for Tom Perkins, Richard Branson, and Steve Fossett.

Links

• Download mp3 (48.0 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

RE2 Sapien 6M Robot Arm

In this episode, Shihan Lu interviews Travis Schneider, Business Development Manager at RE2 Robotics, focusing on their work on mobile outdoor manipulation. Travis introduces several robotic products and services of RE2, including RE2 Sapien robotic arms and human-centered system integration, and how they can be used to help rather than replace workers. He also shares his thoughts about challenges in human-robot collaboration.

Travis Schneider
Travis Schneider, Business Development Manager at RE2 Robotics (RE2), works to foster commercial B2B partnerships with groups interested in leveraging RE2’s robotic technology for applications in a wide variety of markets, including aviation, construction, energy, and medical. With a background in Mechanical Engineering, Travis has an intimate knowledge of many of the fundamentals associated with robotic systems including electromechanical design, motion control, and associated software.

Links

• RE2 Robotics
• Download mp3 (12.7 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

Abate talks to Maxime Gariel, CTO of Xwing about the autonomous flight technology they are developing.

At Xwing, they are converting traditional aircraft into remotely operated aircraft. They do this by retrofitting planes with multiple sensors including cameras, radar, and lidar, and by developing sensor fusion algorithms to allow their planes to understand the world around them, using highly accurate perception algorithms.

Xwing’s autonomous flight technology allows a plane to taxi in the airport, takeoff, fly to a destination, avoid airborne and ground threats, and land, all without any human input. This technology not only enables autonomous flight but may also enhance the safety of manned aircraft by improving a plane’s ability to understand its surroundings.

Maxime Gariel

Maxime Gariel is the CTO of Xwing, a San Francisco based startup whose mission is to dramatically increase human mobility using fully autonomous aerial vehicles. Xwing is developing a Detect-And-Avoid system for unmanned and remotely piloted vehicles. Maxime is a pilot but he is passionate about making airplanes fly themselves.

Maxime joined Xwing from Rockwell Collins where he was a Principal GNC Engineer. He worked on autonomous aircraft projects including DARPA Gremlins and the AgustaWestland SW4 Solo autonomous helicopter. Before becoming Chief Engineer of the SW4 Solo’s flight control system, he was in charge of the system architecture, redundancy, and safety for the project.

Before Rockwell Collins, he worked on ADS-B based conflict detection as a postdoc at MIT and on autoland systems for airliners at Thales. Maxime earned his MS and PhD in Aerospace Engineering from Georgia Tech and his BS from ISAE-Supaéro (France).

Links

• Download mp3 (21.2 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

Most people associate the term “wearable” with a fitness tracker, smartwatch, or wireless earbuds. But what if cutting-edge biotechnology were integrated into your clothing, and could warn you when you were exposed to something dangerous?

A team of researchers from the Wyss Institute for Biologically Inspired Engineering at Harvard University and the Massachusetts Institute of Technology has found a way to embed synthetic biology reactions into fabrics, creating wearable biosensors that can be customized to detect pathogens and toxins and alert the wearer.

The team has integrated this technology into standard face masks to detect the presence of the SARS-CoV-2 virus in a patient’s breath. The button-activated mask gives results within 90 minutes at levels of accuracy comparable to standard nucleic acid-based diagnostic tests like polymerase chain reactions (PCR). The achievement is reported in Nature Biotechnology.

The wFDCF face mask can be integrated into any standard face mask. The wearer pushes a button on the mask that releases a small amount of water into the system, which provides results within 90 minutes. Credit: Wyss Institute at Harvard University

“We have essentially shrunk an entire diagnostic laboratory down into a small, synthetic biology-based sensor that works with any face mask, and combines the high accuracy of PCR tests with the speed and low cost of antigen tests,” said co-first author Peter Nguyen, Ph.D., a Research Scientist at the Wyss Institute. “In addition to face masks, our programmable biosensors can be integrated into other garments to provide on-the-go detection of dangerous substances including viruses, bacteria, toxins, and chemical agents.”

Taking cells out of the equation

The SARS-CoV-2 biosensor is the culmination of three years of work on what the team calls their wearable freeze-dried cell-free (wFDCF) technology, which is built upon earlier iterations created in the lab of Wyss Core Faculty member and senior author Jim Collins. The technique involves extracting and freeze-drying the molecular machinery that cells use to read DNA and produce RNA and proteins. These biological elements are shelf-stable for long periods of time and activating them is simple: just add water. Synthetic genetic circuits can be added to create biosensors that can produce a detectable signal in response of the presence of a target molecule.

The researchers first applied this technology to diagnostics by integrating it into a tool to address the Zika virus outbreak in 2015. They created biosensors that can detect pathogen-derived RNA molecules and coupled them with a colored or fluorescent indicator protein, then embedded the genetic circuit into paper to create a cheap, accurate, portable diagnostic. Following their success embedding their biosensors into paper, they next set their sights on making them wearable.

These flexible, wearable biosensors can be integrated into fabric to create clothing that can detect pathogens and environmental toxins and alert the wearer via a companion smartphone app. Credit: Wyss Institute at Harvard University

“Other groups have created wearables that can sense biomolecules, but those techniques have all required putting living cells into the wearable itself, as if the user were wearing a tiny aquarium. If that aquarium ever broke, then the engineered bugs could leak out onto the wearer, and nobody likes that idea,” said Nguyen. He and his teammates started investigating whether their wFDCF technology could solve this problem, methodically testing it in more than 100 different kinds of fabrics.

Then, the COVID-19 pandemic struck.

Pivoting from wearables to face masks

“We wanted to contribute to the global effort to fight the virus, and we came up with the idea of integrating wFDCF into face masks to detect SARS-CoV-2. The entire project was done under quarantine or strict social distancing starting in May 2020. We worked hard, sometimes bringing non-biological equipment home and assembling devices manually. It was definitely different from the usual lab infrastructure we’re used to working under, but everything we did has helped us ensure that the sensors would work in real-world pandemic conditions,” said co-first author Luis Soenksen, Ph.D., a Postdoctoral Fellow at the Wyss Institute.

The team called upon every resource they had available to them at the Wyss Institute to create their COVID-19-detecting face masks, including toehold switches developed in Core Faculty member Peng Yin’s lab and SHERLOCK sensors developed in the Collins lab. The final product consists of three different freeze-dried biological reactions that are sequentially activated by the release of water from a reservoir via the single push of a button.

The first reaction cuts open the SARS-CoV-2 virus’ membrane to expose its RNA. The second reaction is an amplification step that makes numerous double-stranded copies of the Spike-coding gene from the viral RNA. The final reaction uses CRISPR-based SHERLOCK technology to detect any Spike gene fragments, and in response cut a probe molecule into two smaller pieces that are then reported via a lateral flow assay strip. Whether or not there are any Spike fragments available to cut depends on whether the patient has SARS-CoV-2 in their breath. This difference is reflected in changes in a simple pattern of lines that appears on the readout portion of the device, similar to an at-home pregnancy test.

When SARS-CoV-2 particles are present, the wFDCF system cuts a molecular bond that changes the pattern of lines that form in the readout strip, similar to an at-home pregnancy test. Credit: Wyss Institute at Harvard University

The wFDCF face mask is the first SARS-CoV-2 nucleic acid test that achieves high accuracy rates comparable to current gold standard RT-PCR tests while operating fully at room temperature, eliminating the need for heating or cooling instruments and allowing the rapid screening of patient samples outside of labs.

“This work shows that our freeze-dried, cell-free synthetic biology technology can be extended to wearables and harnessed for novel diagnostic applications, including the development of a face mask diagnostic. I am particularly proud of how our team came together during the pandemic to create deployable solutions for addressing some of the world’s testing challenges,” said Collins, Ph.D., who is also the Termeer Professor of Medical Engineering & Science at MIT.

Beyond the COVID-19 pandemic

The Wyss Institute’s wearable freeze-dried cell-free (wFDCF) technology can quickly diagnose COVID-19 from virus in patients’ breath, and can also be integrated into clothing to detect a wide variety of pathogens and other dangerous substances. Credit: Wyss Institute at Harvard University

The face mask diagnostic is in some ways the icing on the cake for the team, which had to overcome numerous challenges in order to make their technology truly wearable, including capturing droplets of a liquid substance within a flexible, unobtrusive device and preventing evaporation. The face mask diagnostic omits electronic components in favor of ease of manufacturing and low cost, but integrating more permanent elements into the system opens up a wide range of other possible applications.

In their paper, the researchers demonstrate that a network of fiber optic cables can be integrated into their wFCDF technology to detect fluorescent light generated by the biological reactions, indicating detection of the target molecule with a high level of accuracy. This digital signal can be sent to a smartphone app that allows the wearer to monitor their exposure to a vast array of substances.

“This technology could be incorporated into lab coats for scientists working with hazardous materials or pathogens, scrubs for doctors and nurses, or the uniforms of first responders and military personnel who could be exposed to dangerous pathogens or toxins, such as nerve gas,” said co-author Nina Donghia, a Staff Scientist at the Wyss Institute.

The team is actively searching for manufacturing partners who are interested in helping to enable the mass production of the face mask diagnostic for use during the COVID-19 pandemic, as well as for detecting other biological and environmental hazards.

“This team’s ingenuity and dedication to creating a useful tool to combat a deadly pandemic while working under unprecedented conditions is impressive in and of itself. But even more impressive is that these wearable biosensors can be applied to a wide variety of health threats beyond SARS-CoV-2, and we at the Wyss Institute are eager to collaborate with commercial manufacturers to realize that potential,” said Don Ingber, M.D., Ph.D., the Wyss Institute’s Founding Director. Ingber is also the Judah Folkman Professor of Vascular Biology at Harvard Medical School and Boston Children’s Hospital, and Professor of Bioengineering at the Harvard John A. Paulson School of Engineering and Applied Sciences.

Additional authors of the paper include Nicolaas M. Angenent-Mari and Helena de Puig from the Wyss Institute and MIT; former Wyss and MIT member Ally Huang who is now at Ampylus; Rose Lee, Shimyn Slomovic, Geoffrey Lansberry, Hani Sallum, Evan Zhao, and James Niemi from the Wyss Institute; and Tommaso Galbersanini from Dreamlux.

This research was supported by the Defense Threat Reduction Agency under grant HDTRA1-14-1-0006, the Paul G. Allen Frontiers Group, the Wyss Institute for Biologically Inspired Engineering, Harvard University, Johnson & Johnson through the J&J Lab Coat of the Future QuickFire Challenge award, CONACyT grant 342369 / 408970, and MIT-692 TATA Center fellowship 2748460.

• PAPER – Wearable materials with embedded synthetic biology sensors for biomolecule detection. Peter Q. Nguyen, Luis R. Soenksen, Nina M. Donghia, Nicolaas M. Angenent-Mari, Helena de Puig, Ally Huang, Rose Lee, Shimyn Slomovic, Tommaso Galbersanini, Geoffrey Lansberry, Hani M. Sallum, Evan M. Zhao, James B. Niemi and James J. Collins. Nat Biotechnol (2021).

Intel RealSense ID was designed with privacy as a top priority. Purpose-built for user protection, Intel RealSense ID processes all facial images locally and encrypts all user data. (Credit: Intel Corporation)

Intel RealSense is known in the robotics community for its plug-and-play stereo cameras. These cameras make gathering 3D depth data a seamless process, with easy integrations into ROS to simplify the software development for your robots. From the RealSense team, Joel Hagberg talks about how they built this product, which allows roboticists to perform computer vision and machine learning at the edge.

Joel Hagberg

Joel Hagberg leads the Intel® RealSense Marketing. Product Management and Customer Support teams. He joined Intel in 2018 after a few years as an Executive Advisor working with startups in the IoT, AI, Flash Array, and SaaS markets. Before his Executive Advisor role, Joel spent two years as Vice President of Product Line Management at Seagate Technology with responsibility for their $13B product portfolio. He joined Seagate from Toshiba, where Joel spent 4 years as Vice President of Marketing and Product Management for Toshiba’s HDD and SSD product lines. Joel joined Toshiba with Fujitsu’s Storage Business acquisition, where Joel spent 12 years as Vice President of Marketing, Product Management, and Business Development. Joel’s Business Development efforts at Fujitsu focused on building emerging market business units in Security, Biometric Sensors, H.264 HD Video Encoders, 10GbE chips, and Digital Signage. Joel earned his bachelor’s degree in Electrical Engineering and Math from the University of Maryland. Joel also graduated from Fujitsu’s Global Knowledge Institute Executive MBA leadership program.

Links

• Download mp3 (14.6 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

Matt Bilsky, founder and CEO of FLX Solutions, discusses the snake-like robot he invented called the FLX BOT. The FLX BOT consists of modular links, each with a joint that can extend and rotate to get into tight spaces. Each link includes sensors including inertial measurement units and a camera. The robot is used to navigate and work in challenging environments, such as above ceilings and within walls. Matt discusses the key innovations of his product as well as his academic and entrepreneurial journey that led him to the FLX BOT.

Matt Bilksy

Matt Bilsky, PhD, PE is the inventor of the FLX BOT, a licensed Professional Engineer, a Mechanical Engineering professor at Lehigh University, and a former repair/maintenance contractor. In 2017, he was awarded with the Lehigh University Entrepreneurship Educator of the Year. Matt has a Mechanical Engineering Ph.D. from Lehigh University focused on smart product design, Technical Entrepreneurship, and mechatronics. He holds two additional Lehigh degrees: a BS in Mechanical Engineering with an Electrical Engineering minor and a Master of Engineering degree also in Mechanical Engineering. Since he was a child Matt has been an innovator. In his basement shop he designed and built numerous electronic gadgets. In 2003 he started his first company, Mattcomp Services LLC, offering computer repair, networking, home theater, and handyman services. He also created a web hosting company in 2005, Mattcomp Hosting, including all necessary back-end components on dedicated servers.

Links

• Download mp3 (25.1MB)
• FLX Solutions
• FLX Youtube channel
• Matt’s Portfolio
• Matt’s Youtube channel

MIT researchers developed a “Digger Finger” robot that digs through granular material, like sand and gravel, and senses the shapes of buried objects. The technology could aid in disarming buried bombs or inspecting underground cables. Image courtesy of the researchers.

By Daniel Ackerman

Over the years, robots have gotten quite good at identifying objects — as long as they’re out in the open.

Discerning buried items in granular material like sand is a taller order. To do that, a robot would need fingers that were slender enough to penetrate the sand, mobile enough to wriggle free when sand grains jam, and sensitive enough to feel the detailed shape of the buried object.

MIT researchers have now designed a sharp-tipped robot finger equipped with tactile sensing to meet the challenge of identifying buried objects. In experiments, the aptly named Digger Finger was able to dig through granular media such as sand and rice, and it correctly sensed the shapes of submerged items it encountered. The researchers say the robot might one day perform various subterranean duties, such as finding buried cables or disarming buried bombs.

The research will be presented at the next International Symposium on Experimental Robotics. The study’s lead author is Radhen Patel, a postdoc in MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). Co-authors include CSAIL PhD student Branden Romero, Harvard University PhD student Nancy Ouyang, and Edward Adelson, the John and Dorothy Wilson Professor of Vision Science in CSAIL and the Department of Brain and Cognitive Sciences.

Seeking to identify objects buried in granular material — sand, gravel, and other types of loosely packed particles — isn’t a brand new quest. Previously, researchers have used technologies that sense the subterranean from above, such as Ground Penetrating Radar or ultrasonic vibrations. But these techniques provide only a hazy view of submerged objects. They might struggle to differentiate rock from bone, for example.

“So, the idea is to make a finger that has a good sense of touch and can distinguish between the various things it’s feeling,” says Adelson. “That would be helpful if you’re trying to find and disable buried bombs, for example.” Making that idea a reality meant clearing a number of hurdles.

The team’s first challenge was a matter of form: The robotic finger had to be slender and sharp-tipped.

In prior work, the researchers had used a tactile sensor called GelSight. The sensor consisted of a clear gel covered with a reflective membrane that deformed when objects pressed against it. Behind the membrane were three colors of LED lights and a camera. The lights shone through the gel and onto the membrane, while the camera collected the membrane’s pattern of reflection. Computer vision algorithms then extracted the 3D shape of the contact area where the soft finger touched the object. The contraption provided an excellent sense of artificial touch, but it was inconveniently bulky.

A closeup photograph of the new robot and a diagram of its parts. Image courtesy of the researchers.

For the Digger Finger, the researchers slimmed down their GelSight sensor in two main ways. First, they changed the shape to be a slender cylinder with a beveled tip. Next, they ditched two-thirds of the LED lights, using a combination of blue LEDs and colored fluorescent paint. “That saved a lot of complexity and space,” says Ouyang. “That’s how we were able to get it into such a compact form.” The final product featured a device whose tactile sensing membrane was about 2 square centimeters, similar to the tip of a finger.

With size sorted out, the researchers turned their attention to motion, mounting the finger on a robot arm and digging through fine-grained sand and coarse-grained rice. Granular media have a tendency to jam when numerous particles become locked in place. That makes it difficult to penetrate. So, the team added vibration to the Digger Finger’s capabilities and put it through a battery of tests.

“We wanted to see how mechanical vibrations aid in digging deeper and getting through jams,” says Patel. “We ran the vibrating motor at different operating voltages, which changes the amplitude and frequency of the vibrations.” They found that rapid vibrations helped “fluidize” the media, clearing jams and allowing for deeper burrowing — though this fluidizing effect was harder to achieve in sand than in rice.

Top row: 3D printed objects used for the object identification experiment. Middle row: Example image data when the Digger Finger directly touches a 3d printed object. Bottom row: Example image data when the Digger Finger touches a 3d printed object that is buried in sand. Image courtesy of the researchers.

They also tested various twisting motions in both the rice and sand. Sometimes, grains of each type of media would get stuck between the Digger-Finger’s tactile membrane and the buried object it was trying to sense. When this happened with rice, the trapped grains were large enough to completely obscure the shape of the object, though the occlusion could usually be cleared with a little robotic wiggling. Trapped sand was harder to clear, though the grains’ small size meant the Digger Finger could still sense the general contours of target object.

Patel says that operators will have to adjust the Digger Finger’s motion pattern for different settings “depending on the type of media and on the size and shape of the grains.” The team plans to keep exploring new motions to optimize the Digger Finger’s ability to navigate various media.

Adelson says the Digger Finger is part of a program extending the domains in which robotic touch can be used. Humans use their fingers amidst complex environments, whether fishing for a key in a pants pocket or feeling for a tumor during surgery. “As we get better at artificial touch, we want to be able to use it in situations when you’re surrounded by all kinds of distracting information,” says Adelson. “We want to be able to distinguish between the stuff that’s important and the stuff that’s not.”

Funding for this research was provided, in part, by the Toyota Research Institute through the Toyota-CSAIL Joint Research Center; the Office of Naval Research; and the Norwegian Research Council.

• PAPER – Digger Finger: GelSight Tactile Sensor for Object Identification Inside Granular Media. Radhen Patel, Rui Ouyang, Branden Romero and Edward Adelson.

The new technology pairs wireless sensing with artificial intelligence to determine when a patient is using an insulin pen or inhaler, and it flags potential errors in the patient’s administration method. | Image: courtery of the researchers

From swallowing pills to injecting insulin, patients frequently administer their own medication. But they don’t always get it right. Improper adherence to doctors’ orders is commonplace, accounting for thousands of deaths and billions of dollars in medical costs annually. MIT researchers have developed a system to reduce those numbers for some types of medications.

The new technology pairs wireless sensing with artificial intelligence to determine when a patient is using an insulin pen or inhaler, and flags potential errors in the patient’s administration method. “Some past work reports that up to 70% of patients do not take their insulin as prescribed, and many patients do not use inhalers properly,” says Dina Katabi, the Andrew and Erna Viteri Professor at MIT, whose research group has developed the new solution. The researchers say the system, which can be installed in a home, could alert patients and caregivers to medication errors and potentially reduce unnecessary hospital visits.

The research appears today in the journal Nature Medicine. The study’s lead authors are Mingmin Zhao, a PhD student in MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), and Kreshnik Hoti, a former visiting scientist at MIT and current faculty member at the University of Prishtina in Kosovo. Other co-authors include Hao Wang, a former CSAIL postdoc and current faculty member at Rutgers University, and Aniruddh Raghu, a CSAIL PhD student.

Some common drugs entail intricate delivery mechanisms. “For example, insulin pens require priming to make sure there are no air bubbles inside. And after injection, you have to hold for 10 seconds,” says Zhao. “All those little steps are necessary to properly deliver the drug to its active site.” Each step also presents opportunity for errors, especially when there’s no pharmacist present to offer corrective tips. Patients might not even realize when they make a mistake — so Zhao’s team designed an automated system that can.

Their system can be broken down into three broad steps. First, a sensor tracks a patient’s movements within a 10-meter radius, using radio waves that reflect off their body. Next, artificial intelligence scours the reflected signals for signs of a patient self-administering an inhaler or insulin pen. Finally, the system alerts the patient or their health care provider when it detects an error in the patient’s self-administration.

Wireless sensing technology could help improve patients’ technique with inhalers and insulin pens. | Image: Christine Daniloff, MIT

The researchers adapted their sensing method from a wireless technology they’d previously used to monitor people’s sleeping positions. It starts with a wall-mounted device that emits very low-power radio waves. When someone moves, they modulate the signal and reflect it back to the device’s sensor. Each unique movement yields a corresponding pattern of modulated radio waves that the device can decode. “One nice thing about this system is that it doesn’t require the patient to wear any sensors,” says Zhao. “It can even work through occlusions, similar to how you can access your Wi-Fi when you’re in a different room from your router.”

The new sensor sits in the background at home, like a Wi-Fi router, and uses artificial intelligence to interpret the modulated radio waves. The team developed a neural network to key in on patterns indicating the use of an inhaler or insulin pen. They trained the network to learn those patterns by performing example movements, some relevant (e.g. using an inhaler) and some not (e.g. eating). Through repetition and reinforcement, the network successfully detected 96 percent of insulin pen administrations and 99 percent of inhaler uses.

Once it mastered the art of detection, the network also proved useful for correction. Every proper medicine administration follows a similar sequence — picking up the insulin pen, priming it, injecting, etc. So, the system can flag anomalies in any particular step. For example, the network can recognize if a patient holds down their insulin pen for five seconds instead of the prescribed 10 seconds. The system can then relay that information to the patient or directly to their doctor, so they can fix their technique.

“By breaking it down into these steps, we can not only see how frequently the patient is using their device, but also assess their administration technique to see how well they’re doing,” says Zhao.

The researchers say a key feature of their radio wave-based system is its noninvasiveness. “An alternative way to solve this problem is by installing cameras,” says Zhao. “But using a wireless signal is much less intrusive. It doesn’t show peoples’ appearance.”

He adds that their framework could be adapted to medications beyond inhalers and insulin pens — all it would take is retraining the neural network to recognize the appropriate sequence of movements. Zhao says that “with this type of sensing technology at home, we could detect issues early on, so the person can see a doctor before the problem is exacerbated.”

Abate interviews Tessa Lau on her startup Dusty Robotics which is innovating in the field of construction.

At Dusty Robotics, they developed a robot to automate the laying of floor plans on the floors in construction sites. Typically, this is done manually using a tape measure and reading printed out plans. This difficult task can often take a team of two a week to complete. Time-consuming tasks like this are incredibly expensive on a construction site where multiple different teams are waiting on this task to complete. Any errors in this process are even more time-consuming to fix. By using a robot to automatically convert 3d models of building plans into markings on the floors, the amount of time and errors are dramatically reduced.

Dr. Tessa Lau

Dr. Tessa Lau is an experienced entrepreneur with expertise in AI, machine learning, and robotics. She is currently Founder/CEO at Dusty Robotics, a construction robotics company building robot-powered tools for the modern construction workforce. Prior to Dusty, she was CTO/co-founder at Savioke, where she orchestrated the deployment of 75+ delivery robots into hotels and high-rises. Previously, Dr. Lau was a Research Scientist at Willow Garage, where she developed simple interfaces for personal robots. She also spent 11 years at IBM Research working in business process automation and knowledge capture. More generally, Dr. Lau is interested in technology that gives people super-powers, and building businesses that bring that technology into people’s lives. Dr. Lau was recognized as one of the Top 5 Innovative Women to Watch in Robotics by Inc. in 2018 and one of Fast Company’s Most Creative People in 2015. Dr. Lau holds a PhD in Computer Science from the University of Washington.

Links

• Download mp3 (20.9 MB)
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

Humanoid robot hand. Futuristic cyborg concept.

While modern cameras provide machines with a very well-developed sense of vision, robots still lack such a comprehensive solution for their sense of touch. At ETH Zurich, in the group led by Prof. Raffaello D’Andrea at the Institute for Dynamic Systems and Control, we have developed a tactile sensing principle that allows robots to retrieve rich contact feedback from their interactions with the environment. I recently described our approach in a TEDx talk at the last TEDxZurich. The talk features a tech demo that introduces the novel tactile sensing technology targeting the next generation of soft robotic skins.

Design

The sensing technique is based on a camera that tracks fluorescent particles, which are densely and randomly distributed within a soft, deformable gel. The randomness of the patterns simplifies production of the gel and their density provides strain information at each pixel of the resulting image. In addition, the technique does not make any assumption about the shape of the sensing surface, which can exhibit an arbitrary geometry.

Data Processing

While the images capturing the particles’ motion are intuitive and to some extent visually interpretable, the extraction of accurate physical quantities is challenging. In order to overcome the complexity of modeling the behavior of soft materials in real-time, the information extracted from the images is mapped to the distribution of the applied (shear and pressure) contact forces in a data-driven fashion. Specifically, a neural network is employed, which is trained entirely via accurate finite-element simulations to extract the aforementioned mapping.

Applications

This technology has the potential to impact several application fields, the obvious one being robotic manipulation. In a recent proof-of-concept, we have demonstrated how highly dynamic manipulation tasks can be achieved via the sole use of tactile sensing, as shown in the video below.

In addition, the versatility of this approach makes it suitable for various products beyond the robotics domain. In fact, an artificial sense of touch may find applications in smart prosthetic systems, with the potential to restore tactile sensations to people who have lost limbs.

Kate speaks with Anni Kern, Head of Communication, strategy, and teams at Cybathlon for over four years. She describes the motivation and concepts for the Cybathlon organizations to develop a common platform to remove barriers between people with disabilities, technology developers, and the public. Anni also describes the specifics of Cybathlon competitions and the organization and planning.

Anni Kern

Anni Kern, CYBATHLON Head of Communication, Strategy, Teams and CYBATHLON @school. ETH Zurich is the inventor of the CYBATHLON project. The CYBATHLON offers a platform to advance research in the field of assistive technology and to promote dialogue with the public about the inclusion of people with disabilities in everyday life. The core of CYBATHLON of ETH Zurich is a unique competition in which people with physical disabilities compete against each other to complete everyday tasks using state-of-the-art technical assistance systems. At the CYBATHLON 2016, Anni Kern was responsible for the coordination and the support of the participating international CYBATHLON teams. Since 2017 she is among others responsible for the strategy, communication, conception of the live event as well as the CYBATHLON @school programme.

Links

• Download mp3 (11.8MB)
• Youtube channel
• Twitter: @cybathlon
• Facebook: @cybathlon 
• Insta: cybathlon_eth_zurich
• Linkedin: cybathlon

Interesting discussion with Prof. Ali Khademhosseini, CEO of the Terasaki Institute, and one of the pioneers of the Bioengineering field. Prof. Ali’s journey from Harvard and UCLA to the Terasaki Institute is truly inspiring. What does the institute do to bring a product to the real world? Learn about the design challenges of biomaterials, organs on a chip, and soft robotics in this episode of the IEEE RAS Soft Robotics Podcast.

You can also listen to this podcast on Spotify and Apple Podcasts, and explore support options on Patreon.

Interesting discussion with Hod Lipson, head of Creative Machines Lab, Columbia University in New York. Can robots be self-aware? Can they design other robots and self-repair? Why should we evolve robots to do tasks that animals do so well? Why don’t we have useful autonomous robots in the real world yet? Find out Hod’s answers to these questions and updates on VoxCAD development for designing and simulation of soft robots in this episode of the IEEE RAS Soft Robotics Podcast.

What’s more, Hod gave his personal advice to roboticists being interviewed for an assistant professorship and to 1st-year robotics PhD students looking for a thesis topic, and he also commented on his approach to the ethical dilemma of military funding scientific research. I hope you enjoy listening to the episode! You can check Hod Lipson’s portfolio here: https://www.hodlipson.com/

You can also listen to this podcast on Spotify and Apple Podcasts, and explore support options on Patreon.

Graduate student Moritz Graule demonstrates a fabric arm sleeve with embedded sensors. The sensors detect the small changes in the Graule’s forearm muscle through the fabric. Such a sleeve could be used in everything from virtual reality simulations and sportswear to clinical diagnostics for neurodegenerative diseases like Parkinson’s Disease. Credit: Oluwaseun Araromi/Harvard SEAS

By Leah Burrows / SEAS communications

Newly engineered slinky-like strain sensors for textiles and soft robotic systems survive the washing machine, cars and hammers.

Think about your favorite t-shirt, the one you’ve worn a hundred times, and all the abuse you’ve put it through. You’ve washed it more times than you can remember, spilled on it, stretched it, crumbled it up, maybe even singed it leaning over the stove once. We put our clothes through a lot and if the smart textiles of the future are going to survive all that we throw at them, their components are going to need to be resilient.

Now, researchers from the Harvard John A. Paulson School of Engineering and Applied Sciences (SEAS) and the Wyss Institute for Biologically Inspired Engineering have developed an ultra-sensitive, seriously resilient strain sensor that can be embedded in textiles and soft robotic systems. The research is published in Nature.

“Current soft strain gauges are really sensitive but also really fragile,” said Oluwaseun Araromi, Ph.D., a Research Associate in Materials Science and Mechanical Engineering at SEAS and the Wyss Institute and first author of the paper. “The problem is that we’re working in an oxymoronic paradigm — highly sensitivity sensors are usually very fragile and very strong sensors aren’t usually very sensitive. So, we needed to find mechanisms that could give us enough of each property.”

In the end, the researchers created a design that looks and behaves very much like a Slinky.

“A Slinky is a solid cylinder of rigid metal but if you pattern it into this spiral shape, it becomes stretchable,” said Araromi. “That is essentially what we did here. We started with a rigid bulk material, in this case carbon fiber, and patterned it in such a way that the material becomes stretchable.”

The pattern is known as a serpentine meander, because its sharp ups and downs resemble the slithering of a snake. The patterned conductive carbon fibers are then sandwiched between two pre-strained elastic substrates. The overall electrical conductivity of the sensor changes as the edges of the patterned carbon fiber come out of contact with each other, similar to the way the individual spirals of a slinky come out of contact with each other when you pull both ends. This process happens even with small amounts of strain, which is the key to the sensor’s high sensitivity.

A close-up view of the sensor’s patterned conductive carbon fibers. The fibers are sandwiched between two prestrained elastic substrates. The overall electrical conductivity of the sensor changes as the edges of the patterned carbon fiber come out of contact with each other. Credit: James Weaver/Harvard SEAS

Unlike current highly sensitive stretchable sensors, which rely on exotic materials such as silicon or gold nanowires, this sensor doesn’t require special manufacturing techniques or even a clean room. It could be made using any conductive material.

The researchers tested the resiliency of the sensor by stabbing it with a scalpel, hitting it with a hammer, running it over with a car, and throwing it in a washing machine ten times. The sensor emerged from each test unscathed. To demonstrate its sensitivity, the researchers embedded the sensor in a fabric arm sleeve and asked a participant to make different gestures with their hand, including a fist, open palm, and pinching motion. The sensors detected the small changes in the subject’s forearm muscle through the fabric and a machine learning algorithm was able to successfully classify these gestures.

“These features of resilience and the mechanical robustness put this sensor in a whole new camp,” said Araromi.

Such a sleeve could be used in everything from virtual reality simulations and sportswear to clinical diagnostics for neurodegenerative diseases like Parkinson’s Disease. Harvard’s Office of Technology Development has filed to protect the intellectual property associated with this project.

“The combination of high sensitivity and resilience are clear benefits of this type of sensor,” said senior author Robert Wood, Ph.D., Associate Faculty member at the Wyss Institute, and the Charles River Professor of Engineering and Applied Sciences at SEAS. “But another aspect that differentiates this technology is the low cost of the constituent materials and assembly methods. This will hopefully reduce the barriers to get this technology widespread in smart textiles and beyond.”

This ultra-sensitive resilient strain sensor can be embedded in textiles and soft robotic systems. Credit: Oluwaseun Araromi/Harvard SEAS

“We are currently exploring how this sensor can be integrated into apparel due to the intimate interface to the human body it provides,” says co-author and Wyss Associate Faculty member Conor Walsh, Ph.D., who also is the Paul A. Maeder Professor of Engineering and Applied Sciences at SEAS. “This will enable exciting new applications by being able to make biomechanical and physiological measurements throughout a person’s day, not possible with current approaches.”

The combination of high sensitivity and resilience are clear benefits of this type of sensor. But another aspect that differentiates this technology is the low cost of the constituent materials and assembly methods. This will hopefully reduce the barriers to get this technology widespread in smart textiles and beyond.

Robert Wood

The research was co-authored by Moritz A. Graule, Kristen L. Dorsey, Sam Castellanos, Jonathan R. Foster, Wen-Hao Hsu, Arthur E. Passy, James C. Weaver, Senior Staff Scientist at SEAS and Joost J. Vlassak, the Abbott and James Lawrence Professor of Materials Engineering at SEAS. It was funded through the university’s strategic research alliance with Tata. The 6-year, $8.4M alliance was established in 2016 to advance Harvard innovation in fields including robotics, wearable technologies, and the internet of things (IoT).

• PUBLICATION – Ultra-sensitive and resilient compliant strain gauges for soft machines. Oluwaseun A. Araromi, Moritz A. Graule, Kristen L. Dorsey, Sam Castellanos, Jonathan R. Foster, Wen-Hao Hsu, Arthur E. Passy, Joost J. Vlassak, James C. Weaver, Conor J. Walsh and Robert J. Wood. Nature, vol. 587, pp. 219-224 (2020)

In this episode, Shihan Lu interviews Jivko Sinapov, Assistant Professor in the Computer Science Department at Tufts University, about his work on behavior-grounded multisensory perception and exploration in robotics. Dr. Sinapov discusses several perspectives on multisensory perception in robotics, including data collection, data fusion, and robot control and planning. He also shares his experience about using robotics for K-12 education.

Jivko Sinapov

Jivko Sinapov received his Ph.D. in Computer Science and Human-Computer Interaction from Iowa State University (ISU). While working toward his Ph.D. at ISU’s Developmental Robotics Lab, he developed novel methods for behavioral object exploration and multi-modal perception. He went on to be a clinical assistant professor with the Texas Institute for Discovery, Education, and Science at UT Austin and a postdoctoral associate working with Peter Stone at the Artificial Intelligence lab. Sinapov’s research interests include developmental robotics, computational perception, autonomous manipulation, and human-robot interaction.

Links

• Download mp3 (10.6 MB)
• Sinapov’s research
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

An MIT team used MorphSensor to design multiple applications, including a pair of glasses that monitor light absorption to protect eye health. Credits: Photo courtesy of the researchers.

By Rachel Gordon

We’ve come a long way since the first 3D-printed item came to us by way of an eye wash cup, to now being able to rapidly fabricate things like car parts, musical instruments, and even biological tissues and organoids. 

While much of these objects can be freely designed and quickly made, the addition of electronics to embed things like sensors, chips, and tags usually requires that you design both separately, making it difficult to create items where the added functions are easily integrated with the form. 

Now, a 3D design environment from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) lets users iterate an object’s shape and electronic function in one cohesive space, to add existing sensors to early-stage prototypes.

The team tested the system, called MorphSensor, by modeling an N95 mask with a humidity sensor, a temperature-sensing ring, and glasses that monitor light absorption to protect eye health.

MorphSensor automatically converts electronic designs into 3D models, and then lets users iterate on the geometry and manipulate active sensing parts. This might look like a 2D image of a pair of AirPods and a sensor template, where a person could edit the design until the sensor is embedded, printed, and taped onto the item. 

To test the effectiveness of MorphSensor, the researchers created an evaluation based on standard industrial assembly and testing procedures. The data showed that MorphSensor could match the off-the-shelf sensor modules with small error margins, for both the analog and digital sensors.

“MorphSensor fits into my long-term vision of something called ‘rapid function prototyping’, with the objective to create interactive objects where the functions are directly integrated with the form and fabricated in one go, even for non-expert users,” says CSAIL PhD student Junyi Zhu, lead author on a new paper about the project. “This offers the promise that, when prototyping, the object form could follow its designated function, and the function could adapt to its physical form.” 

MorphSensor in action 

Imagine being able to have your own design lab where, instead of needing to buy new items, you could cost-effectively update your own items using a single system for both design and hardware. 

For example, let’s say you want to update your face mask to monitor surrounding air quality. Using MorphSensor, users would first design or import the 3D face mask model and sensor modules from either MorphSensor’s database or online open-sourced files. The system would then generate a 3D model with individual electronic components (with airwires connected between them) and color-coding to highlight the active sensing components.  

Designers can then drag and drop the electronic components directly onto the face mask, and rotate them based on design needs. As a final step, users draw physical wires onto the design where they want them to appear, using the system’s guidance to connect the circuit. 

Once satisfied with the design, the “morphed sensor” can be rapidly fabricated using an inkjet printer and conductive tape, so it can be adhered to the object. Users can also outsource the design to a professional fabrication house.  

To test their system, the team iterated on EarPods for sleep tracking, which only took 45 minutes to design and fabricate. They also updated a “weather-aware” ring to provide weather advice, by integrating a temperature sensor with the ring geometry. In addition, they manipulated an N95 mask to monitor its substrate contamination, enabling it to alert its user when the mask needs to be replaced.

In its current form, MorphSensor helps designers maintain connectivity of the circuit at all times, by highlighting which components contribute to the actual sensing. However, the team notes it would be beneficial to expand this set of support tools even further, where future versions could potentially merge electrical logic of multiple sensor modules together to eliminate redundant components and circuits and save space (or preserve the object form). 

Zhu wrote the paper alongside MIT graduate student Yunyi Zhu; undergraduates Jiaming Cui, Leon Cheng, Jackson Snowden, and Mark Chounlakone; postdoc Michael Wessely; and Professor Stefanie Mueller. The team will virtually present their paper at the ACM User Interface Software and Technology Symposium. 

This material is based upon work supported by the National Science Foundation.

• PUBLICATION – MorphSensor: A 3D Electronic Design Tool for Reforming Sensor Modules. Junyi Zhu, Jiaming Cui, Leon Cheng, Jackson Snowden, Mark Chounlakone, Michael Wessely, Stefanie Mueller. UIST ’20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology.

In this episode, Abate interviews Josh Lessing, co-founder and CEO of Root AI. At Root AI they are developing a system that tracks data on the farm and autonomously harvests crops using delicate grippers and computer vision. Lessing talks about the path they took to build a product with good market fit and how they brought a venture capital backed startup to market.

Josh Lessing

Josh is one of the world’s leading minds on developing robotics and AI systems for the food industry, previously serving as the Director of R&D at Soft Robotics Inc. His current venture, Root AI, is integrating advanced robotics, vision systems and machine perception to automate agriculture. Josh was a Postdoctoral Fellow in Materials Science & Robotics at Harvard University, having earned his Ph.D. studying Biophysics & Physical Chemistry at the Massachusetts Institute of Technology and received an Sc.B. in Chemistry from Brown University.

Links

• Download mp3 (16.0 MB)
• Listen to this episode on Spotify
• Subscribe to Robohub using iTunes, RSS, or Spotify
• Support us on Patreon

In this episode, Lauren Klein interviews Gennaro Notimista, a robotics PhD student in the Georgia Robotics and InTelligent Systems Laboratory at Georgia Tech. Gennaro discusses the SlothBot, a solar-powered robot that slowly traverses wires, like its animal namesake, to monitor the environment.

Gennaro Notomista

Photo from news.gatech.edu

Gennaro Notomista is a robotics PhD student in the Georgia Robotics and InTelligent Systems Laboratory at Georgia Tech. Gennaro studies control frameworks, with the goal of making robots robust against a changing environment so they can handle long-duration deployments. Toward this goal, he explores constraints-driven control and approaches to coverage control, or enabling robots to traverse closed environments.  In addition to the SlothBot, Gennaro has applied his research to areas such as autonomous driving and swarm robotics.

Links

• Download mp3 (12.6 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Abate interviews Erin Bishop from Sense Photonics about the technology in their “Solid State” LiDAR sensors that allows them to detect objects more accurately and over a larger field of view than traditional scanning LiDAR. Erin dives into the technical details of Solid State Lidar, discusses the applications and industries of the technology.

Erin Bishop

Erin operates at the intersection of product management, project engineering, customer development, and product-market fit within 3D camera company Sense Photonics. Over the past several years, she has worked to initiate the market appetite for telepresence robots, indoor mobile robots, and robotic picking in warehouses. Erin has made appearances at CES, UX Week, RoboBusiness, HardwareCon, and Mobile Future Forward.

Erin creates partnerships that result in large-scale deployment of the company’s revolutionary solid-state FLASH architecture. Prior to joining Sense, she held product manager, marketing, and research roles with several robotics companies, including Industrial Perception, Inc. – which was acquired by Google in 2013 – Adept Technology, iRobot, and DEKA. Erin earned her Master’s in Mechanical Engineering from UMass Lowell, where she performed extensive research in Human-Robot Interaction.

Links

• Video of Solid State Lidar in action
• Download mp3 (23.2 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Kate Zhou interviews Farrell Helbling, postdoctoral fellow at Harvard Microrobotics lab, who has worked on developing the RoboBee, an insect-inspired robot that is the lightest vehicle to achieve untethered flight. Farrell discusses challenges with building the robot at centimeter-scale as well as integration of sensors and power electronics particularly in considerations with weight trade-offs.

Farrell Helbling

Farrell Helbling is a post doctoral fellow at Harvard University, where she focuses on the systems-level design of the Harvard RoboBee, an insect-scale flapping wing robot. Her research looks at the integration of the control system, sensors, and power electronics within the strict weight and power constraints of the vehicle. Her work led to the first untethered flight of an insect scale vehicle and was recently featured on the cover of Nature. She is the recipient of a NSF Graduate Research Fellowship, a 2018 Rising Star in EECS, and co-author on the IROS 2015 Best Student Paper for an insect-scale, hybrid aerial-aquatic vehicle. Her work on the RoboBee project is also featured at the Boston Museum of Science, World Economic Forum, London Science Museum, and the Smithsonian, as well as in the popular press (PBS NewsHour, The New York Times, Science Friday, BBC, and Wired). She is interested in the codesign of mechanical and electrical systems for mass-, power-, and computation-constrained robots.

Links

• Harvard Biorobotics Lab
• Download mp3 (17.1 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Abate interviews David Mindell, co-founder of Humatics. David discusses a system they developed that can detect the location of a special tracking device down to a centimeter level accuracy. They are currently developing a device to detect location down to a millimeter level accuracy. This solves a the core problem of localization for robots. David discusses the technology behind these products and their applications.

David Mindell

David co-founded Humatics with a mission to revolutionize how people and machines locate, navigate and collaborate. He is a professor of Aeronautics and Astronautics at MIT, as well as the Dibner Professor of the History of Engineering and Manufacturing, and Chair of the MIT Task Force on the Work of the Future. He has participated in more than 25 oceanographic expeditions and is an inventor on 8 patents in autonomous helicopters, AI-assisted piloting, and RF navigation. He is the author of five books, including Our Robots, Ourselves:Robotics and the Myths of Autonomy (2015) and Digital Apollo: Human and Machine in Spaceflight (2008). David has undergraduate degrees from Yale and a Ph.D. from MIT.

Links

• Download mp3 (15.9 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

By Nicola Nosengo

Drones can do many things, but avoiding obstacles is not their strongest suit yet – especially when they move quickly. Although many flying robots are equipped with cameras that can detect obstacles, it typically takes from 20 to 40 milliseconds for the drone to process the image and react. It may seem quick, but it is not enough to avoid a bird or another drone, or even a static obstacle when the drone itself is flying at high speed. This can be a problem when drones are used in unpredictable environments, or when there are many of them flying in the same area.

Reaction of a few milliseconds
In order to solve this problem, researchers at the University of Zurich have equipped a quadcopter (a drone with four propellers) with special cameras and algorithms that reduced its reaction time down to a few milliseconds – enough to avoid a ball thrown at it from a short distance. The results, published in Science Robotics, can make drones more effective in situations such as the aftermath of a natural disaster. The work was funded by the Swiss National Science Foundation through the National Center of Competence in Research (NCCR) Robotics.

“For search and rescue applications, such as after an earthquake, time is very critical, so we need drones that can navigate as fast as possible in order to accomplish more within their limited battery life” explains Davide Scaramuzza, who leads the Robotics and Perception Group at the University of Zurich as well as the NCCR Robotics Search and Rescue Grand Challenge. “However, by navigating fast drones are also more exposed to the risk of colliding with obstacles, and even more if these are moving. We realised that a novel type of camera, called Event Camera, are a perfect fit for this purpose”.

Event cameras have smart pixels
Traditional video cameras, such as the ones found in every smartphone, work by regularly taking snapshots of the whole scene. This is done by exposing the pixels of the image all at the same time. This way, though, a moving object can only be detected after all the pixels have been analysed by the on-board computer. Event cameras, on the other hand, have smart pixels that work independently of each other. The pixels that detect no changes remain silent, while the ones that see a change in light intensity immediately send out the information. This means that only a tiny fraction of the all pixels of the image will need to be processed by the onboard computer, therefore speeding up the computation a lot.

Event cameras are a recent innovation, and existing object-detection algorithms for drones do not work well with them. So the researchers had to invent their own algorithms that collect all the events recorded by the camera over a very short time, then subtracts the effect of the drone’s own movement – which typically account for most of the changes in what the camera sees.

Only 3.5 milliseconds to detect incoming objects
Scaramuzza and his team first tested the cameras and algorithms alone. They threw objects of various shapes and sizes towards the camera, and measured how efficient the algorithm was in detecting them. The success rate varied between 81 and 97 per cent, depending on the size of the object and the distance of the throw, and the system only took 3.5 milliseconds to detect incoming objects.

Then the most serious test began: putting cameras on an actual drone, flying it both indoor and outdoor and throwing objects directly at it. The drone was able to avoid the objects – including a ball thrown from a 3-meter distance and travelling at 10 meters per second – more than 90 per cent of the time. When the drone “knew” the size of the object in advance, one camera was enough. When, instead, it had to face objects of varying size, two cameras were used to give it stereoscopic vision.

According to Scaramuzza, these results show that event cameras can increase the speed at which drones can navigate by up to ten times, thus expanding their possible applications. “One day drones will be used for a large variety of applications, such as delivery of goods, transportation of people, aerial filmography and, of course, search and rescue,” he says. “But enabling robots to perceive and make decision faster can be a game changer for also for other domains where reliably detecting incoming obstacles plays a crucial role, such as automotive, good delivery, transportation, mining, and remote inspection with robots”.

Nearly as reliable as human pilots
In the future, the team aims to test this system on an even more agile quadrotor. “Our ultimate goal is to make one day autonomous drones navigate as good as human drone pilots. Currently, in all search and rescue applications where drones are involved, the human is actually in control. If we could have autonomous drones navigate as reliable as human pilots we would then be able to use them for missions that fall beyond line of sight or beyond the reach of the remote control”, says Davide Falanga, the PhD student who is the primary author of the article.

In this episode, Shihan Lu interviews Dr. Heather Culbertson, Assistant Professor in the Computer Science Department at the University of Southern California, about her work in haptics. Dr. Culbertson discusses the data-driven realistic texture modeling and rendering, haptic technologies in the social touch, the combination of haptics and robots, expectations and obstacles of haptics in the next 5 years.

Heather Culbertson

Heather Culbertson is a Gabilan Assistant Professor of Computer Science at the University of Southern California. Her research focuses on the design and control of haptic devices and rendering systems, human-robot interaction, and virtual reality. Particularly she is interested in creating haptic interactions that are natural and realistically mimic the touch sensations experienced during interactions with the physical world. Previously, she was a research scientist in the Department of Mechanical Engineering at Stanford University where she worked in the Collaborative Haptics and Robotics in Medicine (CHARM) Lab. She received her PhD in the Department of Mechanical Engineering and Applied Mechanics (MEAM) at the University of Pennsylvania in 2015 working in the Haptics Group, part of the General Robotics, Automation, Sensing and Perception (GRASP) Laboratory.

Links

• Download mp3 (14.8 MB)
• Culbertson’s research website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, our interviewer Audrow Nash speaks to Gil Weinberg, Professor in Georgia Tech’s School of Music and the founding director of the Georgia Tech Center for Music Technology. Weinberg leads a research lab called the Robotic Musicianship group, which focuses on developing artificial creativity and musical expression for robots and on augmented humans. Weinberg discusses several of his improvisational robots and how they work, including Shimon, a multi-armed robot marimba player, as well as his work in prosthetic devices for musicians.

Below is a video that includes Shimon on marimba, Jason Barnes playing drums with a prostetic, and Prof. Gil Weinberg on bass guitar.

Gil Weinberg

Gil Weinberg is a professor in Georgia Tech’s School of Music and the founding director of the Georgia Tech Center for Music Technology, where he leads the Robotic Musicianship group. His research focuses on developing artificial creativity and musical expression for robots and augmented humans. Among his projects are a marimba playing robotic musician called Shimon that uses machine learning for jazz improvisation, and a prosthetic robotic arm for amputees that restores and enhances human drumming abilities. Weinberg has presented his work worldwide in venues such as The Kennedy Center, The World Economic Forum, Ars Electronica, Smithsonian Cooper-Hewitt Museum, SIGGRAPH, TED-Ed, DLD and others. His music has been performed with orchestras such as Deutsches Symphonie-Orchester Berlin, the National Irish Symphony Orchestra, and the Scottish BBC Symphony while his research has been disseminated through numerous journal articles and patents. Weinberg received his M.S. and Ph.D. in Media Arts and Sciences from MIT and his B.A. from the interdisciplinary program for fostering excellence in Tel Aviv University.

Links

• Download mp3 (15.0 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

You can learn more about the urban pest control project here:

This type of prevention is designed to help technicians and companies have better efficiency and control and a faster response, when it comes to controlling rodent pests.

“The project uses a mobile autonomous robotic platform with a robot arm to introduce a camera into the trap. It captures an image that is uploaded to the cloud”.

“The project is in collaboration with Robotnik, which is responsible for the assembly of the robot; and Hispavista, which is in charge of the cloud part”, Marco Lorenzo adds.

Aritz Zabaleta, a Systems Technician at Hispavista Labs explains that the application consists of two components:

“The first manages the entire fleet of robots that communicate with the server in the cloud and it processes the information collected by them. The second component is the one that allows customers to access processed images”.

If you want to watch more fascinating robotics videos, you can explore ROBOTT-NET’s pilot projects on our YouTube-channel.

In this episode, Ron Vanderkley interviews Mark Pivac, Chief Technical Officer and co-founder of FBR (formerly Fastbrick Robotics) about the world’s first end-to-end autonomous bricklaying robot, ‘Hadrian X’. Three years after his first interview, we catch up with Pivac to see how FBR has expanded its operation and chat about their latest commercial prototype, ‘Hadrian X’, as well as the future of the robotic construction industry.

Mark Pivac is the primary inventor of FBR’s automated bricklaying technology. He is an aeronautical and mechanical engineer with over 25 years’ experience working on the development of high technology equipment ranging from lightweight aircraft to heavy off-road equipment. He has 20 years’ experience working with pro/engineer 3D CAD software as well as high-level mathematics, including matrix mathematics, robot transformations and vector mathematics for machine motion. Mark has also worked extensively with design, commissioning and fault finding on servo controlled motion systems achieving very high dynamic performance.

Links

• FBR Website
• Marc Pivac Profile Page
• Hadrian X in action (Video)
• Download mp3 (13.3 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

Source: CC0

In this episode, Audrow Nash interviews Zhuoran Zhang, PhD student at the University of Toronto, about how robots can be used to assist in artificial insemination. Zhang discusses how precise robotic manipulators can be used to extract a single sperm and how sperm can be evaluated for fitness using computer vision. Zhang also discusses his future plans.

Links

• Download mp3 (17.0 MB)
• Advanced Micro and Nano Systems Laboratory
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Audrow Nash interviews Dor Skuler, CEO and co-founder of Intuition Robotics, about a socially assistive robot for older adults named ElliQ. Skuler discusses the motivation for ElliQ, how it infers context and changes its behavior accordingly, and how ElliQ adapts its behavior over time.

Below is a video that shows what interactions with ElliQ look like.

Dor Skuler

Dor Skuler has co-founded five ventures, the most recent being Intuition Robotics. Skuler holds an MBA and Master’s of Science in Marketing from Temple University, has co-authored ‘Cloud Computing: Business Trends and Technologies’ published by Wiley in 2016 and holds board level advisory and director roles for several telecoms, cyber security and tech-led social impact ventures.

Links

• Download mp3 (17.1 MB)
• ElliQ’s webpage
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this interview, Audrow Nash interviews Kristoffer Richardsson, Michael Zillich, and Paulo Alvito.

Kristoffer Richardsson, Developer at Bitcraze in Sweden, speaks about small open-source unmanned aerial vehicle and different ways of localizing them.

Michael Zillich, CTO of Blue Danube Robotics in Austria, discusses a robotics platform for picking up toys, called Kenny. Zillich talks about Kenny’s design, including sensors and its robotic arm. He discusses how children can show the robot where to put specific toys, about how this large European project was coordinated, and about possibly commercializing this platform.

Paulo Alvito, CEO and CTO and Co-founder of IDMind in Portugal, discusses remote inspection robot for sewers. Alvito discusses what it means to be remotely autonomous, the design of the robot, and how these robots communicate data.

Links

• Download mp3 (17.3 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Audrow Nash interviews Alexandros Kogkas, Katie Driggs-Campbell, and Martin Karlsson about the work they presented at the 2018 International Conference on Intelligent Robots and Systems (IROS) in Madrid, Spain.

Alexandros Kogkas is a PhD Candidate at the Imperial College London and he speaks about an eye tracking framework to understand where a person is looking.  This framework can be used to understand a person’s intentions, for example to hand a surgeon the correct tool or helping a person who is paraplegic.   Kogkas discusses how the framework works, possible applications, and his future plans for this framework.

Katie Driggs-Campbell is a Post Doctoral Researcher at Stanford’s Intelligent System Laboratory and is—soon to be—an Assistant Professor at University of Illinois Urbana-Champaign (UIUC). She speaks about making inferences about the world from human actions, specifically in the context of autonomous cars.  In the work she discusses, they use a model of a human driver that they use infer what is happening in the world, for example a human using a crosswalk. Driggs-Campbell talks about how they evaluate this work.

Martin Karlsson is a PhD student at Lund University in Sweden, and he speaks about a haptic interface to mirror robotic arms that requires no force sensing.  He discusses a feedback law that allows a mirroring of forces and his future work to deal with joint friction.

Links

• Download mp3 (12.0 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Marwa ElDiwiny interview Peer Fisher, a Professor of Physical Chemistry at the University of Stuttgart and the Director of the Micro Nano and Molecular Systems Lab at the Max Planck Institute for Intelligent Systems.  Fischer discusses micro robots that has been designed to move inside of environments similar to the human body called, “micro swimmers.”  He talks about how they are fabricated, powered, and how they can move with light or “nano propellers.”  Fischer also discusses simulating human tissue and the future of micro and nano robots, including how they could be a replacement for certain surgeries.

Peer Fischer

Peer Fischer is a Professor of Physical Chemistry at the University of Stuttgart and he heads the independent Micro Nano and Molecular Systems Lab at the Max Planck Institute for Intelligent Systems in Stuttgart.

He received a BSc. degree in Physics from Imperial College London and a Ph.D. from the University of Cambridge. He was a NATO (DAAD) Postdoctoral Fellow at Cornell University, before joining the Rowland Institute at Harvard. At Harvard he held a Rowland Fellowship and directed an interdisciplinary research lab for five years. In 2009 he received an Attract Award from the Fraunhofer Society which led him to set up a photonics lab at the Fraunhofer Institute for Physical Measurement Techniques in Freiburg. In 2011 he moved his lab to the Max Planck Institute for Intelligent Systems in Stuttgart, and since 2013 he is a Professor at the University of Stuttgart.

Links

• Micro, Nano, & Molecular Systems Lab
• Micro Swimmers
• Download mp3 (13.7 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

The pilot projects

The three companies’ accomplished voucher work varies a lot and sets the basis to enlarge the array of industrial challenges tackled in the pilot work of ROBOTT-NET.

Linak’s pilot project will address some of the barriers that are still hindering the introduction of robots in High-Mix-Low-Volume (HMLV) production. This pilot will focus on the development of a modular assembly robot solution called MARS, that can be introduced in a wide-range of industries. You can check out Linak’s voucher project here:

Irabia’s pilot project UrbanPestPatrol aims to use robots for pest management in industrial facilities. The purpose of the project is to minimize the risk of a pest infestation and its resulting cost. You can learn about Irabia’s voucher project here:

https://www.youtube.com/watch?v=1zmHDOa1ErU

Lastly, Nissan will be looking into automated kitting for complex and diverse applications. This pilot will build a robotic test cell that enables Nissan Motor Manufacturting UK to develop automated kitting capability for its part families. Find out more about Nissan’s voucher project here:

ROBOTT-NET pilots help companies develop their voucher work through proof-of-concept level and accelerate it towards commercialisation. A pilot project will be a medium-scale research prototype installation, lasting for up to 18 months developing the robot technology and business case explored in the voucher stage and applying it to an industrial application demonstration at an end user’s site.

If you want to learn more about the pilot projects, you can check out ROBOTT-NETs website.

In this episode, Audrow Nash interviews Juxi Leitner, a Postdoctoral Research Fellow at QUT; and Nicholas Panitz, Ben Wilson, and James Brett, from CSIRO.

Leitner speaks about the Amazon Picking challenge, a challenge to advance the state of robotic grasping, and their robot which won the challenge in 2017. Their robot is similar to a cartesian 3D printer in form and uses either a suction cup or a pinch gripper for grabbing objects. Their robot has a depth camera and uses a digital scale to determine if an object has been picked up successfully. Leitner discusses what their team did differently from other teams that helped them win the competition.

Panitz, Wilson, and Brett speak about their hexapod robots. Their hexapods are for several purposes, such as environmental monitoring and remote inspection. They choose to use hexapods because they are statically stable. They discuss the design of their hexapods and how research works at Commonwealth Scientific and Industrial Research Organization, or CSIRO.

A video of the robot Leitner discusses, #Cartman:

An example of CSIRO’s hexapod robots for inspection:

Links

• Download mp3 (12.9 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this episode, Audrow Nash speaks with Maja Matarić, a professor at the University of Southern California and the Chief Science Officer of Embodied, about socially assistive robotics. Socially assistive robotics aims to endow robots with the ability to help people through individual non-contact assistance in convalescence, rehabilitation, training, and education. For example, a robot could help a child on the autism spectrum to connect to more neurotypical children and could help to motivate a stroke victim to follow their exercise routine for rehabilitation (see the videos below). In this interview, Matarić discusses the care gap in health care, how her work leverages research in psychology to make robots engaging, and opportunities in socially assistive robotics for entrepreneurship.

A short video about how personalized robots might act as a “social bridge” between a child on the autism spectrum and a more neurotypical child.

A short video about how a robot could assist stroke victims in their recovery.

Maja Matarić

Maja Matarić is professor and Chan Soon-Shiong chair in Computer Science Department, Neuroscience Program, and the Department of Pediatrics at the University of Southern California, founding director of the USC Robotics and Autonomous Systems Center (RASC), co-director of the USC Robotics Research Lab and Vice Dean for Research in the USC Viterbi School of Engineering. She received her PhD in Computer Science and Artificial Intelligence from MIT, MS in Computer Science from MIT, and BS in Computer Science from the University of Kansas. 

Links

• Interaction Lab
• Embodied
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon
• Download MP3 (13.7 MB)

Image: ICRA 2017

In this episode, Audrow Nash interviews several companies at the International Conference for Robotics and Automation (ICRA). ICRA is the IEEE Robotics and Automation Society’s biggest conference and one of the leading international forums for robotics researchers to present their work.

Interviews:

Howard Michel, Chief Technology Officer of UBTECH Education, on a humanoid robot for STEM (Science, Technology, Engineering, and Math) education for a range of ages. (Begins at 1m 20s.)

Li Bingbing, Software Engineer and Cofounder of Transforma in Singapore, on a robot for painting tall buildings. (Begins at 10m 25s.)

Xianbao Chen, Associate Researcher at Shanghai Jiao Tong University, on a hexapod robot for large parts machining. (Begins at 17m 5s.)

Lester Teh Chee Onn, Environmental Engineer at Advisian, on a watercraft for environmental monitoring. (Begins at 27m.)

Links

• Download mp3 (18.8 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Support us on Patreon

In this interview, Audrow Nash interviews Marco Hutter, Assistant Professor for Robotic Systems at ETH Zürich, about a quadrupedal robot designed for autonomous operation in challenging environments, called ANYmal. Hutter discusses ANYmal’s design, the ARGOS oil and gas rig inspection challenge, and the advantages and complexities of quadrupedal locomotion. 

Here is a video showing some of the highlights of ANYmal at the ARGOS Challenge.

Here is a video that shows some of the motions ANYmal is capable of.

Marco Hutter

Marco Hutter is assistant professor for Robotic Systems at ETH Zürich since 2015 and Branco Weiss Fellow since 2014. Before this, he was deputy director and group leader in the field of legged robotics at the Autonomous Systems Lab at ETH Zürich. After studying mechanical engineering, he conducted his doctoral degree in robotics at ETH with focus on design, actuation, and control of dynamic legged robotic systems. Beside his commitment within the National Centre of Competence in Research (NCCR) Digital Fabrication since October 2015 Hutter is part of the NCCR robotics and coordinator of several research projects, industrial collaborations, and international competitions (e.g. ARGOS challenge) that target the application of high-mobile autonomous vehicles in challenging environments such as for search and rescue, industrial inspection, or construction operation. His research interests lie in the development of novel machines and actuation concepts together with the underlying control, planning, and optimization algorithms for locomotion and manipulation.

Links

• Download mp3 (11.1 MB)
• Robotic Systems Lab
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

In this episode, Audrow Nash interviews Ayanna Howard, Professor at the Georgia Institute of Technology, about her work to help children with the movement disorder cerebral palsy. Howard discusses how robots and tablet can be used to “gamify” pediatric therapy. The idea is that if therapy is fun and engaging children are more likely to do it, and thus, they are more likely to see the long-term benefits of the therapy. Howard discusses how therapy is “gamified,” how a small humanoid robot is used to coach children, and how they work with pediatricians.

Ayanna Howard

Ayanna Howard, Ph.D. is Professor and Linda J. and Mark C. Smith Endowed Chair in Bioengineering in the School of Electrical and Computer Engineering at the Georgia Institute of Technology. She also holds the position of Associate Chair for Faculty Development in ECE. She received her B.S. in Engineering from Brown University, her M.S.E.E. from the University of Southern California, and her Ph.D. in Electrical Engineering from the University of Southern California.

Her area of research is centered around the concept of humanized intelligence, the process of embedding human cognitive capability into the control path of autonomous systems. This work, which addresses issues of autonomous control as well as aspects of interaction with humans and the surrounding environment, has resulted in over 200 peer-reviewed publications in a number of projects – from scientific rover navigation in glacier environments to assistive robots for the home. To date, her unique accomplishments have been highlighted through a number of awards and articles, including highlights in USA Today, Upscale, and TIME Magazine, as well as being named a MIT Technology Review top young innovator and recognized as one of the 23 most powerful women engineers in the world by Business Insider.

In 2013, she also founded Zyrobotics, which is currently licensing technology derived from her research and has released their first suite of therapy and educational products for children with differing needs. From 1993-2005, Dr. Howard was at NASA’s Jet Propulsion Laboratory, California Institute of Technology. She has also served a term as the Associate Director of Research for the Georgia Tech Institute for Robotics and Intelligent Machines and a term as Chair of the multidisciplinary Robotics Ph.D. program at Georgia Tech.

Links

• Download mp3 (21.9 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Researchers have devised a new way to monitor sleep stages without sensors attached to the body. Their device uses an advanced artificial intelligence algorithm to analyze the radio signals around the person and translate those measurements into sleep stages: light, deep, or rapid eye movement (REM). Image: Christine Daniloff/MIT

More than 50 million Americans suffer from sleep disorders, and diseases including Parkinson’s and Alzheimer’s can also disrupt sleep. Diagnosing and monitoring these conditions usually requires attaching electrodes and a variety of other sensors to patients, which can further disrupt their sleep.

To make it easier to diagnose and study sleep problems, researchers at MIT and Massachusetts General Hospital have devised a new way to monitor sleep stages without sensors attached to the body. Their device uses an advanced artificial intelligence algorithm to analyze the radio signals around the person and translate those measurements into sleep stages: light, deep, or rapid eye movement (REM).

“Imagine if your Wi-Fi router knows when you are dreaming, and can monitor whether you are having enough deep sleep, which is necessary for memory consolidation,” says Dina Katabi, the Andrew and Erna Viterbi Professor of Electrical Engineering and Computer Science, who led the study. “Our vision is developing health sensors that will disappear into the background and capture physiological signals and important health metrics, without asking the user to change her behavior in any way.”

Katabi worked on the study with Matt Bianchi, chief of the division of sleep medicine at MGH, and Tommi Jaakkola, the Thomas Siebel Professor of Electrical Engineering and Computer Science and a member of the Institute for Data, Systems, and Society at MIT. Mingmin Zhao, an MIT graduate student, is the paper’s first author, and Shichao Yue, another MIT graduate student, is also a co-author.

The researchers will present their new sensor at the International Conference on Machine Learning on Aug. 9.

Remote sensing

Katabi and members of her group in MIT’s Computer Science and Artificial Intelligence Laboratory have previously developed radio-based sensors that enable them to remotely measure vital signs and behaviors that can be indicators of health. These sensors consist of a wireless device, about the size of a laptop computer, that emits low-power radio frequency (RF) signals. As the radio waves reflect off of the body, any slight movement of the body alters the frequency of the reflected waves. Analyzing those waves can reveal vital signs such as pulse and breathing rate.

“It’s a smart Wi-Fi-like box that sits in the home and analyzes these reflections and discovers all of these changes in the body, through a signature that the body leaves on the RF signal,” Katabi says.

Katabi and her students have also used this approach to create a sensor called WiGait that can measure walking speed using wireless signals, which could help doctors predict cognitive decline, falls, certain cardiac or pulmonary diseases, or other health problems.

After developing those sensors, Katabi thought that a similar approach could also be useful for monitoring sleep, which is currently done while patients spend the night in a sleep lab hooked up to monitors such as electroencephalography (EEG) machines.

“The opportunity is very big because we don’t understand sleep well, and a high fraction of the population has sleep problems,” says Zhao. “We have this technology that, if we can make it work, can move us from a world where we do sleep studies once every few months in the sleep lab to continuous sleep studies in the home.”

To achieve that, the researchers had to come up with a way to translate their measurements of pulse, breathing rate, and movement into sleep stages. Recent advances in artificial intelligence have made it possible to train computer algorithms known as deep neural networks to extract and analyze information from complex datasets, such as the radio signals obtained from the researchers’ sensor. However, these signals have a great deal of information that is irrelevant to sleep and can be confusing to existing algorithms. The MIT researchers had to come up with a new AI algorithm based on deep neural networks, which eliminates the irrelevant information.

“The surrounding conditions introduce a lot of unwanted variation in what you measure. The novelty lies in preserving the sleep signal while removing the rest,” says Jaakkola. Their algorithm can be used in different locations and with different people, without any calibration.

Using this approach in tests of 25 healthy volunteers, the researchers found that their technique was about 80 percent accurate, which is comparable to the accuracy of ratings determined by sleep specialists based on EEG measurements.

“Our device allows you not only to remove all of these sensors that you put on the person, and make it a much better experience that can be done at home, it also makes the job of the doctor and the sleep technologist much easier,” Katabi says. “They don’t have to go through the data and manually label it.”

Sleep deficiencies

Other researchers have tried to use radio signals to monitor sleep, but these systems are accurate only 65 percent of the time and mainly determine whether a person is awake or asleep, not what sleep stage they are in. Katabi and her colleagues were able to improve on that by training their algorithm to ignore wireless signals that bounce off of other objects in the room and include only data reflected from the sleeping person.

The researchers now plan to use this technology to study how Parkinson’s disease affects sleep.

“When you think about Parkinson’s, you think about it as a movement disorder, but the disease is also associated with very complex sleep deficiencies, which are not very well understood,” Katabi says.

The sensor could also be used to learn more about sleep changes produced by Alzheimer’s disease, as well as sleep disorders such as insomnia and sleep apnea. It may also be useful for studying epileptic seizures that happen during sleep, which are usually difficult to detect.

In this episode, MeiXing Dong interviews Matthias Vanoni, co-founder and CEO of Biowatch. Vanoni speaks about Biowatch, a wrist-veins biometric reader that functions as a security solution for mobile payments and smart devices. They discuss the technical challenges of building a miniaturized wrist-vein reader and how this device changes the usual user authentication process.

Matthias Vanoni

Prior to being the CEO and co-founder of Biowatch SA, Matthias Vanoni (31) was a PhD student at EPFL and IDIAP Research Institute, specialised in biometric authentication and a former captain in French Gendarmerie. His Life is now dedicated to entrepreneurship, his wife and their little boy.

Links

• Download mp3 (14.7 MB)
• Biowatch Website
• Biowatch Video
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

A new system can automatically retouch images in the style of a professional photographer. It can run on a cellphone and display retouched images in real-time. Courtesy of the researchers (edited by MIT News)

The data captured by today’s digital cameras is often treated as the raw material of a final image. Before uploading pictures to social networking sites, even casual cellphone photographers might spend a minute or two balancing color and tuning contrast, with one of the many popular image-processing programs now available.

This week at Siggraph, the premier digital graphics conference, researchers from MIT’s Computer Science and Artificial Intelligence Laboratory and Google are presenting a new system that can automatically retouch images in the style of a professional photographer. It’s so energy-efficient, however, that it can run on a cellphone, and it’s so fast that it can display retouched images in real-time, so that the photographer can see the final version of the image while still framing the shot.

The same system can also speed up existing image-processing algorithms. In tests involving a new Google algorithm for producing high-dynamic-range images, which capture subtleties of color lost in standard digital images, the new system produced results that were visually indistinguishable from those of the algorithm in about one-tenth the time — again, fast enough for real-time display.

The system is a machine-learning system, meaning that it learns to perform tasks by analyzing training data; in this case, for each new task it learned, it was trained on thousands of pairs of images, raw and retouched.

The work builds on an earlier project from the MIT researchers, in which a cellphone would send a low-resolution version of an image to a web server. The server would send back a “transform recipe” that could be used to retouch the high-resolution version of the image on the phone, reducing bandwidth consumption.

“Google heard about the work I’d done on the transform recipe,” says Michaël Gharbi, an MIT graduate student in electrical engineering and computer science and first author on both papers. “They themselves did a follow-up on that, so we met and merged the two approaches. The idea was to do everything we were doing before but, instead of having to process everything on the cloud, to learn it. And the first goal of learning it was to speed it up.”

Short cuts

In the new work, the bulk of the image processing is performed on a low-resolution image, which drastically reduces time and energy consumption. But this introduces a new difficulty, because the color values of the individual pixels in the high-res image have to be inferred from the much coarser output of the machine-learning system.

In the past, researchers have attempted to use machine learning to learn how to “upsample” a low-res image, or increase its resolution by guessing the values of the omitted pixels. During training, the input to the system is a low-res image, and the output is a high-res image. But this doesn’t work well in practice; the low-res image just leaves out too much data.

Gharbi and his colleagues — MIT professor of electrical engineering and computer science Frédo Durand and Jiawen Chen, Jon Barron, and Sam Hasinoff of Google — address this problem with two clever tricks. The first is that the output of their machine-learning system is not an image; rather, it’s a set of simple formulae for modifying the colors of image pixels. During training, the performance of the system is judged according to how well the output formulae, when applied to the original image, approximate the retouched version.

Taking bearings

The second trick is a technique for determining how to apply those formulae to individual pixels in the high-res image. The output of the researchers’ system is a three-dimensional grid, 16 by 16 by 8. The 16-by-16 faces of the grid correspond to pixel locations in the source image; the eight layers stacked on top of them correspond to different pixel intensities. Each cell of the grid contains formulae that determine modifications of the color values of the source images.

That means that each cell of one of the grid’s 16-by-16 faces has to stand in for thousands of pixels in the high-res image. But suppose that each set of formulae corresponds to a single location at the center of its cell. Then any given high-res pixel falls within a square defined by four sets of formulae.

Roughly speaking, the modification of that pixel’s color value is a combination of the formulae at the square’s corners, weighted according to distance. A similar weighting occurs in the third dimension of the grid, the one corresponding to pixel intensity.

The researchers trained their system on a data set created by Durand’s group and Adobe Systems, the creators of Photoshop. The data set includes 5,000 images, each retouched by five different photographers. They also trained their system on thousands of pairs of images produced by the application of particular image-processing algorithms, such as the one for creating high-dynamic-range (HDR) images. The software for performing each modification takes up about as much space in memory as a single digital photo, so in principle, a cellphone could be equipped to process images in a range of styles.

Finally, the researchers compared their system’s performance to that of a machine-learning system that processed images at full resolution rather than low resolution. During processing, the full-res version needed about 12 gigabytes of memory to execute its operations; the researchers’ version needed about 100 megabytes, or one-hundredth as much. The full-resolution version of the HDR system took about 10 times as long to produce an image as the original algorithm, or 100 times as long as the researchers’ system.

“This technology has the potential to be very useful for real-time image enhancement on mobile platforms,” says Barron. “Using machine learning for computational photography is an exciting prospect but is limited by the severe computational and power constraints of mobile phones. This paper may provide us with a way to sidestep these issues and produce new, compelling, real-time photographic experiences without draining your battery or giving you a laggy viewfinder experience.”

In recent years engineers have been developing new technologies to enable robots and humans to move faster and jump higher. Soft, elastic materials store energy in these devices, which, if released carefully, enable elegant dynamic motions. Robots leap over obstacles and prosthetics empower sprinting. A fundamental challenge remains in developing these technologies. Scientists spend long hours building and testing prototypes that can reliably move in specific ways so that, for example, a robot lands right-side up upon landing a jump.

A pair of new computational methods developed by a team of researchers from Massachusetts Institute of Technology (MIT), University of Toronto and Adobe Research takes first steps towards automating the design of the dynamic mechanisms behind these movements. Their methods generate simulations that match the real-world behaviors of flexible devices at rates 70-times faster than previously possible and provide critical improvements in the accuracy of simulated collisions and rebounds. These methods are then both fast and accurate enough to be used to automate the design process used to create dynamic mechanisms for controlled jumping.

The team will present their methods and results from their paper, “Dynamics-Aware Numerical Coarsening for Fabrication Design,” at the SIGGRAPH 2017 conference in Los Angeles, 30 July to 3 August. SIGGRAPH spotlights the most innovative results in computer graphics research and interactive techniques worldwide.

“This research is pioneering work in applying computer graphics techniques to real physical objects with dynamic behavior and contact,” says lead author Desai Chen, a PhD candidate at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). “The techniques we’ve developed open the door to automating the design of highly dynamic, fast-moving objects.”

Chen’s co-authors include David I.W. Levin, assistant professor at the University of Toronto; Wojciech Matusik, associate professor of electrical engineering and computer science at MIT; and Danny M. Kaufman, senior research scientist at Adobe Research.

Major advances in computational design, physical modeling and rapid manufacturing have enabled the fabrication of objects with customized physical properties–such as tailored sneakers, complex prosthetics, and soft robots–while computer graphics research has seen rapid improvements and efficiencies in creating compelling animations of physics for games, virtual reality and film. In this new work, the team aims to combine efficiency and accuracy to enable simulation for design fabrication, and to accurately simulate objects in motion.

“The goal is to bring the physical rules of virtual reality much closer to those of actual reality,” says Levin.

In the research, the team addresses the challenge with simulating elastic objects as they collide – making things accurate enough to match reality and fast enough to automate that design process. Attempting to create such simulations in the presence of contact, impact or friction remains time-consuming and inaccurate.

“It is very important to get this part right, and, until now, our existing computer codes tend to break down here,” says Kaufman. “We realize that if we are doing design for the real world, we have to have code that correctly models things such as high-speed bouncing, collision and friction.”

The researchers demonstrate their new methods, Dynamics-Aware Coarsening (DAC) and Boundary Balanced Impact (BBI), by designing and fabricating mechanisms that flip, throw and jump over obstacles. Their methods perform simulations much faster than existing, state-of-the-art approaches and with greater accuracy when compared to real-world motions.

DAC works by reducing degrees of freedom, the number of values that encode motion, to speed up simulations while still capturing important motions for dynamic scenarios. It finds the roughest meshes that can correctly represent the key shapes that will be taken by dynamics and matches the material properties of these meshes directly to recorded video experiment. BBI is a method for modeling impact behavior of elastic objects. It uses material properties to smoothly project velocities near impact sites to model many real world impact situations such as the impact and rebound between a soft printed material and a table, for instance.

The team was inspired by the need for faster, more accurate design tools that can capture accurate simulations of elastic objects undergoing deformation and collision – especially at high-speeds. These new methods could, down the road, be applied to robotics design, developing robots as they increasingly take on human-like movements and characteristics.

“This project is really a first step for us in pushing methods for simulating reality,” says Kaufman. “We are focusing on pushing them for automatic design and exploring how to effectively use them in design. We can create beautiful images in computer graphics and in animation, let’s extend these methods to actual objects in the real world that are useful, beautiful and efficient.”

A magnetic folding robot arm can grasp and bend thanks to its pattern of origami-inspired folds and a wireless electromagnetic field. Credit: Wyss Institute at Harvard University

The traditional Japanese art of origami transforms a simple sheet of paper into complex, three-dimensional shapes through a very specific pattern of folds, creases, and crimps. Folding robots based on that principle have emerged as an exciting new frontier of robotic design, but generally require onboard batteries or a wired connection to a power source, making them bulkier and clunkier than their paper inspiration and limiting their functionality.

A team of researchers at the Wyss Institute for Biologically Inspired Engineering and the John A. Paulson School of Engineering and Applied Sciences (SEAS) at Harvard University has created battery-free folding robots that are capable of complex, repeatable movements powered and controlled through a wireless magnetic field.

“Like origami, one of the main points of our design is simplicity,” says co-author Je-sung Koh, Ph.D., who conducted the research as a Postdoctoral Fellow at the Wyss Institute and SEAS and is now an Assistant Professor at Ajou University in South Korea. “This system requires only basic, passive electronic components on the robot to deliver an electric current—the structure of the robot itself takes care of the rest.”

The research team’s robots are flat and thin (resembling the paper on which they’re based) plastic tetrahedrons, with the three outer triangles connected to the central triangle by hinges, and a small circuit on the central triangle. Attached to the hinges are coils made of a type of metal called shape-memory alloy (SMA) that can recover its original shape after deformation by being heated to a certain temperature. When the robot’s hinges lie flat, the SMA coils are stretched out in their “deformed” state; when an electric current is passed through the circuit and the coils heat up, they spring back to their original, relaxed state, contracting like tiny muscles and folding the robots’ outer triangles in toward the center. When the current stops, the SMA coils are stretched back out due to the stiffness of the flexure hinge, thus lowering the outer triangles back down.

The power that creates the electrical current needed for the robots’ movement is delivered wirelessly using electromagnetic power transmission, the same technology inside wireless charging pads that recharge the batteries in cell phones and other small electronics. An external coil with its own power source generates a magnetic field, which induces a current in the circuits in the robot, thus heating the SMA coils and inducing folding. In order to control which coils contract, the team built a resonator into each coil unit and tuned it to respond only to a very specific electromagnetic frequency. By changing the frequency of the external magnetic field, they were able to induce each SMA coil to contract independently from the others.

“Not only are our robots’ folding motions repeatable, we can control when and where they happen, which enables more complex movements,” explains lead author Mustafa Boyvat, Ph.D., also a Postdoctoral Fellow at the Wyss Institute and SEAS.

Just like the muscles in the human body, the SMA coils can only contract and relax: it’s the structure of the body of the robot — the origami “joints” — that translates those contractions into specific movements. To demonstrate this capability, the team built a small robotic arm capable of bending to the left and right, as well as opening and closing a gripper around an object. The arm is constructed with a special origami-like pattern to permit it to bend when force is applied, and two SMA coils deliver that force when activated while a third coil pulls the gripper open. By changing the frequency of the magnetic field generated by the external coil, the team was able to control the robot’s bending and gripping motions independently.

There are many applications for this kind of minimalist robotic technology; for example, rather than having an uncomfortable endoscope put down their throat to assist a doctor with surgery, a patient could just swallow a micro-robot that could move around and perform simple tasks, like holding tissue or filming, powered by a coil outside their body. Using a much larger source coil — on the order of yards in diameter — could enable wireless, battery-free communication between multiple “smart” objects in an entire home. The team built a variety of robots — from a quarter-sized flat tetrahedral robot to a hand-sized ship robot made of folded paper — to show that their technology can accommodate a variety of circuit designs and successfully scale for devices large and small. “There is still room for miniaturization. We don’t think we went to the limit of how small these can be, and we’re excited to further develop our designs for biomedical applications,” Boyvat says.

“When people make micro-robots, the question is always asked, ‘How can you put a battery on a robot that small?’ This technology gives a great answer to that question by turning it on its head: you don’t need to put a battery on it, you can power it in a different way,” says corresponding author Rob Wood, Ph.D., a Core Faculty member at the Wyss Institute who co-leads its Bioinspired Robotics Platform and the Charles River Professor of Engineering and Applied Sciences at SEAS.

“Medical devices today are commonly limited by the size of the batteries that power them, whereas these remotely powered origami robots can break through that size barrier and potentially offer entirely new, minimally invasive approaches for medicine and surgery in the future,” says Wyss Founding Director Donald Ingber, who is also the Judah Folkman Professor of Vascular Biology at Harvard Medical School and the Vascular Biology Program at Boston Children’s Hospital, as well as a Professor of Bioengineering at Harvard’s School of Engineering and Applied Sciences.

The research paper was published in Science Robotics.

Update: The response to Tertill’s crowdfunding campaign has amazed and delighted us! Pledges totalling over $250,000 have come from 1000+ backers. We’re shipping to all countries, with over a fifth of Tertill’s supporters coming from outside the United States. But the end is near; Tuesday (11 July) is the last full day of the campaign. After that Tertill’s discounted campaign price will no longer be available and delivery in time for next year’s (northern hemisphere) growing season cannot be assured.

Franklin Robotics has launched a Kickstarter campaign for Tertill, their solar-powered, garden-weeding robot.

Tertill lives in your garden, collecting sunlight to power its weed patrol, and cutting down short plants with a string trimmer/weed whacker with almost no intervention required. Available for about $300USD, the fully autonomous Tertill is the first weeding robot available to home gardeners.

Tertill is round, short, has four wheel drive and extreme camber wheels. It uses proprietary algorithms to ensure that it finds as many weeds as it can, using its sensors to distinguish between weeds and crops based on height. As Tertill sees it: small plants are weeds, big plants are crops. But what if you have seedlings? Simply place a plant collar around seedlings to inform Tertill that this plant is wanted.

Tertill doesn’t use a camera for weed discrimination because Franklin wants to put Tertill to work in your garden now. In the not-too-distant future, inexpensive vision systems may reliably distinguish between plants and crops even when both are seedlings. But, according to Franklin, such system are not yet ready.

Tertill is designed such that one robot can easily keep up with the weeds in a typical sized garden (that’s about 100 square feet in the USA). Tertill will usually handle gardens considerably larger than this, but how much larger depends on weather and local weed types. And Terrill’s can work in teams, coexisting harmoniously in the same space. Multiple Tertill robots will simply avoid each other.

Check out the video below:

But Tertill needs a small barrier to keep it from wandering off beyond the borders of your garden. A garden fence, the wooden border of a raised bed, or any sort of edging that is two or more inches tall will work.

Otherwise, Tertill is designed to live in your garden and require very little attention. Summer showers won’t bother it. And, under normal conditions, you won’t have to worry about charging Tertill—the sun will see to that. But, if Tertill has been in the dark for so long (months) that its battery is completely exhausted, a USB connection can be used to charge the battery.

Franklin believes that small, simple robots can help solve big, complex problems. Franklin’s robots aim to help gardeners by reducing the tedium and physical challenge of weeding, making gardening more fun. The data their robots collect helps gardeners garden better, improving yield and quality. Their robots aim to benefit the environment by eliminating the need for herbicides and returning organic matter to the soil.

Click here to visit Franklin’s website and learn more.

Crops are key for a sustainable food production and we face several challenges in crop production. First, we need to feed a growing world population. Second, our society demands high-quality foods. Third, we have to reduce the amount agrochemicals that we apply to our fields as it directly affects our ecosystem. Precision farming techniques offer a great potential to address these challenges, but we have to acquire and provide the relevant information about the field status to the farmers such that specific actions can be taken.

This paper won the IEEE Robotics & Automation Best Automation Paper Award at ICRA 2017.

Modern approaches as agriculture system management and smart farming typically require detailed knowledge about the current field status. For successful on-field intervention, it is key to know the type and spatial distribution of the crops and weeds on the field at an early state. The earlier mechanical or chemical weeding actions are executed, the higher the chances for obtaining a high yield. A prerequisite to trigger weeding and intervention task is a detailed knowledge about the spread of weeds. Therefore, it is important to have an automatic perception pipeline that monitors and analyzes the crops automatically and provides report about the status on the field.

Fig. 1 illustrates the input and output of our approach. We use a comparably cheap, out-of-the-box UAV system to capture images of a field and compute a class label to each pixel, i.e., determine if that pixel belongs a crop or a weed. Through the use of GNSS and bundle adjustment procedures, we can obtain the spatial distribution of the detected crops and weeds in a global reference frame.

Use Case 1: Farmer Support: In order to reduce the investments on herbicides, pesticides, and fertilizers by keeping or even increasing the yield, farmers must meet the following key questions: Where are weeds in the field? as well as where do the crops grow abnormally? Given the knowledge about the plant specific spatial distribution as illustrated in Fig. 2, farmers can decide, which and how much agro-chemicals are needed to address their specific needs. Furthermore, they can perform individual actions at specific locations in the field.

Figure 1: Low-cost DJI Phantom 4 UAV used for field monitoring (left) as well as an example image analyzed by our approach (right). Here, green refers to detected sugar beets, which we consider as crop, and purple refers to weeds.

Use Case 2: Autonomous Robots: In the future, autonomous robotic systems are likely to be used to manage arable farm land. Robots have the potential to survey the current field status on a continuous time scale and perform specific tasks such as weed control on a per-plant basis only in regions on a field where an action is actually needed. This work is part of the FLOURISH project (flourishproject.eu) were we are developing teams of autonomously and collaboratively operating UAVs and UGVs. Here, the UAVs offer excellent survey capabilities at low cost and can be used to analyze the field status. The analysis of crop and weed growth on the field is an essential building block to infer target locations for a targeted ground intervention. Thus, the UAV can send an UGV to those locations to execute interventions task.

Approach

We address the problem of analyzing UAV imagery to inspect the status of a field in terms of weed types and spatial crop and weed distribution. We focus on a detection on a per-plant basis to estimate the amount of crops as well as various weed species to provide this information to the farmers or as a part of an autonomous robotic perception system. In our work, we propose a vision-based classification system for identifying crops and weeds in both, RGB-only as well as RGB combined with near infra-red (NIR) imagery of agricultural fields captured by an UAV. Our perception pipeline is capable of detecting plants in such images only based on their appearance. Our approach is also able to exploit the geometry of the plant arrangement without requiring a known spatial distribution to be specified explicitly. Optionally, we can utilize prior information regarding the crop arrangement like crop rows. In addition to that it can deal with vegetation, which is located within the intra-row space.

Figure 2: Analyzed field status by our approach. Based on these informations the farmers can schedule their next individual actions and estimate the needed amount of agro-chemicals for their field.

Processing Pipeline

The three key steps of our systems are the following: First, we apply a preprocessing in order to normalize intensity values on a global scale and detect the vegetation in each image. Second, we extract features only for regions, which correspond to vegetation, exploiting a combination of an object-based and a keypoint based approach. We compute around 550 visual features containing intensity statistics, texture and shape information. In addition to that, we exploit geometric features, which encode spatial relationships between the vegetation as well as optional crop row features, which consider the distance from the vegetation to estimated crop rows. Third, we apply a multi-class Random Forest classification and obtain a probability distribution for the predicted class labels.

Figure 3: Example image captured by a DJI Phantom 4 at low altitude at different levels of progress within the classification pipeline. From left to right: normalized RGB image, computed Excess Green Index (ExG), vegetation mask that reflect the spatial distribution of the vegetation and multi-class classification output of our plant/weed classification system

Fig. 3 illustrates the key steps of our pipeline and Fig. 4 depicts an example of the detected crop rows. The papers [1] and [2] provide a detailed description of the pipeline as well as the features we are using for the classification.

Results and Conclusion

Figure 4: Detected set of crop rows (red) estimated based on Hough transform followed by a search algorithm in order to find the best set of parallel lines, which is supported by the vegetation. Example result from a sugar beet field with high weed pressure.

We implemented and tested our approach using different types of cameras and UAVs, on real sugar beet fields. Our experiments suggest that our proposed system is able to perform a classification of sugar beets and different weed types in RGB images captured by a commercial low cost UAV system. We show that the classification system achieves good performance in terms of overall accuracy of up to 96% even in images where neither a row detection of the crops nor a regular pattern of the plantation can be exploited for geometry-based detection. Furthermore, we evaluate our system using RGB+NIR images captured with a comparable expensive camera system to those obtained using the standard equipped RGB camera mounted on a consumer quad-copter (DJI Phantom 4 and camera for around $1,500). Fig. 5 depicts example results achieved on the different datasets respectively.

With our approach, we enable farmers to analyze the field status in terms of the spatial distribution of the crops and weeds on their field by using a low-cost UAV system, which can be used out-of-the-box. Typically, these systems are easy to use without requiring expertise on flying or aerial dynamics. The farmers only has to provide their expertise in terms of labeling into the perception pipeline. The farmers are predestined to perform this task because nobody knows better, which plants and weed types commonly appear on their specific fields. Compared to the area of the field only a small amount of data has to be labeled for analyzing the whole field. In terms of the usage withing autonomous robotic systems in agriculture this approach can be used as part of the perception pipeline of the system. Based on the classification results of the images taken by the UAV a program for decision making or planning can execute different task to achieve the destined goals, e.g. to send the UGV to a target location.

Figure 5: Zoomed view of images analyzed by our plant/weed classification system (left column) and corresponding ground truth images (right column). Top row: Multi-class results for the dataset captured by the DJI Phantom 4 UAV. Here, the crops do not follow a row structure. Thus, the results are based on the visual as well as geometric features. Center row: Result for the dataset captured with an 4-channel RGB+NIR camera system. Bottom row: Result for the dataset captured with a DJI Matrice 100 UAV. Arrows point to weeds detected in intra-row space. Sugar beet (green), chamomile (blue), saltbush (yellow) and other weeds (red).

Spiders are truly amazing creatures. They have evolved over more than 200 million years and can be found in almost every corner of our planet. They are one of the most successful animals. Not less impressive are their webs, highly intricate structures that have been optimised through evolution over approximately 100 million years with the ultimate purpose of catching prey.

However, interestingly, the closer you look at spiders’ webs the more details you can observe and the structures are much more complicated than one would expect from a simple snare. They are made of a variety of different types of silks, use water droplets to keep the tension [see: citation 4], and the structure is highly dynamic [see: citation 4]. Spider’s webs have a great deal more morphological complexity than what you would need to simply catch flies.

Since nature typically does not spoil resources the question arises: why are spiders’ webs so complex? Might they have other functionalities besides being a simple trap? One of the most interesting answers to this question is that spiders might use their webs as computational devices.

How does the spider use the web as a computer?

Despite the fact that most spiders have a lot of eyes (the majority has 8, but some have even up to 12), a lot of the spiders have bad sight. In order to understand what is going on in their webs, they use mechanoreceptors in their legs (called lyriforms) to “listen” to vibrations in the web.  Different species of spiders have different preferred places to sit and observe. While some can be found right at the center, others prefer to sit outside the actual web and to listen to one single thread. It is quite remarkable that based only on the information that comes through this single thread the spider seems to be able to deduce what is going on in their web and where these events are taking place.

For example, they need to know if there is a prey, like a fly, entangled in their web. Or if the vibrations are coming from a dangerous insect like a wasp and they should stay away. The web is also used to communicate with potential mates and the spider even excites the web and listens to the echo. This might be a way for the spider to check if threads are broken or if the tension in the web has to be increased.

From a computational point of view, the spider needs to classify different vibration pattern (e.g., prey vs predator vs mate) and to locate its origin (i.e., where the vibration started).

One way to understand how a spider’s web could help to carry out this computational functionality is the concept of morphological computation. This is a term that describes the understanding that mechanical structures all over in nature are carrying out useful computations. For example, they help to stabilise running, facilitate sensory data processing, and helps animals and plants to interact with complex and unpredictable environments.

One could say computation is outsourced to the physical body (e.g., from the brain to another part of the body).

From this point of view, the spider’s web can be seen as a nonlinear, dynamic filter. It can be understood as some kind of pre-processing unit that makes it easier for the animal to interpret the vibration signals. The web’s dynamic properties and its complex morphological structure mix vibration signals in a nonlinear fashion. It even has some memory. This can be easily seen by pinching the web. It responds with vibrations for some moments after the impact echoing the original input.  The web can also damp unwanted frequencies, which is crucial to get rid of noise. On the other hand, it might even be able to highlight other signals at certain frequencies that carry more relevant information about the events taking place on the web.

These are all useful computations and they make it easier for the spider to “read” and understand the vibration patterns. As a result, the brain of the animal has to do less work and it can concentrate on other tasks. In effect, the spider seems to devolve computation to the web.  This might be also the reason why spiders tend to their webs so intensively. They constantly observe it and adapt the tension if it has changed, e.g. due to change in humidity, and repair it as soon a thread is broken.

From spider webs to sensors

People have speculated for a while that spider webs might have additional functionalities. A great article that discusses that is “The Thoughts of a Spiderweb“.

However, nobody so far has systematically looked into the actual computational capabilities of the web. This is about to change. We recently started a Leverhulme Trust Research project that will investigate naturally spun spider webs of different species to understand how and which computing might take place in these structures.  Moreover, the project does not only try to understand the underlying computational principles but will also develop morphological computation-based sensor technology to measure flow and vibrations.

The project combines our research expertise in Morphological Computation at the University of Bristol and the expertise on spider webs at the Silk Group in Oxford.

In experimental setups we will use solenoids and laser Doppler vibrometers to measure vibrations in the web with very high precision. The goal is to understand how computation is carried out. We will systematically investigate how filtering capabilities, memory, and signal integration can happen in such structures. In parallel, we will develop a general simulation environment for vibrating structures. We will use this to ask specific questions about how different shapes and materials others than spider webs and silk can help to carry out computations. In addition, we will develop real prototypes of vibration and flow sensors, which will be inspired by these findings. It’s very likely they will look different from spider webs and they will use various types of materials.

Such sensors can be used in various applications. For example, morphological computation based flow sensors could be used to detect anomalies in the flow in tubes. Or vibration sensors put at strategic places on buildings could be able to detect earthquakes or structural failure. Also highly dynamic machines, for example, a wind turbine, could be monitored by such sensors to predict failure.

Ultimately, the project will provide not only a new technology to build sensors, but we hope also to get a fundamental understanding how spiders use their webs for computation.

References

[1] Hauser, H.; Ijspeert, A.; Füchslin, R.; Pfeifer, R. & Maass, W.”Towards a theoretical foundation for morphological computation with compliant bodies.”Biological Cybernetics, Springer Berlin / Heidelberg, 2011, 105, 355-370

[2]  Hauser, H.; Ijspeert, A.; Füchslin, R.; Pfeifer, R. & Maass, W. “The role of feedback in morphological computation with compliant bodies”. Biological Cybernetics, Springer Berlin / Heidelberg, 2012, 106, 595-613

[3] Hauser, H.; Füchslin, R.M.; Nakajima, K.“Morphological Computation – The Physical Body as a Computational Resource” Opinions and Outlooks on Morphological Computation, editors Hauser, H.; Füchslin, R.M. and Pfeifer, R., Chapter 20, pp 226-244,  2014, ISBN 978-3-033-04515-6

[4] Mortimer, B., Gordon, S. D., Holland, C., Siviour, C. R., Vollrath, F. and Windmill, J. F. C. (2014), The Speed of Sound in Silk: Linking Material Performance to Biological Function. Adv. Mater., 26: 5179–5183. doi:10.1002/adma.201401027

By Eran Klein, University of Washington and Katherine Pratt, University of Washington

In the 1995 film “Batman Forever,” the Riddler used 3-D television to secretly access viewers’ most personal thoughts in his hunt for Batman’s true identity. By 2011, the metrics company Nielsen had acquired Neurofocus and had created a “consumer neuroscience” division that uses integrated conscious and unconscious data to track customer decision-making habits. What was once a nefarious scheme in a Hollywood blockbuster seems poised to become a reality.

Recent announcements by Elon Musk and Facebook about brain-computer interface (BCI) technology are just the latest headlines in an ongoing science-fiction-becomes-reality story.

BCIs use brain signals to control objects in the outside world. They’re a potentially world-changing innovation – imagine being paralyzed but able to “reach” for something with a prosthetic arm just by thinking about it. But the revolutionary technology also raises concerns. Here at the University of Washington’s Center for Sensorimotor Neural Engineering (CSNE) we and our colleagues are researching BCI technology – and a crucial part of that includes working on issues such as neuroethics and neural security. Ethicists and engineers are working together to understand and quantify risks and develop ways to protect the public now.

Picking up on P300 signals

All BCI technology relies on being able to collect information from a brain that a device can then use or act on in some way. There are numerous places from which signals can be recorded, as well as infinite ways the data can be analyzed, so there are many possibilities for how a BCI can be used.

Some BCI researchers zero in on one particular kind of regularly occurring brain signal that alerts us to important changes in our environment. Neuroscientists call these signals “event-related potentials.” In the lab, they help us identify a reaction to a stimulus.

In particular, we capitalize on one of these specific signals, called the P300. It’s a positive peak of electricity that occurs toward the back of the head about 300 milliseconds after the stimulus is shown. The P300 alerts the rest of your brain to an “oddball” that stands out from the rest of what’s around you.

For example, you don’t stop and stare at each person’s face when you’re searching for your friend at the park. Instead, if we were recording your brain signals as you scanned the crowd, there would be a detectable P300 response when you saw someone who could be your friend. The P300 carries an unconscious message alerting you to something important that deserves attention. These signals are part of a still unknown brain pathway that aids in detection and focusing attention.

Reading your mind using P300s

P300s reliably occur any time you notice something rare or disjointed, like when you find the shirt you were looking for in your closet or your car in a parking lot. Researchers can use the P300 in an experimental setting to determine what is important or relevant to you. That’s led to the creation of devices like spellers that allow paralyzed individuals to type using their thoughts, one character at a time.

It also can be used to determine what you know, in what’s called a “guilty knowledge test.” In the lab, subjects are asked to choose an item to “steal” or hide, and are then shown many images repeatedly of both unrelated and related items. For instance, subjects choose between a watch and a necklace, and are then shown typical items from a jewelry box; a P300 appears when the subject is presented with the image of the item he took.

Everyone’s P300 is unique. In order to know what they’re looking for, researchers need “training” data. These are previously obtained brain signal recordings that researchers are confident contain P300s; they’re then used to calibrate the system. Since the test measures an unconscious neural signal that you don’t even know you have, can you fool it? Maybe, if you know that you’re being probed and what the stimuli are.

Techniques like these are still considered unreliable and unproven, and thus U.S. courts have resisted admitting P300 data as evidence.

Imagine that instead of using a P300 signal to solve the mystery of a “stolen” item in the lab, someone used this technology to extract information about what month you were born or which bank you use – without your telling them. Our research group has collected data suggesting this is possible. Just using an individual’s brain activity – specifically, their P300 response – we could determine a subject’s preferences for things like favorite coffee brand or favorite sports.

But we could do it only when subject-specific training data were available. What if we could figure out someone’s preferences without previous knowledge of their brain signal patterns? Without the need for training, users could simply put on a device and go, skipping the step of loading a personal training profile or spending time in calibration. Research on trained and untrained devices is the subject of continuing experiments at the University of Washington and elsewhere.

It’s when the technology is able to “read” someone’s mind who isn’t actively cooperating that ethical issues become particularly pressing. After all, we willingly trade bits of our privacy all the time – when we open our mouths to have conversations or use GPS devices that allow companies to collect data about us. But in these cases we consent to sharing what’s in our minds. The difference with next-generation P300 technology under development is that the protection consent gives us may get bypassed altogether.

What if it’s possible to decode what you’re thinking or planning without you even knowing? Will you feel violated? Will you feel a loss of control? Privacy implications may be wide-ranging. Maybe advertisers could know your preferred brands and send you personalized ads – which may be convenient or creepy. Or maybe malicious entities could determine where you bank and your account’s PIN – which would be alarming.

With great power comes great responsibility

The potential ability to determine individuals’ preferences and personal information using their own brain signals has spawned a number of difficult but pressing questions: Should we be able to keep our neural signals private? That is, should neural security be a human right? How do we adequately protect and store all the neural data being recorded for research, and soon for leisure? How do consumers know if any protective or anonymization measures are being made with their neural data? As of now, neural data collected for commercial uses are not subject to the same legal protections covering biomedical research or health care. Should neural data be treated differently?

These are the kinds of conundrums that are best addressed by neural engineers and ethicists working together. Putting ethicists in labs alongside engineers – as we have done at the CSNE – is one way to ensure that privacy and security risks of neurotechnology, as well as other ethically important issues, are an active part of the research process instead of an afterthought. For instance, Tim Brown, an ethicist at the CSNE, is “housed” within a neural engineering research lab, allowing him to have daily conversations with researchers about ethical concerns. He’s also easily able to interact with – and, in fact, interview – research subjects about their ethical concerns about brain research.

There are important ethical and legal lessons to be drawn about technology and privacy from other areas, such as genetics and neuromarketing. But there seems to be something important and different about reading neural data. They’re more intimately connected to the mind and who we take ourselves to be. As such, ethical issues raised by BCI demand special attention.

Working on ethics while tech’s in its infancy

As we wrestle with how to address these privacy and security issues, there are two features of current P300 technology that will buy us time.

First, most commercial devices available use dry electrodes, which rely solely on skin contact to conduct electrical signals. This technology is prone to a low signal-to-noise ratio, meaning that we can extract only relatively basic forms of information from users. The brain signals we record are known to be highly variable (even for the same person) due to things like electrode movement and the constantly changing nature of brain signals themselves. Second, electrodes are not always in ideal locations to record.

All together, this inherent lack of reliability means that BCI devices are not nearly as ubiquitous today as they may be in the future. As electrode hardware and signal processing continue to improve, it will be easier to continuously use devices like these, and make it easier to extract personal information from an unknowing individual as well. The safest advice would be to not use these devices at all.

The goal should be that the ethical standards and the technology will mature together to ensure future BCI users are confident their privacy is being protected as they use these kinds of devices. It’s a rare opportunity for scientists, engineers, ethicists and eventually regulators to work together to create even better products than were originally dreamed of in science fiction.

Credit: sk.ru

In this episode, Audrow Nash and Christina Brester conduct interviews at the 2016 International Association of Science Parks and Areas of Innovation conference in Moscow, Russia. They speak with Vadim Kotenev of Rehabot and Motorica about prosthetic hands and rehabilatative devices; and Vagan Martirosyan, CEO of TryFit, a company that uses robotic sensors to help people find shoes that fit them well.

An image of one of the rehabilitative devices from Rehabot.

The robotic platform that scans your feet from TryFit.

Links

• Download mp3 (13.3 MB)
• IASP 2016 conference website
• Skolkovo’s robotics page
• Rehabot’s website
• TryFit’s website
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Would you like to make a robot to grasp something, but you think that is impossible to you just because you can’t buy a robot arm? I’m here to tell that you can definitely achieve this without buying a real robot. Let’s see how:

How is that possible?

The Smart Grasping Sandbox built by Shadow Robotics is now available for everybody on the ROS Development Studio – a system that allows you to create and test your robot programs through simulations using only a web browser.

Wait, what is the Smart Grasping Sandbox?

The Smart Grasping Sandbox is a public simulation for the Shadow’s Smart Grasping System with theUR10 robot from Universal Robots. It allows you to make a robot to grasp something without having to learn everything related to Machine Learning, and being available on the ROS Development Studio, it allows you to test it without the hassle of installing all the requirements.

As Ugo Cupcic said:

I don’t want to have to specify every aspect of a problem — I’d rather the system learn the best way to approach a given problem itself.

Using the Development Environment for Grasping

In order to user the Smart Grasping Sandbox on ROS Development Studio—also known as RDS—just go to http://rds.theconstructsim.com and Sign In. Once logged in, go to the Public Simulations, select the Smart Grasping Sandbox simulation and press the red Launch this simulation button.

After pressing the Launch this simulation, a new screen will appears asking you to select the launch file. Just keep the default main.launch and press the red run button.

After a few seconds, the simulation will be loaded and you will be able interact with the simulation.

Once the simulation is loaded, you can see the instructions about how to control it on the left side.

In the center you have the simulation itself, and on the right side you can see the Integrated Development Environment and the Web Shell that allows you to modify the code and send Linuxcommands to the simulation respectively.

You can modify, the simulation, the control programs or even create your own manipulation control programs, make the grasping system learn using deep learning or else. This is up to you as developer!

You can also see what the robot sees using the online ROS 3D Visualizer (RViz) typing the following command on the Web Shell:

$ rosrun rviz rviz

After running Rviz, you press the red Open ROS Graphic Tools button on the bottom left side of the system and a new tab should appear with Rviz.

Conclusion

Here we have seen how to use a simulation to programme a robot to grasp something.

The simulation uses ROS as the middleware to control the robot.

If you want to learn ROS or master your ROS Skills, we recommend you to give a try to Robot Ignite Academy.

If you are a robot builder and want to have your own simulation available for everybody on RDS, or you want to have a course specifically for your robot, contact us through the email info@theconstructsim.com.

I was recently asked about the differences between RADAR and LIDAR. I gave the generic answer about LIDAR having higher resolution and accuracy than RADAR. And RADAR having a longer range and performing better in dust and smokey conditions. When prompted for why RADAR is less accurate and lower resolution, I sort of mumbled through a response about the wavelength. However, I did not have a good response, so this post will be my better response.

LIDAR

LIDAR—which is short for Light Detection and Ranging—uses a laser that is emitted and then received back in the sensor. In most of the LIDAR sensors used for mapping (and self driving vehicles) the time between the emission and reception is computed to determine the time of flight (ToF). Knowing the speed of light and (1/2 of the) time for the wave to return (since the signal traveled out and back) we can compute how far away the object was that caused the light to bounce back. That value is the range information that is reported by the sensor. LIDAR’s generally use light in the near-infrared, visible (but not really visible), and UV spectrum’s.

There are some sensors that use triangulation to compute the position (instead of ToF). These are usually high accuracy, high resolution sensors. These sensors are great for verifying components on an assembly lines or inspecting thermal tile damage on the space shuttle. However that is not the focus of this post.

LIDAR data. The top shows the reflectivity data. The bottom shows the range data with brighter points being farther away.

The laser beam can also be focused to have a small spot size that does not expand much. This small spot size can help give a high resolution. If you have a spinning mirror (which is often the case) then you can shoot the laser every degree or so (based on the accuracy of the pointing mechanism) for improved resolution. It is not uncommon to find LIDAR’s operating at 0.25 degrees of angular resolution.

RADAR

RADAR—which is short for Radio Detection and Ranging—uses radio waves to compute velocity, and/or range to an object. Radio waves have less absorption (so less attenuation) than the light waves when contacting objects, so they can work over a longer distance. As you can see in the image below the RF waves have a larger wavelength than the LIDAR waves. The down side is that if an object is much smaller than the RF wave being used, the object might not reflect back enough energy to be detected. For that reason many RADAR’s in use for obstacle detection will be “high frequency” so that the wavelength is shorter (hence why we often use mm-wave in robotics) and can detect smaller objects. However, since LIDAR’s have the significantly smaller wavelength, they will still usually have a finer resolution.

Electromagnetic spectrum showing radio waves all the way on the left for RADAR and near-infrared/visible/ultra-violet waves towards the right for LIDAR usage.

Most of the RADAR’s that I have seen will have a narrow field of view (10’s of degrees) and then just return a single value (or they might have up to a few dozen channels, see multi-mode RADAR below) for the range of the detected object. There are tricks that some systems can do using multiple channels to also get the angle for the range measurement. The angle will not be as high resolution as most LIDAR’s. There are also some RADAR’s on the market that scan to get multiple measurements. The 2 approaches are a spinning antenna (such as what you see at airports or on ships) or electronically “spinning” which is a device using multiple internal antennas with no moving parts. More advanced (newer) RADAR’s can do things like track multiple objects. In many cases they will not actually return the points that outline the objects (like a LIDAR), but will return a range, bearing, and velocity to an estimated centroid of the detected item. If multiple objects are near each other, the sensor might confuse them as being one large object and return one centroid range [Here is the source and a good reference to read].

Using the Doppler frequency shift, the velocity of an object can also be easily determined with relatively little computational effort. If the RADAR sensor and the detected object are both moving than you get a relative velocity between the two objects.

With RADAR there are commonly two operating modes:

1. Time of Flight – This operates similar to the LIDAR sensor above, however it uses radio wave pulses for the Time of Flight calculations. Since the sensors are pulsed, it knows when the pulse was sent. So, computing the range can be easier than with the continuous wave sensors (described below). The resolution of the sensor can be adjusted by changing the pulse width and the length of time you listen for a response (a ping back). These sensors often have fixed antennas leading to a small operating field of views (compared to LIDAR).

There are some systems that will combine multiple ToF radio waves into one package with different pulse widths. These will allow for various ranges to be detected with higher accuracy. These are sometimes called multi-mode RADAR

2. Continuous Wave – This approach frequency modulates (FMCW) a wave and then compares the frequency of the reflected signal to the transmitted signal in order to determine a frequency shift. That frequency shift can be used to determine the range to the object that reflected it. The larger the shift the farther the object is from the sensor (within some bounds). Computing the frequency shift and the corresponding range is computationally easier than ToF, plus the electronics to do so are easier and cheaper. This makes continuous frequency modulated systems popular. Also, since separate transmit and receive antennas are often used, this approach can continuously be transmitting and receiving at the same time unlike the pulsed ToF approach, which needs to transmit then wait for a response. This feature and the simple electronics can make FMCW RADAR very fast at detecting objects.

There is another version of the Continuous Wave RADAR where the wave is not modulated. These systems are cheap and good for quickly detecting speed using the Doppler effect, however they can not determine range. They are often used by police to detect vehicle speed where the range is not important.

SONAR

Unrelated, I should note that SONAR or Sound Navigation and Ranging, can work in both of the modes as RADAR. The wavelength used is even larger than RADAR. It is farther to the left in the spectrum image shown before, and is off the chart.

I should point out that there are cool imaging SONAR sensors. The general idea is that you can make the sensing wave vertical so that the horizontal resolution is very fine (<1 degree) and the vertical resolution is larger (10+ degrees). You can then put many of these beams near each other in a sensor package. There are similar packages that do this with small wave length RADAR.

Cost

LIDAR Sensors tend to cost more than RADAR sensors for several reasons:

1. LIDAR using ToF needs high speed electronics that cost more.
2. LIDAR sensors needs CCD receivers, optics, motors, and lasers to generate and receive the waves used. RADAR only needs some stationary antennas.
3. LIDAR sensors have spinning parts for scanning. That requires motors and encoders. RADAR only needs some stationary antennas. (I know this is sort of similar to the line above).

Computationally Speaking

RADAR sensors tend to generate a lot less data as they just return a single point, or a few dozen points. When the sensor is multi-channel it is often just returning a range/velocity to a few centroid(ish) objects. The LIDAR sensors are sending lots of data about each individual laser point of range data. It is then up to the user to make that data useful. With the RADAR you have a simple number, but that number might not be the best. With LIDAR it depends on the roboticist to generate algorithms to detect the various objects and discern what is being viewed by the sensor.

If your goal is to detect a car in front of you (or driving towards you) and get its velocity, the RADAR can be great. If you are trying to determine the precise location of an item, generate a surface map, or find a small fence-post, a LIDAR might do better. Just remember that if you are in dust or rain the LIDAR might return a cloud of points right near the sensor (as it reads all of those particles/drops); while the RADAR might do much better.

Credit: nutonomy

Over the past few weeks, we’ve blogged about how machine learning is transforming research, nuclear decommissioning, and astronomy. Building on our “Ask the Experts” panel discussion on driverless cars last July, how might machine learning shape the way we move through and interact with the immediate environment around us?

As the last in our series of blog posts on machine learning in research, we spoke to Dr Nathan Griffiths to find out more about machine learning in transport. Nathan is a Reader in the Department of Computer Science at the University of Warwick, whose research into the application of machine learning for autonomous vehicles (or “driverless cars”) has been supported by a Royal Society University Research Fellowship.

What inspired you to study science? And why Computer Science particularly?

I have always been interested in how things work and understanding the patterns we see in the world around us. Science helps us to understand patterns, provides insights into how things work, and ultimately enables us to understand how complex structures can arise from simple rules. Computer Science appeals because it can be both a creative and an analytical science – we can use computers to create patterns, or use computers as tools to understand our world, including the patterns that exist within it, and predict patterns which may arise.

How is machine learning used in your research?

My research uses machine learning to give insights into how objects or people interact and how patterns emerge and evolve. This involves the use of computational models, known as agent-based modelling, to simulate the actions and interactions of entities (such as drivers or autonomous vehicles) to see how their decisions affect the wider systems in which they operate.

More specifically, machine learning can be used to analyse how cooperation emerges through interactions between objects. In the context of autonomous vehicles, there are many situations in which it is desirable to establish a cooperative environment, without the need for centralised control. For example, who should give way at a road junction where the rules are ambiguous or typically misinterpreted, such as flashing headlights to let another driver proceed? In this case, machine learning can be trained to predict what decisions a driver or autonomous vehicle will make. Machine learning algorithms will examine previous behaviours and learn from these behaviours, to then predict what will happen in the future. Many of the initial predictions will be incorrect, however the algorithm will modify its next predictions based on what it has learnt and continue to reduce its mistakes until it can make accurate predictions. An accurate algorithm could then be used to inform the decisions vehicles make and predict vehicle journeys and routes.

Modern vehicles are equipped with numerous sensors to collect vast amounts of data during vehicle journeys. For example cameras provide live images to the vehicle which are fed into an object-recognition machine learning algorithm which learns to identify and distinguish between entities such as pedestrians, cyclists or other vehicles on the road. The images can tell the vehicle how close the objects are and change the vehicle’s behaviour accordingly, such as by reducing speed to stay a safe distance from the car in front. The use of these data also enables transport to be made more efficient, such as through automatic traffic light signal detection and live notifications of parking opportunities which could help to reduce congestion.

What are the challenges in using machine learning in your field of research?

The techniques for extracting information from data and building predictive models are advanced, but perhaps the greatest challenge is to support the interface between machine learning and human users, which is likely to lead to greater public confidence in something like autonomous vehicles.

I also expect to see advances in techniques for dealing with “concept drift”, where changes in the world   need to be accompanied by changes in a machine learnt model. Concept drift is the phenomena whereby predictions may become less accurate over time if the models do not adapt to changes in the environment. This could be an issue if learning models are not able to adapt to changes sufficiently quickly to react.

Currently, autonomous vehicles are good at detecting the world around them and reacting as necessary to changes on the road. However the next challenge is to enable machine learning systems to not only react to changes and adapt to previously unseen environments (such as a swerving car or the vehicle being driven in adverse conditions) but be able to understand the probabilities of what it thinks might happen. For example, a car which is driving alongside a busy footpath will not just react to a pedestrian stepping out into the road, but will understand that there is a chance that this might happen and actively take measures to reduce risk in case it does, perhaps by positioning itself further away from the curb.

So what are the future prospects for machine learning?

The future is very bright for machine learning. Machine learning is fundamental to the development of driverless vehicles and is rapidly transforming the automotive world; vehicle manufacturers are very keen to get computer scientists on board for the betterment of their projects.

Machine learning is already central to how our vehicles work, underpinning systems such as lane-keeping assistance, reactive cruise control and automated vehicle personalisation and recognition. There is a wide and increasing range of applications that machine learning will allow us to tackle and it is enabling us to develop fully autonomous vehicles, which will be ubiquitous in the future. I also envisage that autonomous vehicles will be connected, such that the information and insights that one vehicle has learnt can be transferred to another; which may prove particularly useful for managing concept drift.

The Royal Society is undertaking a project about machine learning and its impact for the UK economy and society. To find out more please visit our project page.

This post was originally published on the Royal Society. Click here to view the original post.

I recently found a SICK LMS291 in the trash (dumpster diving time!). It is a bit of an older model—manufactured in August 2000—however the fundamentals should be similiar to the current generation of sensors. If you look at robots from that era you will often see them covered in this type of LIDAR sensor.

Here is a series of posts about how LIDAR works and evaluating multiple sensors.

If you have always wanted to know how a SICK LIDAR worked inside, now is your chance. If you did not always want to know, this is still your chance to see. Keep reading for some cool pictures.

The sensor feels solid and feels like a brick. I could probably throw it off a cliff and it will still work, not that you should try…

SICK LMS 291. You can see the front black area where the laser shoots out from. The 3 indicator lights at the top front, and the cutout in the back for the connectors.

SICK LMS 291 power and serial communications ports. There is a large sealed connector that plugs into each slot.

Front Plastic Shield

Mirror and encoder wheel. The laser shoots down from above the spinning mirror. The green post on the left reads the position from the encoder wheel shown on the bottom. The encoder wheel is the metal wheel with the slots cut into it. The 6 black tubes on the bottom I think are IR LEDs that are used for detecting dust or anything else occluding the front black visor.

Front Metal Case removed. Walls are about 3/10″ or 7.6mm thick

With the front removed here are the internals of the sensor. On the bottom you can see the metal ring which is a motor, above it is the spinning mirror encoder. The top section has the control boards and the laser hidden behind them.

Bottom view of the open sensor. The motor for spinning the mirror and encoder wheel is clearly visible at the bottom. The black rectangles on the top board are for detecting if the bottom LEDs are visible for fault detection.

These are the wings that go below and above the spinning mirror assembly. On the left side you can see the LEDs and the board on the right has the black square receivers.

The encoder is directly mounted to the mirror for increased accuracy of the laser position. It is very solidly built, and you can see that a designer was having fun with cutting the SICK logo into the metal disk. I was surprised when I saw this since I expected a higher resolution encoder wheel with more slots. But I guess the larger thicker slots can be more reliable. Towards the top of the wheel it looks like one slot is missing, I wonder if it is some sort of index.

Laser and receiver black can assembly visible after removing the front PCB. You can see the 2 gold coax connectors probably for some high frequency signals from the receiver. On the top board in the front you can see the 3 indicator lights that are visible on the exterior of the enclosure. On the left of the front board you can see the board interconnect connector that connects to the board on the right side of this image. When the boards are screwed down that connector is not coming loose.

Front board that I think converts the analog RF receiver signal to digital for the processor board. The board on top has the reciever, a coax cable goes from there to the gold can. Followed by going to this board. I am really curious what is under the gold can.

Look at the washers used above the screw holes. Excellent attention to detail to avoid screws coming loose from vibration. Not shown but there are many really nice touches, such as captive standoffs. I was surprised that I did not see any loctite.

Close up of the front board. Perhaps some RF signal conditioning for the receiver of the laser signal.

The main processor board that was connected to the front board shown before from the left side board connector. You can see the Xilinx FPGA that is running the sensor.

Bodge wire on the rear of the main processing board. I am surprised to see this. However it is really well done. Look at the epoxy blob holding the wire down.

Metal gold can to shield some components (or so I thought) from EM interference. Desiccant packet on the left is to control any moisture that might get into the sensor.

Inside of the shielding can. Is that for creating a time delay or some phase shift in the RF signal coming from the receiver?

Receiver of the incoming laser signal on the top board center. The black can has some mirrors and the laser.

Laser receiver module closeup. What is all the metal heatsinking for?

The motor and mirror removed. You can sort of look up into the laser and reciever assembly. The flat metal part between the ring and the laser assembly is the back of the power supply module.

Looking up from the mirror into the laser assembly. Most of the center that you see is a mirror. The slot in the plastic has its own mirror that points the laser from the gold block down to the spinning mirror. The returned signal goes on the sides of that slot to the receiver above.

Laser that is connected to the slot in the black mirror can.

Connected to gold pins above to (control) the laser. What is under that shield??

Under the shield. The DM74ALS seems to be a quad 2 input nand gate.

The power supply for the sensor that was located behind the spinning mirror module.

All of the components in the LIDAR sensor. The PCB that is still in the rear metal enclosure is the board that connects to the power and serial connectors that we saw on the external of the sensor before.

I hope you enjoyed seeing the internals of this sensor. It is interesting to see how they made it and the spinning mirror that we often hear about. Leave comments below to help identify parts or other questions you may have.

If you take humans out of the driving seat, could traffic jams, accidents and high fuel bills become a thing of the past? As cars become more automated and connected, attention is turning to how to best choreograph the interaction between the tens or hundreds of automated vehicles that will one day share the same segment of Europe’s road network.

It is one of the most keenly studied fields in transport – how to make sure that automated cars get to their destinations safely and efficiently. But the prospect of having a multitude of vehicles taking decisions while interacting on Europe’s roads is leading researchers to design new traffic management systems suitable for an era of connected transport.

The idea is to ensure that traffic flows as smoothly and efficiently as possible, potentially avoiding the jams and delays caused by human behaviour.

‘Travelling distances and time gaps between vehicles are crucial,’ said Professor Markos Papageorgiou, head of the Dynamic Systems & Simulation Laboratory at the Technical University of Crete, Greece. ‘It is also important to consider things such as how vehicles decide which lane to drive in.’

Prof. Papageorgiou’s TRAMAN21 project, funded by the EU’s European Research Council, is studying ways to manage the behaviour of individual vehicles, as well as highway control systems.

For example, the researchers have been looking at how adaptive cruise control (ACC) could improve traffic flows. ACC is a ‘smart’ system that speeds up and slows down a car as necessary to keep up with the one in front. Highway control systems using ACC to adjust time gaps between cars could help to reduce congestion.

‘It may be possible to have a traffic control system that looks at the traffic situation and recommends or even orders ACC cars to adopt a shorter time gap from the car in front,’ Prof. Papageorgiou said.

‘So during a peak period, or if you are near a bottleneck, the system could work out a gap that helps you avoid the congestion and gives higher flow and higher capacity at the time and place where this is needed.’

Variable speed limits

TRAMAN21, which runs to 2018, has been running tests on a highway near Melbourne, Australia, and is currently using variable speed limits to actively intervene in traffic to improve flows.

An active traffic management system of this kind could even help when only relatively few vehicles on the highway have sophisticated automation. But he believes that self-driving vehicle systems must be robust enough to be able to communicate with each other even when there are no overall traffic control systems.

‘Schools of fish and flocks of birds do not have central controls, and the individuals base their movement on the information from their own senses and the behaviour of their neighbours,’ Prof. Papageorgiou said.

‘In theory this could also work in traffic flow, but there is a lot of work to be done if this is to be perfected. Nature has had a long head-start.’

One way of managing traffic flow is platooning – a way to schedule trucks to meet up and drive in convoy on the highway. Magnus Adolfson from Swedish truckmaker Scania AB, who coordinated the EU-funded COMPANION project, says that platooning – which has already been demonstrated on Europe’s roads – can also reduce fuel costs and accidents.

The three-year project tested different combinations of distances between trucks, speeds and unexpected disruptions or stoppages.

Fuel savings

In tests with three-vehicle platoons, researchers achieved fuel savings of 5 %. And by keeping radio contact with each other, the trucks can also reduce the risk of accidents.

‘About 90 percent of road accidents are caused by driver error, and this system, particularly by taking speed out of the driver’s control, can make it safer than driving with an actual driver,’ Adolfson said.

The COMPANION project also showed the benefits of close communication between vehicles to reduce the likelihood of braking too hard and causing traffic jams further back.

‘There is enough evidence to show that using such a system can have a noticeable impact, so it would be good to get it into production as soon as possible,’ Adolfson said. The researchers have extended their collaboration to working with the Swedish authorities on possible implementation.

Rutger Beekelaar, a project manager at Dutch-based research organisation TNO, says that researchers need to demonstrate how automated cars can work safely together in order to increase their popularity.

‘Collaboration is essential to ensure vehicles can work together,’ he said. ‘We believe that in the near future, there will be more and more automation in traffic, in cars and trucks. But automated driving is not widely accepted yet.’

To tackle this, Beekelaar led a group of researchers in the EU-funded i-GAME project, which developed technology that uses wireless communication that contributes to managing and controlling automated vehicles.

They demonstrated these systems in highway conditions in the 2016 Grand Cooperative Driving Challenge in Helmond, in the Netherlands, which put groups of real vehicles through their paces to demonstrate cooperation, how they safely negotiated an intersection crossing, and merged with another column of traffic.

Beekelaar says that their technology is now being used in other European research projects, but that researchers, auto manufacturers, policymakers, and road authorities still need to work together to develop protocols, systems and standardisation, along with extra efforts to address cyber security, ethics and particularly the issue of public acceptance.

Level 3 automation, where the car handles all aspects of driving with the driver on standby, is being tested in Sweden. Image courtesy of Volvo cars

Today’s self-driving car isn’t exactly autonomous – the driver has to be able to take over in a pinch, and therein lies the roadblock researchers are trying to overcome. Automated cars are hurtling towards us at breakneck speed, with all-electric Teslas already running limited autopilot systems on roads worldwide and Google trialling its own autonomous pod cars.

However, before we can reply to emails while being driven to work, we have to have a foolproof way to determine when drivers can safely take control and when it should be left to the car.

‘Even in a limited number of tests, we have found that humans are not always performing as required,’ explained Dr Riender Happee, from Delft University of Technology in the Netherlands, who is coordinating the EU-funded HFAuto project to examine the problem and potential solutions.

‘We are close to concluding that the technology always has to be ready to resolve the situation if the driver doesn’t take back control.’

But in these car-to-human transitions, how can a computer decide whether it should hand back control?

‘Eye tracking can indicate driver state and attention,’ said Dr Happee. ‘We’re still to prove the practical usability, but if the car detects the driver is not in an adequate state, the car can stop in the safety lane instead of giving back control.’

Next level

It’s all a question of the level of automation. According to the scale of US-based standards organisation SAE International, Level 1 automation already exists in the form of automated braking and self-parking.

Level 4 & 5 automation, where you punch in the destination and sit back for a nap, is still on the horizon.

But we’ll soon reach Level 3 automation, where drivers can hand over control in situations like motorway driving and let their attention wander, as long as they can safely intervene when the car asks them to.

HFAuto’s 13 PhD students have been researching this human-machine transition challenge since 2013.

Backed with Marie Skłodowska-Curie action funding, the students have travelled Europe for secondments, to examine carmakers’ latest prototypes, and to carry out simulator and on-road tests of transition takeovers.

Alongside further trials of their transition interface, HFAuto partner Volvo has already started testing 100 highly automated Level 3 cars on Swedish public roads.

Another European research group is approaching the problem with a self-driving system that uses external sensors together with cameras inside the cab to monitor the driver’s attentiveness and actions.

Blink

‘Looking at what’s happening in the scene outside of the cars is nothing without the perspective of what’s happening inside the car,’ explained Dr Oihana Otaegui, head of the Vicomtech-IK4 applied research centre in San Sebastián, Spain.

She coordinates the work as part of the EU-funded VI-DAS project. The idea is to avoid high-risk transitions by monitoring factors like a driver’s gaze, blinking frequency and head pose — and combining this with real-time on-road factors to calculate how much time a driver needs to take the wheel.

Its self-driving system uses external cameras as affordable sensors, collecting data for the underlying artificial intelligence system, which tries to understand road situations like a human would.

VI-DAS is also studying real accidents to discern challenging situations where humans fail and using this to help train the system to detect and avoid such situations.

The group aims to have its first interface prototype working by September, with iterated prototypes appearing at the end of 2018 and 2019.

Dr Otaegui says the system could have potential security sector uses given its focus on creating artificial intelligence perception in any given environment, and hopes it could lead to fully automated driving.

‘It could even go down the path of Levels 4 and 5, depending on how well we can teach our system to react — and it will indeed be improving all the time we are working on this automation.’

The question of transitions is so important because it has an impact on liability – who is responsible in the case of an accident.

It’s clear that Level 2 drivers can be held liable if they cause a fender bender, while carmakers will take the rap once Level 4 is deployed. However, with Level 3 transitions, liability remains a burning question.

HFAuto’s Dr Happee believes the solution lies in specialist insurance options that will emerge.

‘Insurance solutions are expected (to emerge) where a car can be bought with risk insurance covering your own errors, and those which can be blamed on carmakers,’ he said.

Yet it goes further than that. Should a car choose to hit pedestrians in the road, or swerve into the path of an oncoming lorry, killing its occupants?

‘One thing coming out of our discussions is that no one would buy a car which will sacrifice its owner for the lives of others,’ said Dr Happee. ‘So it comes down to making these as safe as possible.’

The five levels of automation:

1. Driver Assistance: the car can either steer or regulate speed on its own.
2. Partial Automation: the vehicle can handle both steering and speed selection on its own in specific controlled situations, such as on a motorway.
3. Conditional Automation: the vehicle can be instructed to handle all aspects of driving, but the driver needs to be on standby to intervene if needed.
4. High Automation: the vehicle can be instructed to handle all aspects of driving, even if the driver is not available to intervene.
5. Level 5 – Full Automation: the vehicle handles all aspects of driving, all the time.

Figure 15: Adding 40m overshot to both ends of each survey line

UgCS is easy-to-use software for planning and flying UAV drone-survey missions. It supports almost any UAV platform, providing convenient tools for areal and linear surveys and enabling direct drone control. What’s more, UgCS enables professional land survey mission planning using photogrammetry techniques.

How to plan photogrammetry mission with UgCS

Standard land surveying photogrammetry mission planning with UgCS can be divided into following steps :

1. Obtain input data
2. Plan mission
3. Deploy ground control points
4. Fly mission
5. Image geotagging
6. Data processing
7. Map import to UgCS (optional)

https://www.youtube.com/watch?v=SalCQoG6WUw&feature=youtu.be

Step one: Obtain input data

Firstly, to reach the desired result, input settings have to be defined:

• Required GSD (ground sampling distance – size of single pixel on ground),
• Survey area boundaries,
• Required forward and side overlap.

GSD and area boundaries are usually defined by the customer’s requirements for output material parameters, for example by scale and resolution of digital map. Overlap should be chosen according to specific conditions of surveying area and requirements of data processing software.

Each data processing software (e.g., Pix4D, Agisoft Photoscan, Dronedeploy, Acute 3d) has specific requirements for side and forward overlaps for different surfaces. To choose correct values, please refer to documentation of chosen software. In general, 75% forward and 60% side overlap will be a good choice. Overlapping should be increased for areas with small amount of visual cues, for example for deserts or forests.

Often, aerial photogrammetry beginners are excited about the option to produce digital maps with extremely high resolution (1-2cm/pixel), and to use very small GSD for mission planning. This is very bad practice. Small GSD will result in longer flight time, hundreds of photos for each acre, tens of hours of processing and heavy output files. GSD should be set according to the output requirements of the digital map.

Other limitations can occur. For example, GSD of 10cm/pixel is required, but designed to use a Sony A6000 camera. Based on mentioned GSD and camera’s parameters, the flight altitude would be set to 510 meters. In most countries, maximum allowed altitude of UAV’s (without special permission) is limited to 120m/400ft AGL (above ground). Taking into account the maximum allowed altitude, the maximum possible GSD in this case could be no more than 2.3cm.

Step two: Plan your mission

Mission planning consists of two stages:

• Initial planning,
• Route optimisation.

-Initial planning:

The first step is to set surveying area using UgCS Photogrammetry tool. Area can be set using visual cues on underlying map or using exact coordinates of edges. The result – survey area is marked with yellow boundaries (Figure 1).

Figure 1: Setting the survey area

The next step is to set GSD and overlapping for the camera in Photogrammetry tool’s settings window (Figure 2).

Figure 2: Setting camera’s Ground Sampling Distance and overlapping

To take photos in Photogrammetry tool’s setting window, define the control action of the camera (Figure 3). Set camera by distance triggering action with default values.

Figure 3: Setting camera’s control action

At this point, initial route planning is completed. UgCS will automatically calculate photogrammetry route (see Figure 4).

Figure 4: Calculated photogrammetry survey route before optimisation

-Route optimisation

To optimise the route, it’s calculated parameters should be known: altitude, estimated flight time, number of shots, etc.

Part of the route’s calculated information can be found in the elevation profile window. To access the elevation profile window (if it is not visible on screen) click the parameters icon on the route card (lower-right corner, see Figure 5), and from the drop-down menu select show elevation:

Figure 5: Accessing elevation window from Route cards Parameters settings

The elevation profile window will present an estimated route length, duration, waypoint count and min/max altitude data:

Figure 6: Route values in elevation profile window

To get other calculated values, open route log by clicking on route status indicator: the green check-mark (upper-right corner, see Figure 7) of the route card:

Figure 7: Route card and status indicator, Route log

Using route parameters, it can be optimised to be more efficient and safe.

-Survey line direction

By default, UgCS will trace survey lines from south to north. But, in most cases, it will be more optimal to fly parallel to the longest boundary line of the survey area. To change survey line direction, edit direction angle field in the photogrammetry tool. In the example, by changing angle to 135 degrees, the number of passes is reduced from five (Figure 4) to four (Figure 8) and route length is 1km instead of 1.3km.

Figure 8: Changed survey line angle to be parallel to longest boundary

-Altitude type

UgCS Photogrammetry tool has the option to define how to trace the route according to altitude, with constant altitude above ground (AGL) or above mean sea level (AMSL). Please refer to your data processing software requirements as to which altitude tracking method it recommend.

In the UgCS team’s experience, the choice of altitude type depends on desired result. For orthophotomap (standard aerial land survey output format) it is better to choose AGL to ensure constant GSD for the entire map. If the aim is to produce DEM or 3D reconstruction, use AMSL so the data processing software has more data to correctly determine ground elevation by photos in order to provide more qualitative output.

Figure 9: Elevation profile with constant altitude above mean sea level (AMSL)

In this case, UgCS will calculate flight altitude based on the lowest point of the survey area.

If AGL is selected in photogrammetry tool’s settings, UgCS will calculate the altitude for each waypoint. But in this case, terrain following will be rough if no “additional waypoints” are added (see Figure 10).

Figure 10: Elevation profile with AGL without additional waypoints

Therefore, if AGL is used, add some “additional waypoints” flags and UgCS will calculate a flight plan with elevation profile accordingly (see Figure 11).

Figure 11: Elevation profile with AGL with additional waypoints

-Speed

In general, if flight speed is increased it will minimise flight time. But high speed in combination with large camera exposure can result in blurred images. In most cases 10m/s is the best choice.

-Camera control method

UgCS supports 3 camera control methods (actions):

1. Make a shot (trigger camera) in waypoint,
2. Make shot every N seconds,
3. Make shot every N meters.

Not all autopilots support all 3 camera control options. For example (quite old) DJI A2 does support all three options, but newer (starting from Phantom 3 and up to M600) cameras support only triggering in waypoints and by time. DJI promised to implement triggering by distance, but it’s not available yet.

Here are some benefits and drawbacks for all three methods:

Table 1: Benefits and Drawback for camera triggering methods

In conclusion:

• Trigger in waypoints should be preferred when possible
• Trigger by time should be used only if no other method is possible
• Trigger by distance should be used when triggering in waypoints is not possible to use

To select triggering method in UgCS Photogrammetry tool accordingly, use one of three available icons:

• Set camera mode
• Set camera by time
• Set camera by distance

-Glibal control

Drones, e.g., DJI Phantom 3, Phantom 4, Inspire, M100 or M600 with integrated gimbal, have the option to control camera position as part of an automatic route plan.

It is advisable to set camera to nadir position in the first waypoint, and in horizontal position before landing to prevent lenses from potential damage.

To set camera position, select the waypoint preceding the photogrammetry area and click set camera attitude/zoom (Figure 12) and enter “90” in the “Tilt” field (Figure 13).

Figure 12: Setting camera attitude

Figure 13: Setting camera position

As described previously, this waypoint should be a Stop&Turn type, otherwise the drone could skip this action.

To set camera to horizontal position, select last waypoint of survey route and click set camera attitude/zoom and enter “0” in the “Tilt” field.

-Turn types

Most autopilots or multirotor drones support different turn types in waypoints. Most popular DJI drones have three turn-types:

• Stop and Turn: drone flies to the fixed point accurately, stays at that fixed point and then flies to next fixed point.
• Bank Turn: the drone would fly with constant speed from one point to another without stopping.
• Adaptive Bank Turn: It is almost the same performance like Bank Turn mode (Figure 13), but the real flight routine will be more accurately than Bank Turn.

It is advisable not to use Bank Turn for photogrammetry missions. Drone interprets Bank Turns as “recommendation destination waypoint”. The drone will fly towards this direction but will almost never pass through the waypoint. Because drone will not pass the waypoint, no action will be executed, meaning the camera will not be triggered, etc.

Adaptive Bank Turn should be used with caution because a drone can miss waypoints and, again, no camera triggering will be initiated.

Figure 14: Illustration of typical DJI drone trajectories for Bank Turn and Adaptive Bank Turn types

Sometimes, adaptive bank turn type has to be used in order to have shorter flight time compared to stop and turn. When using adaptive bank turns, it is recommended to use overshot (see below) for the photogrammetry area.

-Overshot

Initially overshot was implemented for fixed wing (airplane) drones in order to have enough space for manoeuvring a U-turn.

Overshot can be set in photogrammetry tool to add an extra segment to both ends of each survey line.

Figure 15: Adding 40m overshot to both ends of each survey line

In the example (Figure 15) can be seen that UgCS added 40m additional segments to both ends of each survey line (comparing to Figure 8).

Adding overshot is useful for copter-UAVs in two situations:

1. When Adaptive Bank Turns are used (or similar method for non-DJI drones), adding overshot will increase the chance that drone will precisely enter survey line and camera control action will be triggered. UgCS Team recommends to specify overshot that is approximately equal to distance between the parallel survey lines.
2. When Stop and Turn type is in use in combination with action to trigger camera in waypoints, there is a possibility that before making the shot, drone will start rotation to next waypoint – it can result in having photos with wrong orientation or blurred. To avoid that, shorter overshot has to be set, for example 5m. Don’t specify too short value (< 3m) because some drones could ignore waypoints, that are too close.

Figure 16: Example of blurred image taken by drone in rotation to next waypoint

-Takeoff point

It is important to check the takeoff area at site before flying any mission! To better explain best practice on how to set takeoff point, first discuss an example of how it should not be done. Supposing that the takeoff point in our example mission (Figure 17) would be from the point marked with the airplane-icon, and drone pilot would upload the route on the ground with set automatic mission for automatic take-off.

Figure 17: Take-off point example

Most drones in automatic takeoff mode would climb to low altitude about 3-10meters and then fly straight towards the first waypoint. Other drones would fly towards first waypoint straight from ground. Looking closely at the example map (Figure 17), some trees between the takeoff point and the first waypoint can be noticed. In this example, the drone more likely will not reach a safe altitude and will hit the trees.

Not only the surroundings can affect takeoff planning. Drone manufacturers can change drones elevation behavior in drone firmware, therefore after firmware updates it is recommended that you check drones automatic takeoff mode.

Also, a very important consideration is that most small UAVs use relative altitude for mission planing. Altitude counted relatively according to first waypoint is a second reason why an actual takeoff point should be near the first waypoint, and on the same terrain level.

UgCS Team recommends placing the first waypoint as close as possible to actual takeoff point and specifying a safe takeoff altitude (≈30m in most situations will be above any trees, see Figure 18). This is the only method that warrants safe takeoff for any mission. It also protects from any weird drone behaviour and unpredictable firmware updates, etc.

Figure 18: Route with safe take-off

-Entry point to the survey grid

In the previous example, (see Figure 18), it can be noticed, that after adding the takeoff point, the route’s survey grid entry point was changed. This is because if additional waypoint is added subsequently to the photogrammetry area, UgCS will plan to fly the survey grid starting from nearest corner to the previous waypoint.

To change the entry point to survey grid, set additional waypoint close to the desired starting corner (see Figure 19).

Figure 19: Changing survey grid entry point by adding additional waypoint

-Landing point

If no landing point will be added outside the photogrammetry area after the survey mission, the drone will fly and hover in the last waypoint. There are two options for landing:

1. Take manual control over the drone and fly to landing point manually,
2. Activate the Return Home command in UgCS or from Remote Controller (RC).

In situations when the radio link with the drone is lost, for example if the survey area is large or there are problems with the remote controller, depending on the drone and it’s settings, one of these actions can occur:

• Drone will return to home location automatically if lost radio link with ground station,
• Drone will fly to last waypoint of survey area and hover as long as battery capacity will enable that, then: drone will perform emergency landing, or it will try to fly to home location.

The recommendation is to add an explicit landing points to the route in order to avoid relying on unpredictable drone behavior or settings.

If the drone doesn’t support automatic landing, or the pilot prefers to land manually, place the route’s last waypoint over the planned landing point with an altitude for comfortable manual drone descending and landing above any obstacles in the surrounding area. In general 30m is best choice.

-Action execution

Photogrammetry tool has a magic parameter “Action Execution” with three possible values:

• Every point
• At start
• Forward passes

This parameter defines how and where camera actions specified for photogrammetry tool will be executed.

The most useful option for photogrammetry/survey missions is to set forward passes, the drone will make photos only on survey lines, but will not make excess photos on perpendicular lines.

-Complex survey areas

UgCS enables photogrammetry/survey mission planning for irregular areas, having functionality to combine any number of photogrammetry areas in one route, avoiding splitting the area in separate routes.

For example, if a mission has to be planned for two fields connected in a T-shape, and if these two fields are marked as one photogrammetry area, the whole route will not be optimal, regardless any direction of survey lines.

Figure 20: Complex survey area before optimisation

If the survey area is marked as two photogrammetry areas within one route, survey lines for each area can be optimised individually (see Figure 21).

Figure 21: Optimised survey flight passes for each part of a complex photogrammetry area

Step three: deploy ground control points

Ground control points are mandatory if the survey output map has to be precisely aligned to coordinates on Earth.

There are lots of discussions about the necessity of ground control points in cases when a drone is equipped with Real Time Kinematics (RTK) GPS receivers with centimeter-level accuracy. This is useful, but the drone coordinates are not in themselves sufficient because, for precise map aligning, image center coordinates are necessary.

Data processing softwares like Agisoft Photoscan, Dronedeplay, Pix4d, Icarus OneButton and others will produce very accurate maps using geotagged images, but the real precision of the map will not be known without ground control points.

Conclusion: ground control points have to be used to create survey-grade result. For a map with approximate precision, it is sufficient to rely just on RTK GPS and the capabilities of data processing software.

Step four: fly your mission

For carefully planned missions, flying it is the most straightforward step. Mission execution differs according to the type of UAV and equipment used, therefore it will not be described in detail in this article (please refer to equipment’s and UgCS documentation).

Important issues before flying:

• In most countries there are strict regulations for UAV usage. Always comply with the regulations! Usually these rules can be found on web-site of local aviation authority.
• In some countries special permission for any kind of aerial photo/video shooting is needed. Please check local regulations.
• In most cases missions are planned before arriving to flying location (e.g., in office, at home) using satellite imaginary from Google maps, Bing, etc. Before flying always check actual circumstances at the location. There could be a need to adjust take-off/landing points, for example, to avoid tall obstacles (e.g., trees, masts, power lines) in your survey area.

Step five: image geotagging

Image geotagging is optional if ground control points were used, but almost any data processing software will require less time to process geotagged images.

Some of the latest and professional drones with integrated cameras can geotag images automatically during flight. In other cases images can be geotagged in UgCS after flight.

Very important: UgCS uses the telemetry log from drone, that is received via radio channel, to extract the drone’s altitude for any given moment (when pictures were taken). To geotag pictures using UgCS, assure robust telemetry reception during flight.

For detailed information how to geotag images using UgCS refer to UgCS User Manual.

Step six: data processing

For data processing, use third party software or services available on the market.

From UgCS Team experience, the most powerful and flexible software is Agisoft Photoscan (http://www.agisoft.com/), but sometimes too much user input is required to get necessary results. The most uncomplicated solution for users is online service Dronedeploy (https://www.dronedeploy.com/). All other software packages and services will fit somewhere between these two in terms of complexity and power.

Step seven (optional): import created map to UgCS

Should the need arise for the mission to be repeated in the future, UgCS enables importing the GeoTiff file as a map layer and using it for mission planning. More detailed instructions can be found in UgCS User Manual. See the result of an imported map created using UgCS photogrammetry tool imported as GeoTiff file in Figure 22.

Figure 22: Imported GeoTiff map as layer. The map is output of a Photogrammetry survey mission panned with UgCS

Visit the UgCS homepage

Download this tutorial as a PDF

If you liked this tutorial, you may also enjoy these:

• Programming for robotics: Introduction to ROS
• Python programming your NAO robot
• Robotics, maths, python: A fledgling computer scientist’s guide to inverse kinematics
• Adjust how you learn and quickly pick up robotics programming
• 3-D printing hydraulically-powered robots, no assembly required

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

We have just taken another momentous step in the journey to unveil the hidden wonders of our own planet!

Since the launch of the Shell Ocean Discovery XPRIZE at the American Geophysical Union Fall Meeting in San Francisco in December 2015, individuals from around the world have been racing to form Teams and develop a range of groundbreaking technologies to access the deep-sea. Registration closed at the end of September 2016 with 32 bold Teams stepping forward to take on the challenge of mapping and imaging our ocean as never before.

Today, we announce the 21 semi-finalists Teams advancing in the Ocean Discovery XPRIZE. These innovative semi-finalist Teams, consisting of almost 350 individuals from 25 countries, represent a broad, impressive diversity of backgrounds and expertise, including middle and high school students, university students, maker-movement enthusiasts, and water and ocean industry professionals.

This diversity is also reflected in the array of technologies and techniques the semi-finalist Teams are developing. Some are proposing to use drones as a mechanism to drop subsea instruments into the water, while others are proposing to use drones that not only go through air, but then dive into the watery depths. Non-aerial Entries include autonomous surface vehicles carrying subsea robots that will return to the ‘mothership’ when their work is done, as well as vehicles and robots that will remain beneath the sea surface from the moment they leave the shore.

The 21 semi-finalist Teams will now have to prove their technology in the field in Round 1 testing, which will take place later this year. As with every XPRIZE, the Shell Ocean Discovery XPRIZE sets audacious goals. In Round 1, the Entries that these Teams are developing will have only 16 hours to map the sea-floor at depths of 2000m and produce a high-resolution map (at least 5m horizontal resolution, at least 0.5m vertical resolution) of at least 20% of the 500km2 Competition Area. Additionally, Teams will have to bring back 5 images of an archeological, biological, or geological feature, as well as an image of an object that we will specify. As if this wasn’t enough, they have an additional hurdle to overcome – they will have to deploy from the shore with no humans allowed in the competition area.

To put this challenge into perspective, it can take days to map 500 km2 of the ocean using current state-of-the-art technologies. If you want 5m accuracy, mapping can take over a week and requires going out on a ship, which can easily cost over $60,000 per day, making it a very expensive endeavor. The cutting-edge technology that will come out of this XPRIZE will truly revolutionize our ability to access the ocean.

In addition to the main Prize of $6 million, a dozen Teams have also opted to compete for the National Oceanic and Atmospheric Administration Bonus Prize of $1 million. These Teams are developing pioneering technology to detect a chemical or biological signal underwater and then autonomously track that signal back to its source.

Partnering with us to truly make this competition a global success are industry titans, Fugro and Esri. Fugro, the industry leader in ocean mapping and survey, will be provide the baseline maps against which the judging panel will assess the Entries. Esri, the global leader in GIS, is providing software to the Teams to produce their maps.

We are looking forward to an exciting competition with exponentially evolving technologies, including smart underwater robots, robot swarms, drones, and artificial intelligence to perform the tasks that have been laid out. These Teams will really take us where no one has gone before! As their journeys of discovery set sail, we invite you to follow the competion and this phenomenal mission to map our beautiful planet.

If you liked this article, you may also be interested in:

• XPRIZE calling all innovators to help map our ocean
• Unmanned and underneath: Stay in the know about underwater drones
• Into the deep: Underwater machines keep an eye on climate change
• What you need to know about underwater drones
• CoCoRo: Tracking the development of the world’s largest autonomous underwater swarm
• Ocean-going robot fleet successfully completes fish-tracking mission

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Jussi Aaltonen explaining some robotics-specific topics.

The UNEXMIN (Underwater Explorer for Flooded Mines) project is almost one year old. After a busy first year of work, UNEXMIN is on-schedule to deliver the first mechanical UX-1 prototype. Jussi Aaltonen, from TUT (Tampere University of Technology), leaders of WP1 – Robotic Functions Validations, talks about what has been done over the past year concerning his team’s work in UNEXMIN’s development scene.

Luís Lopes, LPRC: What is WP1 (Robotic Functions Validations) and what has been developed from it?

Jussi Aaltonen: The purpose of WP1 is to develop and validate a robotic platform, i.e. all mechanical and mechatronic structures, systems and components of UX-1 (the name of the robot). Also, low level (directional) control principles are developed and validated in this work package. WP1 has successfully established a robotic platform which fulfils all necessary requirements.

LL: How important is the work developed in WP1 for the future UX-1 robots

JA: Work in WP1 is fundamental for the successful completion of the project. All later development stages rely on this work.

LL: The systems/subsystems that the robot will employ were defined in this WP. What can you tell about them? How crucial are they for the robot’s overall functionality?

JA: All systems are needed for the robot to achieve full functionality. However, survivability, i.e. the robot’s capability to return from its missions, is more heavily affected by some systems than others. It is, for example, possible to return with a broken multi-spectral camera, but losing a sonar may cause the robot to be lost.

LL: What kinds of tests were developed to test the hardware/software systems? How about the results obtained, were they positive?

JA: Tests are done as so-called hardware-in-the-loop (HWIL) tests in laboratory. In HWIL testing, computer simulations and real hardware components interact, making it possible to do laboratory testing more closely to real operating conditions. Components and systems have also been tested in the pool by using a plastic mock-up of UX-1. Tests have been successful, which, unfortunately, does not always mean positive results. But weaknesses were found and lessons learned.

LL: In what conditions was the prototype tested?

JA: Tests have been done in shallow depths (below 5 m) and moderate temperatures. Pressure testing at 60 bars (equivalent of 600 m depth) will start soon.

LL: The robot is designed considering stakeholders’ wants. How hard was it to define the requirements and specifications for the UX-1 robots?

JA: I have to say that requirements and specifications are the easy part. The hardest is to design a robot complying them all.

LL: Being leader of such an important WP of the UNEXMIN project must be challenging. Has it worked out as you imagined? Do you feel that everything has been accomplished?

JA: Over this past year of cooperation, UNEXMIN teams from the various partners have gotten to know each other and each other’s ways of working. The possibilities to meet the others in person and to really collaborate in real life conditions, such as in workshops, have been a great benefit and a very important thing, bonding people from all around Europe together and working towards  a common goal. Of course there are always some challenges. The biggest challenge has been finding a common and efficient enough platform for information exchange.

LL: TUT will organise a UNEXMIN workshop in the beginning of February (2017) in Tampere, Finland. What can be expected to come out of this?

JA: This event will conclude the work of the first year in the UNEXMIN project. Most of the work packages have already started or about to start (exception goes to WP7: Demonstration/Pilots), and thus we will hear a lot about results and developments in this meeting. We will also hear about the plans for a couple of essential work packages starting right after the meeting. I expect that we will have a good and fruitful meeting with the same good spirit and warm atmosphere that we have always had. There will probably be some heated discussions, but that is only a sign of people being passionate about their work.

LL: You are a specialist in the robotics field. What possible impact do you think the UNEXMIN project will have on the robotics field?

JA: UX-1 [UNEXMIN robot design-prototypes] will have a very different structure and layout than what autonomous underwater vehicles usually have. This will most probably have some effects on the field of robotics and will open new doors to further miniaturised underwater robots – the future starts here. Also, the environment where UX-1 will operate is probably the most challenging after Mars and asteroid landers and that will also give tiger leaps in the field of robotics, developing new and better robotic systems.

LL: After one year, how do you feel the UNEXMIN project has progressed?

JA: We are roughly in the place where we should be according to our plans. Work has been harder than what was expected. On the other hand, we learned a lot more, while still managing to keep pace.

This post was originally published on UNEXMIN.

If you enjoyed this article, you might also be interested in:

• Surveying flooded mines with UNEXMIN: Interview with Norbert Zajzon
• When robots become art: Interview with Yi-Wei Keng
• A European perspective on robot law: Interview with Mady Delavaux
• Art meets robotics in Patrick Tresset’s OPLINE Prize entry
• Robots Podcast #171: Robotics in theatre, film and television, with Grant Imahara and Richard McKenna

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this episode, Ron Vanderkley spoke to Dr Moritz Tenorth, head of software development at Magazino, a Munich-based startup developing mobile pick-and-place robots for item-specific logistics. They discussed his work on the Toru robot and what it means to the warehouse industry today and in the future.

Dr Moritz Tenorth

Dr. Moritz Tenorth

After obtaining his PhD in robotics from TU Munich Dr Moritz Tenorth spent several years as a post-doc and freelance robotics consultant, with stays at the CMU Robotics Institute and the ATR labs in Japan. His research focuses on knowledge representation methods that can help autonomous robots make smarter decisions, and on methods for task coordination in robotics.

 Links

• Magazino (website)
• Toru Cube – Pick by robot (Youtube)
• Pick-by-Robot | technical features of TORU Cube (YouTube)
• Magazino TORU Cube – DHL Logistik-Center (YouTube)
• Download mp3 (10.1 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Image: Clearpath

Sriram Narasimhan’s research team are shaking things up in the Civil Engineering Structures Lab at the University of Waterloo. The research, which is led by Ph.D Candidate Kevin Goorts, is developing a new mobile damping system for suppressing unwanted vibrations in lightweight, flexible bridges. Whereas damping systems are often permanent fixtures built into the bridge, their system is designed to be adaptable, autonomous, and better suited for rapid, temporary deployment.

A SHIFT TOWARDS FLEXIBLE, LIGHT-WEIGHT STRUCTURES

Their work follows a recent industry shift towards using lightweight materials in the construction of civil engineering structures. Driven primarily by cost savings, as well as both the ease and speed of deployment, these lightweight structures could be a pedestrian bridge or a temporary bridge used in disaster relief scenarios, and are often more sensitive to external forces due to the reduced self-weight, and therefore require auxiliary damping devices.

Husky-based mobile bridge damping system on a full-scale aluminum pedestrian bridge. Image: Clearpath

HUSKY UGV: THE IDEAL MOBILE ROBOTIC PLATFORM

At the center of the team’s mobile control system is a Husky unmanned ground vehicle (UGV). An electromechanical mass damper mounted on top of the Husky is used to generate inertial control forces which are magnified by the body dynamics of the Husky. When situated on a bridge, the system is able to respond to changes in the structural response by autonomously positioning itself at the appropriate location and applying the desired control force.

For Goorts’ research, Husky UGV was the ideal mobile base upon which to develop their system. “The Husky is a rugged vehicle suitable for outdoor applications with sufficient payload capacity for both the damper and associated computational equipment.” said Goorts. “The low-profile and large lug thread tires are well suited for providing the necessary static friction to prevent sliding and transfer control forces. Moreover, Husky’s readily available ROS (Robot Operating System) integration allowed us to use several sensor types and position control algorithms.”

Linear motor and front-facing Kinect vision sensor mounted on Husky UGV mobile base. Image: Clearpath

The Husky is also equipped with a laptop running ROS and Kinect vision sensors on the front and back of the vehicle. Using wheel encoder data and measurements from the Kinect sensors, the system is able to perform SLAM (Simultaneous Localization and Mapping) and autonomously navigate back and forth along the span of the bridge. One of the immediate challenges the team faced was getting the robot to accurately localize on a bridge with a repetitive structural design – everything looked the same to the robot from different positions. To overcome this, they placed unique AR (augmented reality) tags along the bridge between which the robot could navigate.

A National Instruments (NI) cRIO is being used for all data processing tasks and execution of the structural control loops, and a TCP data link communicates between the ROS laptop and cRIO. Image: Clearpath.

BRIDGE VIBRATIONS REDUCED BY 70%

The effectiveness of the control system is tested using real-time hybrid simulation (RTHS), a dynamic testing method that couples the physical control system with a numerical model of the structure. These tests were carried out on a full-scale aluminum pedestrian bridge with re-designed panels to measure the forces being applied by the system. Results from the simulations show the system can provide up to a 70% reduction in the lateral displacement of the bridge.

The current prototype is suitable for bridges with a mass up to 1-tonne and is scalable to accommodate larger structures, either with a larger vehicle or with multiple vehicles that work cooperatively. Looking forward, the team is exploring the idea of making the system vehicle agnostic, which could allow any vehicle to be turned into an autonomous bridge stabilizing machine.

The post Customer Story: Deployable, Autonomous Vibration Control of Bridges Using Husky UGV appeared first on Clearpath Robotics.

If you liked this article, you may also be interested in:

• Clearpath Robotics launches “Warthog UGV”, the amphibious robot
• Interview: International Ground Vehicle Ground Competition (IGVC) winner on “Bigfoot 2” Husky UGV
• 4 reasons why Industry 4.0 will leave AGVs behind
• 5 innovation challenges in materials transport
• Clearpath and Christie demo 3D video game with robots

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this episode, Christina Brester interviews Vladimir Stanovov, PhD student and researcher at the Siberian State Aerospace University (Krasnoyarsk, Russia). Stanovov speaks about a speech-controlled wheelchair, which seeks to provide people that are quadriplegic, that is people with partial or total loss of use of their limbs and torso, with the possibility to control their wheelchairs through voice commands. In this interview Stanovov discusses the basic parts of the speech-controlled wheelchair, the fuzzy controller he created, and the trials they had in the medical center.

Below is a video that explains the speech-controlled wheelchair.

Vladimir Stanovov

Vladimir Stanovov received his Bachelor’s (2012) and Master’s (2014) degree at the Siberian State Aerospace University (SibSAU) in Krasnoyarsk, Russia. Stanovov is currently a PhD student in Computer Science and Engineering at SibSAU, where he performs the research under the supervision of Professor Semenkin. Stanovov’s PhD thesis is about evolutionary algorithms, fuzzy logic and their applications in machine learning.

Links:

• Download mp3 (11.6 MB)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS
• Vladimir Stanovov on reserchgate.net

In this episode, Abate De Mey interviews Edward Neff, founder of SMAC Corporation. Mr. Neff discusses how breakthroughs in his company have allowed them to develop linear actuators compact enough to be used to actuate robotic fingers. Companies like Apple and Samsung push for the development of robotic fingers to perform lifelike tests on their phones.

The robotic fingers developed using SMAC’s linear servo motors simulate a button press on a screen by employing a strong downwards force, as well as small sideways forces tangential to the screen. This mimics a realistic way that people interact with their touchscreens. The robotic fingers can also gently bump into surfaces, pull, and push objects. Other applications for these fingers include electronic circuit board assembly, and screwing bottle caps onto bottles.

Edward Neff

Edward Neff grew up in Birmingham, Michigan (USA) and is a graduate of the University of Michigan. He has a career history of managing automation companies in Belgium and Japan. Mr. Neff is the founder of SMAC Corporation in Carlsbad, California. SMAC is a leading manufacturer of precision programmable electric actuators. The company is planning a 2018 IPO. Mr Neff has over 40 patents in robotics and serves as a Director in BioTech and Medical Device companies.

Links

• SMAC Website
• Robotic Finger using a SmartPhone (YouTube)
• Robotic Finger at the Tokyo Robot Show (YouTube)
• Robotic Finger VS Human (YouTube)
• Smart Assembly Video (YouTube)
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS

Research and development of robotic assistive technologies has gained tremendous momentum in the last decade due to several factors such as the maturity level reached by several technologies, the advances in robotics and AI and the fact that more than 700 million of persons have some kind of disability or handicap. For many people with mobility impairments, essential and simple tasks, such as dressing or feeding, require the assistance of dedicated people. Thus, the use of devices providing independent mobility can have a large impact on their quality of life.

A team of researchers from the University of Cassino, under the direction of Professor Filippo Arrichiello, is working on the development of a control architecture for assistive robotic systems operated via a Brain-Computer Interface (BCI) to allow a user to operate a robot using his thoughts. The proposed software architecture relies on widely used frameworks to operate BCIs and robots (namely, BCI2000 for the operation of the BCI and ROS for the control of the robot) integrating control, perception and communication modules developed for the application at hand.

In particular, using the BCI2000 framework, they developed a Graphical User Interface (GUI) based on P300 paradigm to allow the user to select options via a BCI Epoch+. The user can select objects from the scene and the action to perform by focusing attention on the corresponding flashing icons on a screen. By counting how many times the icon he wants to select flashes, the user generates a P300 potential that is detected via the BCI and translated in a selection operation on the GUI.

Messages related to the object and actions selected via the GUI are automatically sent to a 7DOF lightweight robot manipulator Jaco2 that has to operate in the environment. The robot control architecture relies on a perception module that allows to detection and localization of objects, and the face/mouth of the user, using a RGB-D sensor (Kinect one). The motion of the manipulator is then controlled relying on a closed loop inverse kinematic algorithm that simultaneously manages multiple setbased and equality-based tasks. Preliminary experiments have been performed where the user commands the robot via the BCI in order to move objects on a table or to select a bottle from where to drink.

Team members: prof. Filippo Arrichiello, Paolo Di Lillo (PhD student), Daniele Di Vito (PhD student), prof. Gianluca Antonelli, prof. Stefano Chiaverini.

Click here for more information on the Robotics Research Group of the DIEI.

If you enjoyed this article, you might also be interested in:

• Virtual race: Competing in Brain Computer Interface at Cybathlon
• Brain Computer Interface used to control the movement and actions of an android robot
• Can a brain-computer interface convert your thoughts to text?

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

But what does it mean to describe systems as complex? How do these complex systems differ from the more easily understood ‘modular’ systems that we are familiar with? Vocabulary in this area is often dangerously inconsistent. For example, the terms ’emergence’, ‘complex’, and ‘complicated’ are used differently by different disciplines, and often differently even within the same discipline. This makes it very difficult to understand whether people are really talking about the same thing, and whether the systems being described are different in superficial or profound ways. On the one hand, failing to identify the underlying similarities between different systems (whether modular or complex) results in missed opportunities for sharing knowledge, best practices and methods. On the other hand, failing to identify the underlying differences between different systems results in practices and methods being misapplied. More broadly, many of today’s real-world problems require engineers, designers and policy-makers across all domains to think in terms of complex systems.
To address problems with translating between disciplines, Chih-Chun Chen and Nathan Crilly at the University of Cambridge have produced ‘A primer on the design and science of complex systems’. This introduces complex systems constructs by building them up from basic concepts, and contrasting them with more familiar constructs that are associated with modularity. For example, ’emergence’ can be understood with respect to a breakdown in how a system’s functions are mapped to the structures that perform those functions. Abstract diagrams that are independent of any particular domain are used to represent the constructs that are discussed. These are illustrated with worked examples that make the explanations accessible for those who have no experience with ‘complexity’. The primer is intended to provide both an introduction to complex systems constructs for those new to the topics discussed, and also a basis for cross-domain translations for researchers and practitioners wishing to engage with other fields when addressing the systems problems they are working on.

Being able to communicate unambiguously across disciplines and application areas would greatly expand the space of solutions available to all domains in solving problems currently deemed to be too ‘complex’. As a mature complex systems engineering discipline which is by its nature interdisciplinary, Swarm Robotics will no doubt have much to contribute to – and take from – this endeavour.

If you liked this article, you may also want to read these other articles on swarm robotics:

• mROBerTO: The modular millirobot for swarm behavior studies
• Swarms of precision agriculture robots could help put food on the table
• Machines can learn by simply observing, without being told what to look for
• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• Scaling up underwater swarmbot research from tabletop ‘aquarium’ to the Venice Lagoon (CoCoRo Video #50/52)

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Simulation results for a unmanned drone flying over downtown Los Angeles showing the true trajectory (red line), from GPS only (yellow line), and GPS aided with cellular signals (blue line). Image: ASPIN Laboratory at UC Riverside

A team of researchers at the University of California, Riverside has developed a highly reliable and accurate navigation system that exploits existing environmental signals such as cellular and Wi-Fi, rather than the Global Positioning System (GPS). The technology can be used as a standalone alternative to GPS, or complement current GPS-based systems to enable highly reliable, consistent, and tamper-proof navigation. The technology could be used to develop navigation systems that meet the stringent requirements of fully autonomous vehicles, such as driverless cars and unmanned drones.

Led by Zak Kassas, assistant professor of electrical and computer engineering in UCR’s Bourns College of Engineering, the team presented its research at the 2016 Institute of Navigation Global Navigation Satellite System Conference (ION GNSS+), in Portland, Ore., in September. The two studies, “Signals of Opportunity Aided Inertial Navigation” and “Performance Characterization of Positioning in LTE Systems,” both won best paper presentation awards.

Most navigation systems in cars and portable electronics use the space-based Global Navigation Satellite System (GNSS), which includes the U.S. system GPS, Russian system GLONASS, European system Galileo, and Chinese system Beidou. For precision technologies, such as aerospace and missiles, navigation systems typically combine GPS with a high-quality on-board Inertial Navigation System (INS), which delivers a high level of short-term accuracy but eventually drifts when it loses touch with external signals.

Despite advances in this technology, current GPS/INS systems will not meet the demands of future autonomous vehicles for several reasons: First, GPS signals alone are extremely weak and unusable in certain environments like deep canyons; second, GPS signals are susceptible to intentional and unintentional jamming and interference; and third, civilian GPS signals are unencrypted, unauthenticated, and specified in publicly available documents, making them spoofable (i.e., hackable). Current trends in autonomous vehicle navigation systems therefore rely not only on GPS/INS, but a suite of other sensor-based technologies such as cameras, lasers, and sonar.

“By adding more and more sensors, researchers are throwing ‘everything but the kitchen sink’ to prepare autonomous vehicle navigation systems for the inevitable scenario that GPS signals become unavailable. We took a different approach, which is to exploit signals that are already out there in the environment,” Kassas said.

A schematic showing how Zak Kassas, assistant professor of electrical and computer engineering at UC Riverside, and his team are using existing communications signals to complement satellite-based navigation systems like GPS for the control of driverless cars and unmanned aerial vehicles (UAVs). Image: ASPIN Laboratory at UC Riverside

Instead of adding more internal sensors, Kassas and his team in UCR’s Autonomous Systems Perception, Intelligence, and Navigation (ASPIN) Laboratory have been developing autonomous vehicles that could tap into the hundreds of signals around us at any point in time, like cellular, radio, television, Wi-Fi, and other satellite signals.

In the research presented at the ION GNSS+ Conference, Kassas’ team showcased ongoing research that exploits these existing communications signals, called “signals of opportunity (SOP)” for navigation. The system can be used by itself, or, more likely, to supplement INS data in the event that GPS fails. The team’s end-to-end research approach includes theoretical analysis of SOPs in the environment, building specialized software-defined radios (SDRs) that will extract relevant timing and positioning information from SOPs, developing practical navigation algorithms, and finally testing the system on ground vehicles and unmanned drones.

“Autonomous vehicles will inevitably result in a socio-cultural revolution. My team is addressing the challenges associated with realizing practical, cost-effective, and trustworthy autonomous vehicles. Our overarching goal is to get these vehicles to operate with no human-in-the loop for prolonged periods of time, performing missions such as search, rescue, surveillance, mapping, farming, firefighting, package delivery, and transportation,” Kassas said.

In addition to Kassas, graduate students Joshua Morales, Joe Khalife, Kimia Shamaei, Jesse Garcia, and Sonya Ragothaman and undergraduate student Souradeep (Gogol) Bhattacharya contributed to this research, which was funded with support from the Office of Naval Research (ONR). Additional videos of the team’s research can be viewed on the ASPIN Laboratory’s YouTube page.

The UCR Office of Technology Commercialization has filed patents for the inventions above.

This article was originally published by UCR Today.

If you liked this article, you may also enjoy reading:

• Swarms of precision agriculture robots could help put food on the table
• Quadrotor automatically recovers from failure or aggressive launch, without GPS
• Robots Podcast #214: Project Ngulia: from Phone to Drone, with Fredrik Gustafsson
• Oxbotica’s autonomous vehicle software learns its environment and how it changes over time

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

Developed by a team at the University of Toronto, mROBerTO (milli-ROBot TORonto) is designed for swarm-robotics researchers who might wish to test their collective-behavior algorithms with real physical robots. With just a 16 mm x 16 mm footprint, mROBerTO can be used in a multitude of other miniature robot projects too—its modular design allowing for easy addition or removal of components.

mROBerTOs can determine any nearby robots’ relative distances and bearings by their unique robot IDs (up to 150 mm sensing radius) with the use of modulated IR signals. To achieve this, mROBerTO has an all-around coverage of IR phototransistors and emitters that can receive and send modulated IR signals. Furthermore, mROBerTO is equipped with Bluetooth Smart and ANT Wireless communication capabilities, which allow point-to-point and mesh network communication among robots. As for movement, there are two small motors located at the back of the robot that act as wheels and move the robot in a differential-drive configuration.

mROBerTO modular components. Image: University of Toronto

The majority of swarm robotics research has so far been limited to running virtual simulations for the purpose gathering experimental data. In order to test swarm behavior algorithms in a more realistic setting, we have to use physical robots in an environment where real-world constraints act upon the experiments. However, a swarm will often consist of tens—even hundreds—of robots operating together. So robots should ideally be as small as possible if these swarm algorithms are to be tested in a lab setting with limited work-space. And since we need a large number of these robots for experiments, we should aim to use only off-the-shelf components for easy assembly, production, and maintenance. In addition, certain swarm algorithms may require special sensing capabilities in the swarm robots, requiring modular in-design to allow quick exchanges of processing and sensing hardware. mROBerTO was designed to address these specific issues.

At the moment, there are no commercially available mROBerTOs. The BoM costs approximately 60 USD, and the source files of both the hardware and software can be provided by the developers. mROBerTOs were developed using the Eclipse IDE with SEGGER J-Link programmer/debugger but any integrated development environment that supports ARM GCC can be used to develop on mROBerTOs. For wireless debugging and development purposes, we recommend the user getting an nRF51 or nRF52 development board to wirelessly communicate with mROBerTOs.

mROBerTO was developed by Goldie Nejat, Justin Y. Kim, Tyler Colaco, Zendai Kashino and Beno Benhabib at the University of Toronto, Department of Mechanical Engineering. The latest version of the robot was completed in June 2016 and first featured in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) as ‘mROBerTO: A Modular Millirobot for Swarm-Behavior Studies’.

If you liked this article, you may also want to read these other articles on swarm robotics:

• Swarms of precision agriculture robots could help put food on the table
• Machines can learn by simply observing, without being told what to look for
• Raffaello D’Andrea at TED2016: Novel flying machines and swarms of tiny flying robots
• Scaling up underwater swarmbot research from tabletop ‘aquarium’ to the Venice Lagoon (CoCoRo Video #50/52)

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

In this episode, Audrow Nash interviews Fredrik Gustafsson, Professor in Sensor Informatics at Department of Electrical Engineering in Linköping University, about an initiative to reduce poaching in a rhino sanctuary in Ngulia, Kenya. Gustafsson discusses how he first became involved in this project, how he has worked with the rangers to develop solutions, and the future of this work.

Fredrik Gustafsson

Fredrik Gustafsson is professor in Sensor Informatics at the Department of Electrical Engineering, Linköping University, since 2005. He received the M.Sc. degree in electrical engineering 1988 and the Ph.D. degree in Automatic Control, 1992, both from Linköping University. His research interests are in stochastic signal processing, adaptive filtering and change detection, with applications to communication, vehicular, airborne, and audio systems. His work in the sensor fusion area involves design and implementation of nonlinear filtering algorithms for localization, navigation and tracking of all kind of platforms, including cars, aircraft, spacecraft, UAV’s, surface and underwater vessels, cell phones and film cameras for augmented reality. He is a co-founder of the companies NIRA Dynamics (automotive safety, including tire pressure monitoring systems found in more than 20 million, here), Softube (plug-ins for music studios and software solutions for Marshall and Fender), and Senionlab (leading manufacturer of indoor navigation solutions for sm, here).

Links:

• Download mp3 (16.9 MB)
• Project Ngulia website
• A PDF overview of Project Ngulia
• Subscribe to Robots using iTunes
• Subscribe to Robots using RSS