Robohub.org
 

Are gestures the future of robotic control?


by
29 April 2017



share this:

A few decades ago, touchscreens were impressive, yet clunky pieces of technology reserved for applications that did little more than show off that touchscreens were possible. Today’s touchscreens are commonplace and readily accepted as an easy way to interact with an operating system; they’re accurate, support multi-touch, are comfortable to use, and can even operate in multiple dimensions (pressing hard).

We may not have perfected touchscreens yet, but we’re getting close. Accordingly, engineers and researchers are already speculating about the next phase of UI development, especially for robotics control. So far, the leading candidate is gesture-based control—the use of physical gestures to relay commands.


The idea

The major limitation for touchscreens is the fact that they operate only in two dimensions; the third dimension introduced with force touch is extremely limited. Comparatively, hand gestures and physical movements can operate in three dimensions, and depending on how they’re designed, could feel more natural than swiping and tapping a smartphone screen.

Demand for three-dimensional gestural control is increasing with the onset of virtual reality (VR) and augmented reality (AR) technology; because the digital world we experience will be moving from two dimensions to three dimensions, the controls we use to manage those experiences will also need to change. With enough sophistication, these systems could provide better feedback to the users in control; rather than merely responding with visual feedback like movement or lights, users could feasibly be immersed with physical feedback like vibration or resistance.


Where we stand

Currently, one of the most advanced gestural systems is the Real-Time 3D Gesture Analysis for Natural Interaction with Smart Devices, a project headed by researchers in Linnaeus University in Sweden. However, simpler forms of gesture-based control are already available.

For example, there are projects that use Arduino to create a robot that can respond to four different simple hand gestures (plus a neutral position). Of course, iPhone technology also makes use of some simple “gestures,” such as shaking the phone to undo typing or rotating the phone to experience an immersive 360-degree view of an image or video.


The main obstacles

There are a few obstacles preventing gestures from being solidified as the next landmark in robotics control, however:

  • Gesture complexity. The biggest obstacle described by researchers at Linnaeus University is the sheer complexity of physical gestures; to be effective, their recognition systems need to be able to gather and recognize thousands of tiny data points and interpret complex patterns to “understand” what movement is being attempted. Moreover, this big data-based interpretation needs to happen in real-time—especially for applications like live robotics control, or engagement in a VR-based video game. That demands not only an incredibly intelligent system, but a processor that can operate quickly.
  • Accessibility. Gesture recognition systems would likely be developed to accommodate a “standard” human model, such as a “standard” right human hand. How would the system accommodate somebody whose hand was missing, or who is missing a few fingers? What about left-handed people? And people with Parkinson’s disease, or who are unable to operate their hands with precision control?
  • Applications. For gestures to be accepted as a mainstream way to interact with robotics and virtual environments, robotics and virtual environments need to be available. Virtual reality (VR) technology has been advancing strongly for the past several years and is poised to take a big step forward in user adoption by 2020, but until those user adoption numbers are reliable, the demand for gestural systems is relatively low. Using hand movements and other physical gestures to control a two-dimensional screen, for example, would be largely ineffective.

On paper, gestures seem like the best form of control for the digital devices, robotics, and VR systems of the future. However, there are still many obstacles to overcome before we’re ready for large-scale adoption. Fortunately, researchers are ahead of the curve, already preparing the intelligent gesture-based recognition programs we’ll need when touchscreens become obsolete.



tags: , , , ,


Anna Johansson is a freelance writer, researcher, and business consultant.
Anna Johansson is a freelance writer, researcher, and business consultant.

            AUAI is supported by:



Subscribe to Robohub newsletter on substack



Related posts :

Robot Talk Episode 171 – Programming robot behaviour, with Sebastian Castro

  20 Nov 2026
In the latest episode of the Robot Talk podcast, Claire chatted to Sebastian Castro about developing open-source software for robot task planning and behaviour.

Soft exosuit makes it easier to stand up

Robotic system helps users stand up and walk more efficiently.

Robot swarms trained by trial and error in simulation keep working on real robots

  17 Nov 2026
Researchers have developed a new approach to reinforcement learning that helps robot swarms, allowing behaviours learned in simulation to transfer reliably to real-world robots.

Watch the IROS 2026 plenary and keynote talks

  16 Nov 2026
Catch up on the invited presentations from the International Conference on Intelligent Robots and Systems.

Robot Talk Episode 170 – Robots in the real world, with Daniela Shchegelskaya

  13 Nov 2026
In the latest episode of the Robot Talk podcast, Claire chatted to Daniela Shchegelskaya from the University of Twente about getting robots to operate reliably in safety-critical environments.

AI100 report released: embodied AI highlighted as grand challenge

  11 Nov 2026
Embodied AI is listed as one of the most inspiring open grand challenges.

AIhub coffee corner: Are humanoids the future of embodied AI?

  09 Nov 2026
The AIhub coffee corner captures the musings of AI experts over a short conversation.

Robot Talk Episode 169 – Robot sensing inspired by nature, with Dexter Shepherd

  06 Nov 2026
In the latest episode of the Robot Talk podcast, Claire chatted to Dexter Shepherd from the University of Sussex about using artificial intelligence to help robots understand and adapt to their environment.


↑


AUAI is supported by:







Subscribe to Robohub newsletter on substack




 















©2026.05 - Association for the Understanding of Artificial Intelligence