The LUKE Arm in the Shoulder Configuration. (Photo: Mobius Bionics/Business Wire)

Picture the scene: one day you wake up in a hospital with no memory of what happened and you discover in horror that your left hand is no longer there. The doctors say you’ve been in a car accident and from that moment on, you are a “trans-radial mono-lateral amputated person.”  After a slow and painful recovery from the wound, you start looking for solutions to at least partially restore the daily living functions you have lost as you’ve quickly learned that 99% of what is around us was designed to be operated by hands. There must be a way, you say to yourself, in which robotics and rehabilitation medicine can help you.

Upon the suggestion of a friend, you venture into the nearest rehabilitation clinic, are enlisted in a state-supported free program for the upper-limb disabled, and fitted with a mechatronic hand/wrist prosthesis. The prosthesis seamlessly and unobtrusively connects to your body, does exactly what you want it to do whenever you want it to do it, and can happily be worn every morning anew. Well, not exactly: sometimes at the beginning of the program, it will fail, to say, to make a fist the way you wanted. In that case, you would close your fist and your prosthesis would learn in one shot how a fist should look like. After three months of practice, you have essentially forgotten this is a prosthesis. It feels it like a part of your body, and you are socially accepted exactly as before the accident.

And now, for the sad reality.

Every year 1,900 new upper-limb amputations occur in Europe, maintaining a population of such disabled persons hovering around 90,000[1]. What can assistive robotics, rehabilitation science and engineerings do for these people? At the time of writing this article, very little.

While multi-fingered hand prostheses appeared on the clinical market about eight years ago, there is little hope to have a control system which will enable the patients do the things I sketched above. The situation becomes even harder in case of a proximal amputation, for instance, if you have lost the whole arm including the shoulder, not to mention bilateral amputations – losing both upper limbs. Only the most advanced laboratories in the world, and indeed those which have access to very large funding schemas, can try to solve the problem. Currently, there is only one fully-fledged motorized prosthetic arm, the DEKA arm, which has recently received FDA approval in the USA. What it can do in controlled conditions is spectacular; nevertheless, the clinical practice is still far from this, mainly because amputees can hardly control such complex artefacts [2].

Industrial robot or motorbike?

At our Institution, we’ve been concentrating on this issue for around ten years, and that is how to enforce reliable, natural and dexterous control of an upper-limb prosthesis. The human-machine interface involved must detect the subject’s “intent” to perform a certain action; it must do it reliably and stably; and it must convert this knowledge, in real time, into proper control commands, exploiting all the mechatronic capabilities of the prosthesis.

The challenges are formidable and include how to connect the prosthesis to the patient’s body, how to collect the required amount information from it, how to ensure that the system will do what the patient wants, and how to avoid disastrous error by the control system. We have strived to contribute to these open issues by connecting with hospitals and clinics in an effort to understand the real needs of this specific population of patients.

The main bottleneck we have identified is that current prostheses work like industrial robots. They are calibrated upon a specific subject in the beginning, and are then supposed to work seamlessly for the rest of the patient’s life. Due to the variability of signals gathered from human subjects, this is essentially an impossible task.

We need to have a prosthetic artefact – including the socket, the sensors, the control electronics and the prosthesis itself – which connects to the patient more and more tightly day by day. This really is a two-way learning process: on one hand, control systems based upon machine learning can be used in an „incremental“ fashion, periodically adapting to new situations [3]; but on the other hand, our patients naturally learn how to use the prosthesis as time goes by and they produce signals which drift and change during weeks and months [4].

It’s like learning to drive a new motorbike. You need to master the power of the engine, the stability of the steering mechanism, the ability to brake without slipping; you even need to exploit its weaknesses – and this can only happen as you drive it more and more, a little more every day. After an initial training phase, it will bring you exactly where you want, respecting your driving style. Novel situations can occur, in which case a new, small adaptation will happen. In the long run, it will feel like an extension to your body, and will be ready to seamlessly serve to you to its purpose, every single day of your life.

Incremental to interactive to interdisciplinary

We use incremental machine learning exactly in order to foster and exploit this “reciprocal adaptation” effect. Incremental machine learning methods allow for the continual updating of their own model of the world.  In this case, it is the predictive algorithm which turns bodily signals (denoting the subject’s intent) into control commands for the prosthesis. Whenever the subject deems that the control has become unreliable, for instance because she is under stress and her muscle activation is different from what it was beforehand, such a system just needs to be shown the signal pattern corresponding to the desired command in this particular novel situation. This new knowledge is then seamlessly incorporated in the old model and, from this point on, whenever a similar situation or a reasonably similar one occurs, the control will be stable.

This characteristic can actually seem marginal, but it is indeed crucial. There is an essentially unlimited variety of different situations in which the subject might want to, for instance, steadily hold a power grasp. Imagine a shopping afternoon in a mall: the shopping bag must be held at all times while our patient walks, stands, squats; it must be reliably released when she decides to sit down and rest for a while; and the grasp must be stable with respect to all possible wrist, elbow and shoulder rotations and displacement. We claim that no sensible calibration strategy can be put into effect, which will enable the control system to predict all these situations once at the beginning of the rehabilitation program. Rather, it seems to us more natural to let the subject “teach” the prosthesis each time she feels that it required – even to learn completely novel actions [5].

Incremental learning of the control system leads to interaction with the subject, which we consider the practical enforcement to put the patient at the center of the therapy. Additionally, no quick test in laboratory-controlled conditions can ever be enough to check that such an approach works. We need to devise a specific experimental protocol taking into account the particularity of the subject / socket / control system / prosthesis complex, and we need to build apt setups in our labs, enabling our pioneer amputees test these systems in daily-life conditions. This means that we engineers, mathematicians and roboticists must definitely transfer our expertise to the clinics, learn how to interact with end-users and physicians, and start testing mechatronic artifacts on the field since the start [6].

The long-term perspective

With the advent of sockets composed of adaptive biocompatible material and embedding multi-modal sensors, incremental learning could be the key to improving the acceptance of mechatronic prostheses among amputees. The current abandonment rates are astonishingly high and there is no established rehabilitation program in the world as the one we described at the beginning of this article.

In the medium and long term, we shall be able to sell prostheses that work like a motorbike: accompanied by a short instruction manual, they will never cease to adapt to the patient and will push them to obtain an optimal synergy, with minimal supervision by the medical advisors – except for maybe a routine check every now and then. Only in this way will we be able to justify the huge amount of effort that the scientific community puts in this field, and actually improve the life of the large population of disabled persons that we target.

For more information see the original research papers:

Upper-Limb Prosthetic Myocontrol: Two Recommendations
http://journal.frontiersin.org/article/10.3389/fnins.2015.00496/full

Stable myoelectric control of a hand prosthesis using non-linear incremental learning
http://journal.frontiersin.org/article/10.3389/fnbot.2014.00008/full

References

[1] Silvestro Micera, Jacopo Carpaneto and Staniša Raspopovic, Control of hand prostheses using peripheral information. IEEE Reviews in Biomedical Engineering, 3:48–68, 2010.

[2] Jiang, N., Dosen, S., Müller, K.-R., and Farina, D., Myoelectric control of artificial limbs: is there the need for a change of focus? IEEE Signal Processing Magazine, 29(5):149–152, 2012.

[3] Gijsberts, A., Bohra, R., Sierra González, D., Werner, A., Nowak, M., Caputo, B., Roa, M., and Castellini, C., Stable myoelectric control of a hand prosthesis using non-linear incremental learning. Frontiers in Neurorobotics, 8(8), 2014.

[4] Powell, M. A. and Thakor, N. V. A training strategy for learning pattern recognition control for myoelectric prostheses. Journal of Prosthetics and Orthotics, 25(1):30–41, 2013.

[5] Castellini C., Incremental learning of muscle synergies: from calibrating a prosthesis to interacting with it, in Human and Robot Hands – Sensorimotor Synergies to Bridge the Gap between Neuroscience and Robotics, eds Moscatelli A., Bianchi M., editors. (Springer Netherlands; ). Springer series on Touch and Haptic Systems, 2015.

[6] Castellini, C., Bongers, R. M., Nowak, M., and van der Sluis, C. K., Upper-limb prosthetic myocontrol: two recommendations. Frontiers in Neuroscience, 9(496), 2015.

If you enjoyed this article, you may also like:

• Towards building brain-like cognition and control for robots
• Creating a synthetic “second skin” with soft pneumatic actuators
• A helping hand for high-tech firms
• Putting humanoid robots in contact with their environment
• Teaching a brain-controlled robotic prosthetic to learn from its mistakes
• OpenBionics prosthetic hands: Open source, affordable, lightweight, anthropomorphic
• This robotic prosthetic hand can be made for just $1000

See all the latest robotics news on Robohub, or sign up for our weekly newsletter.

The UCLA Biomechatronics Lab develops a language of touch that can be “felt” by computers and humans alike.

Research engineers and students in the University of California, Los Angeles (UCLA) Biomechatronics Lab are designing artificial limbs to be more sensational, with the emphasis on sensation.

With support from the National Science Foundation (NSF), the team, led by mechanical engineer Veronica J. Santos, is constructing a language of touch that both a computer and a human can understand. The researchers are quantifying this with mechanical touch sensors that interact with objects of various shapes, sizes and textures. Using an array of instrumentation, Santos’ team is able to translate that interaction into data a computer can understand.

The data is used to create a formula or algorithm that gives the computer the ability to identify patterns among the items it has in its library of experiences and something it has never felt before. This research will help the team develop artificial haptic intelligence, which is, essentially, giving robots, as well as prostheses, the “human touch.”

To help HAPTIX performers to quickly and effectively conduct their research, DARPA is providing each team with open source simulation software in which to test their designs. The software includes a variant of the DARPA Robotics Challenge Simulator from the June 2013 Virtual Robotics Challenge, the first stage of the DARPA Robotics Challenge. Source: DARPA.

The Gazebo team has been hard at work setting up a simulation environment for the Defense Advanced Research Projects Agency (DARPA)’s Hand Proprioception and Touch Interfaces (HAPTIX) program. The goal of the HAPTIX program is to provide amputees with prosthetic limb systems that feel and function like natural limbs, and to develop next-generation sensorimotor interfaces to drive and receive rich sensory content from these limbs. Managed by Dr. Doug Weber, HAPTIX is being run out of DARPA’s Biological Technologies Office (BTO).

As the organization maintaining Gazebo, OSRF has been tasked with extending Gazebo to simulate prosthetic hands and test environments, and develop both graphical and programming interfaces to the hands. OSRF is officially releasing a new version of Gazebo for use by HAPTIX participants. Highlights of the new release include support for OptiTrack motion capture system; the NVIDIA 3D vision system; numerous teleoperation options including the Razer Hydra, SpaceNavigator, mouse, mixer board and keyboard; a high-dexterity prosthetic arm; and programmatic control of the simulated arm using Linux, Windows and MATLAB. More information and tutorials are available at the Gazebo website. Here’s an overview video:

“Our track record of success in simulation as part of the DARPA Robotics Challenge makes OSRF a natural partner for the HAPTIX program,” according to John Hsu, Chief Scientist at Open Source Robotics Foundation. “Simulation of prosthetic hands and the accompanying GUI will significantly enhance the HAPTIX program’s ability to help restore more natural functionality to wounded service members.”

Gazebo is an open source simulator that makes it possible to rapidly test algorithms, design robots, and perform regression testing using realistic scenarios. Gazebo provides users with a robust physics engine, high-quality graphics, and convenient programmatic and graphical interfaces. Gazebo was the simulation environment for the VRC, the Virtual Robotics Challenge stage of the DARPA Robotics Challenge.

DARPA’s Hand Proprioception and Touch Interfaces (HAPTIX) program aims to develop fully implantable, modular and reconfigurable neural-interface systems that would enable intuitive, dexterous control of advanced upper-limb prosthetic devices. Source: DARPA.

This project also marks the first time Windows and MATLAB users can interact with Gazebo, thanks to our new cross-platform transport library. The scope is limited to the HAPTIX project, however plans are in motion to bring the entire Gazebo package to Windows.

Teams participating on HAPTIX will have access to a customized version of Gazebo that the Johns Hopkins University Applied Physics Laboratory Modular Prosthetic Limb (MPL) developed under the DARPA Revolutionizing Prosthetics program, as well as representative physical therapy objects used in clinical research environments.

More details on HAPTIX can be found in the DARPA announcement.

Knee orthosis as worn by first author Mike Tucker (photo: ReLab, ETHZ and Alain Herzog).

Much has been made of the numerous advances in robotic prosthetics and orthotics (P/O) over recent years, and the question of how to control these devices so that they work in accordance with the intention of the user is a continuing dilemma for roboticists. 

A team from four labs within NCCR Robotics, across ETH Zurich and EPFL (ReLab, ETH Zurich; LSRO, EPFL; SMS, ETH Zurich and CNBI, EPFL) have recently published a joint paper in the Journal of NeuroEngineering and Rehabilitation, in which 10 experts from the field review the state of the art in control approaches for active lower limb P/Os. They argue that for P/Os to be fully viable and to advance further, they must be treated as part of a framework whereby the control system becomes integrated with the user’s sensorimotor system.

Traditionally, the fields of orthotics and prosthetics have been viewed separately, with hardware and controllers developed with a specific portion of the body in mind (i.e. knee, ankle and hips). By taking a broad survey that includes research for all joints of the lower limbs across the different fields (rather than just looking at a small subset), it is hoped that future developments can blur the lines between fields and create technologies that can ultimately restore walking to those with physical or neurological impairments.

This open access review pieces together where the state of the art is now and what work still needs to be done, providing valuable background about the field.

The authors behind the paper have been working to enhance communication between research groups, and to promote a more holistic approach to P/O devices. One of the co-authors is organizing next year’s Cybathlon, where teams comprised of bionic technology developers and a pilot will compete in one of six races.The competition’s ultimate aim is to increase discussion between academia, industry and end users through friendly competition.