Watch this service robot predict your organizing preferences

Photo courtesy of University of Freiburg and University of Bonn

How many of us have envisioned intelligent robots operating and assisting us in our homes with everyday tasks? Different users typically have varying preferences and expectations when it comes to tasks, such as tidying up or organizing objects in containers. For example, every person has their own way of organizing groceries or books on their shelves. Moreover, each environment is unique with respect to the items and number of containers that a robot could encounter. This makes it challenging for experts to pre-program robots with strategies for performing such tasks in order to cope with all potential situations and to accommodate different users.

To address this challenge, researchers from the University of Freiburg and University of Bonn have developed a technique to enable service robots to predict personalized preferences of their users. The work is based on collaborative filtering, a data mining paradigm widely used by recommender systems like Amazon or Netflix to make personalized recommendations. For example, an online store could use information about what an individual has purchased previously and predict what products they might buy in the future. Analogously, this work aims at predicting how users prefer to arrange objects (e.g., grocery items, toys, books, etc) in containers (e.g., shelves and drawers) by learning from different users over time. By observing how a user has already organized objects in the environment, the robot can predict the best way to organize new objects without needing to constantly query the user.

To train their system, the researchers gathered data from over 1,200 survey participants using crowdsourcing. Each participant answered questions about their preference for grouping different object classes (e.g. sugar and pepper) together in the same container. Using such data, the approach discovers latent patterns in user preferences with respect to grouping pairs of objects. Accordingly, the robot can then predict the best way to organize objects for a new user based on partial knowledge of his or her preferences. When sorting the objects, the technique also takes into account the available number of containers in the environment. The video below explains the main idea behind the work and illustrates the method using a PR2 robot.

Moreover, the approach is able to utilize information mined from the Web to assist the robot in case it encounters objects it has not learned about yet. The idea is to leverage semantic hierarchies of objects that are mined from online stores, from which the robot can compute similarities between objects. By combining data from multiple online sources into a mixture of experts, the robot can for example learn to group cake mix with flour since they both tend to be found under the “baking” section of online stores. This is done in a way that complements the collaborative filtering method above by considering the preferences of the specific user in question.

Finally, the nature of this collaborative filtering approach allows robots to improve their predictions by learning from users over time. With more data in the system (e.g., perceiving a variety of scenes), robots will get better at learning models of tidying up by discovering patterns shared between users. This will provide future robots with the flexibility to tailor tasks to the unique environments they are in. This concept is illustrated in the video below, which shows how the arrangements predicted by the robot change according to the scene it perceives:

Relevant papers

Nichola Abdo, Cyrill Stachniss, Luciano Spinello, Wolfram Burgard
Collaborative Filtering for Predicting User Preferences for Organizing Objects

Nichola Abdo, Cyrill Stachniss, Luciano Spinello, Wolfram Burgard
Robot, Organize my Shelves! Tidying up Objects by Predicting User Preferences
IEEE International Conference on Robotics and Automation (ICRA), Seattle, USA, 2015

Nichola Abdo Nichola received his undergraduate degree in Mechatronics engineering from the University of Jordan in 2006.

Nichola Abdo Nichola received his undergraduate degree in Mechatronics engineering from the University of Jordan in 2006.