Robohub.org
A computer vision system for mining artistic influence
Figure 1: An example of an often-cited comparison in the context of influence. Left: Diego Velazquez’s Portrait of Pope Innocent X (1650), and, Right: Francis Bacon’s Study After Velazquez’s Portrait of Pope Innocent X (1953). Similar compositions, pose, and subject matter but a different view of the work.
When we look at an image we not only recognize object categories and scene category, but we can also infer various aesthetic, cultural and historical aspects. For example, an expert (or even an average person) can look at a fine art painting and infer information about its style (e.g. Baroque vs. Impressionism), genre (e.g. a portrait or a landscape), and even the artist who painted it. People can also look at two paintings and find similarities between them in terms of composition, color, texture, subject matter, etc. This impressive ability of human perception for learning and judging complex aesthetic-related visual concepts has long been thought to not be a logical process. In our research, however, we tackle this problem using a computational methodology, to show that machines can in fact learn such aesthetic-related concepts.
When we look at an image we not only recognize object categories and scene category, but we can also infer various aesthetic, cultural and historical aspects. For example, an expert (or even an average person) can look at a fine art painting and infer information about its style (e.g. Baroque vs. Impressionism), genre (e.g. a portrait or a landscape), and even the artist who painted it. People can also look at two paintings and find similarities between them in terms of composition, color, texture, subject matter, etc. This impressive ability of human perception for learning and judging complex aesthetic-related visual concepts has long been thought to not be a logical process. In our research, however, we tackle this problem using a computational methodology, to show that machines can in fact learn such aesthetic-related concepts.
Finding influences and connections between artists is an important task for art historians. By doing so, the conversation of art continues and new intuitions about art can be made. An artist might be inspired by one painting, a body of work, or even an entire style of art. Art historians study which artists might influence others by examining the descriptive attributes of art. However, finding influences is a sophisticated process that involves studying the historical, social, and personal context related to the artist and to the art. After all, we will not know if an artist was ever truly inspired by a work unless he or she has said so. However, for the sake of finding connections and progressing through movements of art, a general consensus is agreed upon if the argument is convincing enough. For example, Figure.1 illustrates a commonly cited comparison for studying influence, in the work of Francis Bacon’s Study After Velazquez’s Portrait of Pope Innocent X (1953), where similarity is clear in composition, pose, and subject matter.
Is artistic influence a task that an intelligent agent can measure?
In the past decade there have been impressive advances in developing computer vision algorithms for different object recognition-related problems including: instance recognition, categorization, scene recognition, pose estimation, etc. Although there has been some research on automated classification of paintings, there is almost no research done on computer-based measuring and determining of influence between artists. However, there is limited research on the classification of painting style. Measuring influence is a very difficult task because of the broad criteria for what influence between artists can mean. Besides the scientific merit of the problem from the computational point of view, there are various application-oriented motivations.
With the increasing volumes of digitized art databases on the Internet comes the daunting task of organizing and retrieving paintings. There are millions of paintings present on the Internet; to manage properly the databases of these paintings, it becomes essential to classify paintings into different categories and sub-categories. This classification structure can be used as an index and thus can improve the speed of retrieval process. Also it will be of great significance if we can infer new information about an unknown painting using already existing databases of paintings.
Figure 2: Frederic Bazille’s Studio 9 Rue de la Condamine (left) and Norman Rockwell’s Shuffleton’s Barber Shop (right). The composition of both paintings is divided in a similar way. Yellow circles indicate similar objects, red lines indicate composition, and the blue square represents similar structural element. The objects in the painting – a fire stove, three men clustered, chairs, and window – are seen in both paintings, and are also similarly positioned within them. After browsing through many publications and websites, we concluded that this particular comparison has not previously been made by an art historian.
Although the meaning of a painting is unique to each artist and is completely subjective, it can somewhat be measured by the symbols and objects in the painting. Symbols are visual words that often express something about the meaning of a work as well. For example, the works of Renaissance artists such as Giovanni Bellini and Jan Van-Eyck use religious symbols such as a cross, wings, and animals to tell stories in the Bible. This shows the need for an object-based representation of images. We should be able to describe the painting from a list of many different object classes. By having an object-based representation, the image is described in a high-level semantic as opposed to low-level features such as color and texture, which facilitates suggesting influences based on subject matter.
Paintings do not necessarily have to look alike, but if they do, or if they have reoccurring objects (high-level semantics), then they might be considered similar. If influence is found by looking at similar characteristics within a group of paintings, the importance of finding a good similarity measure becomes prominent. Time is also an essential factor in determining influence: an artist cannot influence another artist in the past, and the linearity of paintings therefor cuts down the possibilities of influence.
The contribution of our work [1,2,3,4] is in exploring the problem of computer-automated suggestion of influences between artists, a problem that has not previously been addressed in a general setting. From a machine-learning point of view, we approach the problem as an unsupervised knowledge discovery problem.
Our methodology is based on three components:
1) studying different representations of painting to determine which is more useful for the task of influence detection;
2) measuring similarity between paintings;
3) studying different measures of similarity between artists.
We collected a comprehensive painting dataset for conducting our study. The data set contains 1710 high-resolution images of paintings by 66 artist spanning the time period of 1412-1996 and containing 13 painting styles. We also collected a ground-truth data set for the task of artistic influences, which mainly contains positive influences claimed by art historians. This ground-truth was only used for the overall evaluation of our discovered/suggested influences, and was not used in the learning process.
We hypothesized that a high-level semantic representation of painting would be more useful for the task of influence detection. We tried to find this useful representation by considering the task of classifying painting style. The hypothesis was that features and representations that are good for style classification (which is a supervised learning problem) would also be good for determining influences (which is an unsupervised problem). We therefor performed a comprehensive comparative study of different features and classification models for the task of classifying painting style among seven different styles [1,3]. The study spanned different levels of state-of-the-art features from low-level to semantic-level and discriminative vs. generative learning methodologies (See Figure 3). More recently, we also studied the use of metric learning to combine various features for the task of style classification [4]. The conclusion of this study confirms our hypothesis that high-level semantic features would be more useful for the task of style classification, and hence useful for determining influences.
Figure 3 An illustrative diagram of the different features and methodologies used for style classification of fine art paintings
Using the right features to represent the painting paves the way for judging similarity between paintings in a quantifiable way. Figure 2 illustrates an example of similar paintings detected by our automated methodology: Frederic Bazille’s “Studio 9 Rue de la Condamine” (1870) and Norman Rockwell’s “ Shuffleton’s Barber Shop” (1950). After browsing through many publications and websites, we concluded, to the best of our knowledge that this comparison has not been previously made by an art historian. The paintings may not look similar at first glance; however, a closer look reveals striking similarity in composition and subject matter, which was detected by our automated methodology.
Measuring similarity between paintings is fundamental to discovering influences, however, it is not clear how painting similarity might be used to suggest influences between artists. The paintings of a given artist can span an extended period of time and can be influenced by several other contemporary and prior artists. Therefore, we investigated several artist distance measures to judge similarity in their work and suggest influences. As a result of these distance measures, we can achieve visualizations of how artists are similar to each other, which we denote by a map of artists. An example of these maps is shown in Figure 4.
After all, we are not asserting truths, but are instead merely scratching the surface and suggesting a possible path towards the difficult task of automated measuring of influence.
References:
[1] R. Aroara and A. Elgammal “Towards Automated Classification of Fine-Art Painting Style: A Comparative Study”, 21st International Conference on Pattern Recognition (ICPR’12), Tsukuba Science City, Japan, November 2012.
[2] Kanako Abe, Babak Saleh and Ahmed Elgammal “An Early Framework for Determining Artistic Influence” The 2nd International Workshop on Multimedia for Cultural Heritage, MM4CH Naples, Italy 2013
[3] Babak Saleh, Kanako Abe, Ravneet Arora, Ahmed Elgammal “Toward Automated Discovery of Artistic Influence” Multimedia Tools and Applications Journal – Springer – 2014
[4] Babak Saleh, Kanako Abe and Ahmed Elgammal “Knowledge Discovery of Artistic Influences: A Metric Learning Approach” The Fifth International Conference on Computational Creativity, ICCC 2014.
tags: c-Arts-Entertainment, computer vision, cx-Research-Innovation
