Next Article in Journal
Bioactive Properties of Breads Made with Sourdough of Hull-Less Barley or Conventional and Pigmented Wheat Flours
Previous Article in Journal
Performance Evaluation of Stewart-Gough Flight Simulator Based on L1 Adaptive Control
Previous Article in Special Issue
Designing Procedure Execution Tools with Emerging Technologies for Future Astronauts
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

The Cognition of Audience to Artistic Style Transfer

1
Graduate School of Creative Industry Design, National Taiwan University of Arts, New Taipei City 22058, Taiwan
2
Department of Crafts and Design, National Taiwan University of Arts, New Taipei City 22058, Taiwan
*
Author to whom correspondence should be addressed.
Submission received: 31 December 2020 / Revised: 25 March 2021 / Accepted: 5 April 2021 / Published: 6 April 2021
(This article belongs to the Special Issue User Experience for Advanced Human–Computer Interaction)

Abstract

:
Artificial Intelligence (AI) is becoming more popular in various fields, including the area of art creation. Advances in AI technology bring new opportunities and challenges in the creation, experience, and appreciation of art. The neural style transfer (NST) realizes the intelligent conversion of any artistic style using neural networks. However, the artistic style is the product of cognition that involving from visual to feel. The purpose of this paper is to study factors affecting audience cognitive difference and preference on artistic style transfer. Those factors are discussed to investigate the application of the AI generator model in art creation. Therefore, based on the artist’s encoding attributes (color, stroke, texture) and the audience’s decoding cognitive levels (technical, semantic, effectiveness), this study proposed a framework to evaluate artistic style transfer in the perspective of cognition. Thirty-one subjects with a background in art, aesthetics, and design were recruited to participate in the experiment. The experimental process consists of four style groups, including Fauvism, Expressionism, Cubism, and Renaissance. According to the finding in this study, participants can still recognize different artistic styles after transferred by neural networks. Besides, the features of texture and stroke are more impact on the perception of fitness than color. The audience may prefer the samples with high cognition in the semantic and effectiveness levels. The above indicates that through AI automated routine work, the cognition of the audience to artistic style still can be kept and transferred.

1. Introduction

In recent years, more significant progress has been made in Artificial Intelligence (AI) research, including the area of art creation. The continuous development of technology has promoted increasing interaction between art and AI. The neural network as a popular machine learning method (ML) can also be applied in artistic stylization, called neural style transfer (NST). There have been many methods proposed by computer science researchers to get a better conversion effect. However, when methods of NST are adopted to transfer artistic style, evaluating their results is so complex that only employing quantitative methods commonly used in the computer graphics community is not enough [1,2]. So far, there are few evaluations of machine learning artistic style from human cognitive factors. The artwork encodes from the artist’s inner expression to the outer form, while the audience decodes it from external recognition to inner feeling. Even at the intersection of AI and art, human cognition can still feedback to the optimization direction and application mechanism of the AI generator model in the field of art.
This study is intended to provide researchers who focus on AI application in art with a framework of how to obtain the art cognitive effects for machine-generated results. As shown in Figure 1, this study can be divided into three sections. In Section 1, a literature review was used to explore the communication matrix for evaluating artistic style transfer. Section 2 invited nine experts in artistic and/or aesthetic backgrounds to select painting samples according to cognitive attributes and choose the suitable method for transferring abstract art style. In Section 3, the data collected from the grouping experiments of four art schools were used for analysis and discussion. Finally, the conclusion of this study was given.

1.1. Application of NST into Art

Neural Style Transfer (NST) is a type of algorithm that uses deep neural networks to render a content image in different styles. In the early research stage of Non-Photorealistic Rendering (NPR), many stylization algorithms were designed to automatically turn photos into synthetic artworks by stimulating artists to create art [3,4]. However, these patch-based methods’ limitation is that only low-level image features are adopted while often failing to capture image structures effectively [5]. Gatys et al. [6] are the first to apply Convolutional Neural Networks (CNNs) to transfer famous painting styles into photography. Since the algorithm of Gatys et al. does not have any explicit restrictions on the type of style images and needs ground truth results for training, it breaks the constraints of previous approaches. In this case, it ushered in the new field called Neural Style Transfer (NST). The method put forward by Gatys et al. based on the Gram-matrix-matching-based style representations requires a slow iterative optimization process, which is computationally expensive. Still, it is usually regarded as the gold-standard method in the community of NST [5,7].
During the follow-up years, there are a series of optimization methods for NST that can fall into paired and non-paired methods. To be specific, the former is to transfer the style of one image to another, while the latter is to learn the style of multiple images in a dataset and then transfer it. As the experiment in this study tends to provide each painting and its corresponding conversion as a paired stimulus, it is more suitable to choose a pairing method. In addition to Gatys’ original algorithm, this type of methods consists of several representative algorithms, such as AdaIN [7] and WCT [8] for improving speedy, as well as MST [9] and SEMST [10] which focuses on the matching of semantic patterns between content images and style images. Inspired by the instance normalization, Huang and Belongie [7] proposed a new interpretation by normalizing feature statistics to adaptive instance normalization (AdaIN), which first enables arbitrary style transfer in real-time. Like Huang and Belongie’s method but replacing the AdaIN layer between the encoder and the decoder, Li et al. [8] embedded a pair of feature transforms, whitening, and coloring (WCT) into an image reconstruction network. However, these algorithms based on feed-forward networks failed to consider the feature details and local structure and suffered from wash-out artifacts [5,9]. To match the content structure, Zhang et al. [9] first introduced a multimodal style transfer method called MST. He considered the semantic content structure and the matching with style patterns by using K-means to split style patterns from style image and combine it with content images via graph-cut. Recently, Chen [10] analyzed the shortcomings of MST, such as the inability to consider the structural information of the content image or the loss of the style characteristics due to the feature’s high dimensional and low-resolution characteristics space and the characteristics of the graph cut method. In further research, he proposed a structure-emphasized multimodal style transfer (SEMST) that can flexibly match the content cluster and the style cluster even if the number of clusters differs. Although there are many other papers and studies in the NST field, the purpose of this study is to explore the differences in audience cognition to the results of artistic style transfer, rather than review all methods.
The way to compare different NST algorithms includes both qualitative and quantitative methods. The main quantitative evaluation metrics focus on transfer speed and loss comparison [5,9]. Regarding the qualitative evaluation, the observation of style features and user study by voting are usually used. In the experiment of Li et al. [8], they invited 80 subjects to vote for their favorite result for each method. Then, the study shows the WCT received more votes than other methods like AdaIN and Gatys et al. Yet, as the qualitative assessment of Zhang et al. [9], the result showed that the MST obtained the highest percentage of votes, while Gatys et al. ranked second above AdaIN and WCT. Due to the varied choices of samples and users, the results by using the qualitative methods are also different [8,9]. The evaluation of response to artistic imagery is a complex process that includes both perceptions of shape and evaluating aesthetics and other properties of images [1]. To obtain accurate and stable results from the qualitative assessment, a reasonable evaluation mechanism needs to be constructed.
The field of NST has received increasing attention from academics and industry in recent years, resulting in some related industrial applications, such as the websites Ostagram [11] and Deep Dream Generator [12], the mobile application Prisma [13], and even film production [14]. People can use those platforms to create some interesting images for pleasure or spawning creativity. Machine-based intelligence learning from human intelligence will also be inspired by human insight. Therefore, the evaluation results from audience cognition can provide the algorithm with some suggestions for further optimization and the application regarding the intersection with art.

1.2. Attributes of Artistic Style

In the visual arts, style is a distinctive manner that permits the grouping of works into related categories [15], and it refers to similar critical features for recognition, such as characteristic subject matter or materials, distinctive ways of drawing or applying paints, preferences for specific color combinations, brushstrokes, distortion, and exaggeration. These visual pieces of information can direct the viewer’s attention and affect the viewer’s perception [16,17]. The perception of visual art is a complex performed by the brain to perceive different elements’ features in the painting [18,19]. The various painting attributes, such as colors, shapes, and boundaries, are selectively redistributed to the brain for processing [20]. There are three critical features in the receptive field: position, shape, and specificity, similar to modern art, with its accent on simplification [21].
Artistic visual styles such as Fauvism, Expressionism, and Cubism in the early 20th century have the commonality of abstraction and expression. Meanwhile, they have distinctive attributes that support the grouping of artworks into related art movements [22,23,24,25]. Both Fauvism and Expressionism use pure colors and subject distortion. However, unlike the thick coating method or squeezing the paint directly from the tube in Fauvism, Expressionism is characterized by broad brush strokes to exaggerate artists’ inner emotions and feelings [23,26]. Cubism is known for its reduction of subjects into geometric shapes to produce a more three-dimensional perspective. It is a calm and reflective experimental art that balances representation and abstraction and abandons colors in favor of an almost monochrome palette [22]. Different from the modernist style, realism such as the Renaissance involves several techniques that make the subjects and backgrounds look like what they would be in real life. As Renaissance artists began using the scientific perspective and new ways of painting light, their paintings became more meaningful and realistic [27].

1.3. Communication between Audience and Artwork

Art is the media that provides an understanding of visual communication [28]. As a communication channel, artwork can connect the artist and audience through the creator’s encoding and the audience’s decoding [28,29,30]. The interplay between the internal (cognitive) representation and the external (physical) representation is a fascinating problem in cognitive psychology, art, science, and philosophy [18]. Each of us approaches an art object with a significantly different perspective because of our unique personal social experiences [18,31]. Whoever can feel the beauty of colors and forms has understood non-objective painting [32]. The aesthetic experience involves the processing of stylistic information closely related to the artist’s movements [31,33] and simultaneously performing the movements can enhance the preference for paintings of the corresponding style [34]. In abstract paintings, the ideas, emotions, and visual sensations are expressed solely through lines, shapes, color, and textures that have no symbolic significance [35]. According to people’s feelings, color may be warm or cold, and cheerful or somber [20]. Audiences can perceive the painter’s actions by merely observing the brushstroke of the painting [36]. Texture includes all painting areas enriched by combing lines, shapes, tones, and colors [16,35]. Therefore, from the perspective of cognition, we can learn more about the process of creation [37].
In the symbol communication mode, cognition can be divided into three levels: technical, semantic, and effectiveness levels [38]. To evaluate the artwork, it is necessary to find out the cognitive factors affecting them. Lin et al. [39] proposed a communication matrix that combined the cognition and communication theory, which integrated into three dimensions for evaluating artwork. There are three stages for artists to express their thought through artworks: inspiration, ideation, and implementation. For audiences, the three stages of experience include aesthetic, meaning, and emotional. Shusterman [40] and Bergeron and Lopes [41] suggest that an aesthetic experience is featured with an evaluative dimension involving semantic and affective aspects to confer the aesthetic quality to the experience. However, in the literature, there is little evidence of discussing the evaluation concerning the results of artistic style transfer by computer algorithms from the aspects of aesthetic cognition. In the creative process of AI-Art, the artists choose AI algorithms according to their intentions for creating the artwork, and audience acceptance is a critical defining step to decide whether it is “art” [42]. Studying the factors that affect artistic cognition from the perspective of the audience can help build a bridge between artists and the audience [43,44].

1.4. Research Purpose

The perception of artistic style is the complex processing of visual information. The audience is a vital participant of art, especially when AI technology is involved in the art field. The artistic style transfer methods by using neural networks have improved efficiency and effectiveness in Non-Photorealistic Rendering. If trying to further optimize industry applications, the audience’s perception of the style transfer results should be considered. Technology comes from humanity, so the integration of Hi-tech and Hi-touch is necessary for art creation and product design, especially in the 21st-century digital technology world [42].
In this study, the style transfer results of four painting schools were evaluated from the perspective of audience cognition. Through four cognitive experiments, including three abstract schools of modernism in the early 20th century (Fauvism, expressionism, cubism) and Realism (Renaissance), it tends to explore whether the audience can still distinguish the different art schools and keep the cognition from technical, semantic and effectiveness levels. The main factors affecting the audience’s cognition in artistic style will be discussed to understand the difference before and after style transfer. The result of this study can be used as an optimization suggestion to improve the performance of NST methods closer to human cognition.

2. Materials and Method

2.1. Pre-Experiment

To select the appropriate algorithm for this experiment, the method of Gatys et al. [6], AdaIN [7], WCT [8], MST [9], and SEMST [10] were used to transfer four samples’ style to the same content image. Gatys et al. [6]’s seminal work synthesized stylization through an iterative optimization scheme. AdaIN [7] and WCT [8] are methods using feed-forward networks to enhance efficiency to a great extent. MST [9] and SEMST [10] solved the above methods’ artifact problem by matching the semantic pattern in style and content images via K-means. It should be mentioned that the codes employed here come from GitHub provided by the author or written according to papers. The links are shown in Table 1.
The five methods were adopted with their default parameters to create stylized images by learning the features of four schools’ representative paintings. A photograph of Guangong in opera was used as the content image for its rich semantic involving character, decorative patterns, and background. Then, nine experts with artistic and/or aesthetic backgrounds were invited to vote for the best method through perception, and the best method in each art school was marked with the blue line in Figure 2. The feedback from experts is that the results of Fauvism and Expressionism can keep the balance between style and content by using the method of Gatys et al. method. Although the use of WCT in the former two schools produced serious distortions and lacked content structure, the result of Cubism still retained the feeling of geometric fragments. For the Renaissance style, SEMST is more semantic and more suitable for transferring realistic style. However, the sense of abstract style was lost in the other three schools of modernists. Based on the purpose of this research, the method of Gatys et al. was selected to produce all the stimuli of experiments.

2.2. Stimuli

Based on previous studies [29,30,39,43,44], the artwork can be evaluated from technical, semantic, and effectiveness levels. The technical level focus on the visual elements. The semantic level means letting the audience accurately understand the meaning of the artwork through his/her realization. The effectiveness level concerns the inner feeling of the audience on the emotional expression of the artist. Color, stroke, and texture as important factors of painting style can still be evaluated from the above three levels. Therefore, the matrix for evaluating the cognitive effect of artistic style transfer through machine learning was reconstructed, as shown in Figure 3.
The nine experts who are mentioned above continued to pick up paintings with typical color, stroke, and texture from wikiart.org and then chosen one sample matching each cognitive attribute in the evaluation matrix, and they are Color-technical, Color-semantic, Color-effectiveness, Stroke-technical, Stroke-semantic, Stroke -effectiveness, Texture-technical, Texture-semantic, and Texture-effectiveness. After obtaining nine paintings of each school, the experts further describe their characteristics according to the representative attribute of each work, which was used as the questionnaire items. For example, according to the feature of the F1 sample in the attribute it represents, the Color-technical item in the Fauvism group was described to be “complementary”, which is shown in Table 2. Theoretically, the F1 sample should get the highest score on this option. Besides, a Mona Lisa portrait was inserted into each group as the reference sample, and the score should be higher in the Renaissance group. In the Fauvism, Expressionism, and Cubism groups, the Mona Lisa should have significant cognitive differences from other samples due to different periods and styles.
The content image, a photograph of a character in opera called Guangong, was combined with the style of paintings using the method of Gatys et al., shown in Table 2. Before transfer, all of the style and content images were resized to 600 px high. Furthermore, the default parameters suggested by the authors and the pre-trained VGG19 network were also adopted. In layer options, style reconstructions used the conv1_1, conv2_1, conv3_1, conv4_1, and conv5_1 and content reconstructions used the conv4_2 layer. The process was initialized with the content image instead of random noise in the paper to exclude random results. For the style setting, the ratio content_weight/style_weight was 1 × 10−3. Experiments conducted by Gatys et al. [7] prove that this ratio can better balance content and style. Besides, 1.0 as the scale of the style was used to keep the proportional transfer of texture. The number of iterations was 1000 as default parameters because there is no obvious change after increasing more.

2.3. Participants

All of the study participants were artists, professors, or Ph.D. with art and/or design background. A total of 31 subjects (17 male and 14 female) participated in the experiment. The mean age of the male participants was 48.29 (SD = 9.45) years. The mean age of the female participants was 42.71 (SD = 10.91) years.

2.4. Experimental Procedure

This study consisted of four groups of experiments. The stimuli of each group contained nine samples representing nine cognitive attributes of one art school, as well as the corresponding results combining with their artistic style and content images by using the method of Gatys et al. The famous painting Mona Lisa was inserted into each group as the reference sample to find whether there is a difference in its cognition from others.
The researcher first introduced the process to the subjects. Then, starting from Fauvism, there were ten sets of stimuli, including nine pairs of paintings belonging to Fauvism and one pair of Mona Lisa. Each painting and its transfer result were projected onto the screen simultaneously. Subjects had 2–3 min to complete the evaluation of the samples. For every group, the experiment lasted for less than 1 h with at least one week apart to avoid learning effects, and the duration was four weeks.
The subjects were informed of the experimental task and then asked to rate each pair of samples according to their subjective opinion. The dependent variables were presented in the questionnaire on the participants’ perception. In the first part, the degree of fitness for style attributes involving the color, stroke, texture, and overall fitness between the original painting and transfer result, with a score range from 0 to 100%—the higher the score, the higher the fit. The second part explored the degree of fitness on the nine cognitive attributes in each pair of stimuli. The attribute category includes “Color-technical”, “Color-semantic”, “Color-effectiveness”, “Stroke-technical”, “Stroke-semantic”, “Stroke-effectiveness”, “Texture-technical”, “Texture-semantic” and “Texture-effectiveness”. The item description of each attribute in the questionnaire was provided by the nine experts mentioned above, based on each sample’s feeling. A 5-point Likert scale was used to score the fitness of nine cognitive attributes of each pair from 1 (“Very unfit”) to 5 (“Very fit”). Finally, the subjects were asked to choose a favorite sample and a least favorite sample respectively in each group.

2.5. Statistical Analysis

Firstly, one-way ANOVA was adopted to test whether the style attributes and cognitive attributes of ten pairs of painting factors per group were significant (significance level was set at 0.05). For programs reaching the significance level, Duncan was used to figure out whether there was a significant difference among ten averages. In addition, a Pearson correlation coefficient was also employed to investigate the potential correlation between overall fitness and three style factors, including color, stroke, and texture.

3. Results and Discussion

3.1. Results of Descriptive Statistics and ANOVA Analysis

According to the results of variation analysis on the first part of the questionnaire as shown in Table 3, after the same subjects viewed the four groups of paintings, they displayed a significant difference in their feelings for style attributes (i.e., “color fitness”, “stroke fitness”, “texture fitness”, and “overall fitness”). Different from the results in Fauvism, Expressionism, and Cubism that the sample of Mona Lisa independently exists in the lowest score group, it was classified into the same group as other samples in Renaissance. The result indicates that subjects could still distinguish the different schools of painting even after style transfer. Otherwise, similar to the literature discussion, the method based on iterative optimization and feed-forward networks lacking to consider the feature details and content structure [5,9], the algorithm of Gatys et al. is more suitable to transfer abstract style than realism style.
It is worth noting that the samples which belong to stroke and texture attributes in Fauvism, Expressionism, and Cubism got a higher score in their fitness evaluation. However, among the Renaissance stimuli, the samples with color characteristics scored higher than others. The relevant scores have been marked with red lines in Table 3.
As the results of the evaluation for cognitive levels in part two, some results of samples with certain cognitive features can still be perceived by subjects, as shown in Table 4. The score belongs to style attributes of color, strokes, and texture was marked by the red line, as well as the gray background color emphasizes the score of samples on their corresponding attributes. Furthermore, the highest score on the corresponding attribute is marked by the blue dashed line, which is mainly distributed in stroke and texture attributes in Fauvism, Expressionism, and Cubism. However, in the Renaissance group which lacked stroke and texture features, the participants had a stronger perception of color. The reference sample (Mona Lisa) obtain the lowest scores in all cognitive attributes of Fauvism, Expressionism, and Cubism. However, in the style that it belongs to, Mona Lisa had a higher ranking in various indicators of Renaissance, further explaining that the subjects can distinguish the different schools of painting even after style transfer.

3.2. Pearson Correlation Coefficient

A strong positive association was established using a Pearson correlation coefficient between overall fitness and three style attributes including color, stroke, and texture, shown in Table 5. On the one hand, it means that color, stroke, and texture can be used as factors to evaluate the algorithm of artistic style transfer. In addition, stroke and texture have a higher correlation with the overall fit than color. Hence, compared with color, the simulation stroke and texture in artistic style transfer play a more critical role and should be paid more attention.

3.3. The Favorite and Least Favorite Paintings

To further determine the critical point that affects participants’ preference, in this study, subjects were invited to choose their favorite and least favorite samples respectively from each group. Table 6 shows the proportion of each painting being selected as a favorite. The stimulus ranking top in the preference vote had higher scores of overall fitness. Besides, their score in corresponding cognitive attributes is the highest. Based on the previous discussion, users have a higher perception of abstract style strokes and textures. The samples that can accurately transfer the cognitive features of strokes and textures are more welcome. However, as the realistic style without evident stroke and texture features, the transition result of Renaissance received more attention in color. In Figure 4, the most favorite sample of each group was listed. Subjects can feel the corresponding cognitive attribute of P9, P8, P5, and P3: Cute Pattern, Wantonly, Bright & Fast, and Lively.
The proportion of each painting being selected as the least favorite was shown in Table 7. Subjects voted P10 as their least favorite sample in all groups except Renaissance. In the Renaissance group, there was no significant difference in cognition because the Mona Lisa portrait belonged to the same style as other samples. Therefore, it can be further explained that the subjects can distinguish different artistic styles even reproduced by machine learning. In the Fauvism, expressionism, and Cubism groups, the image with the lowest score was also the subjects’ least favorite samples, which showed a positive correlation between the preference and cognitive fitness. From Figure 5, it can be found that the two least favorite samples also lost the content’s details seriously. The more details lost, the fewer subjects like it. The wash-out artifact problem in the feed-forward networks-based method [5,9] causes the bad transfer result of realistic style.

4. Conclusions

The experimental results show that subjects could still distinguish the different art schools and keep the cognition from technical, semantic, and effectiveness levels. The cognitive attributes have a strong relationship with the overall feeling and audience’s preference. It supported that the matrix based on style attributes and cognitive levels can be used for subjective evaluation of the NST method. Among the three style attributes, the fitness of stroke and texture have a higher degree of correlation with overall fitness than color. However, for the Renaissance featured with the style without obvious stroke and texture, the audience turned their attention to the feeling of color and the painting’s content. Therefore, which style attribute should be strengthened is based on the style sample’s features during the NST encoding. Besides, the sample whose semantic or effectiveness level was accurately perceived usually has a higher preference, so the two cognitive levels are the key to improve the audience’s preference. The correlation between style feature and their corresponding cognitive effects should be further discussed.
This study presented a framework that can be used as the fundamental research for subject evaluating AI algorithms in the art field. However, this study still had several limitations. Firstly, as the classification of painting styles is complex and subjective, there are some uncertainties in the choice of painting samples and the corresponding evaluation criteria. Therefore, in the future study, the two samples with the highest scores in each group will be selected and evaluated by a large number of subjects without professional background for further discussion. Secondly, the methods of comparison in this study are difficult to fully cover the newest technologies with the rapid development in this field. The algorithms selected in this study only involve the typical paired methods, while failing to include the non-paired ways that represent the features of datasets.

Author Contributions

Y.L. performed the writing conceptualization, formal analysis, original draft, editing investigation, resources; C.-L.L. contributed to the methodology, review; P.-H.L. contributed review; R.L. contributed to review, editing. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Data sharing not applicable.

Acknowledgments

The authors would like to appreciate the experts and participants that took part in the experiments. We also thank the researchers who shared the source code on GitHub.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Hertzmann, A. Non-Photorealistic Rendering and the Science of Art. In Proceedings of the 8th International Symposium on Non-Photorealistic Animation and Rendering (NPAR 2010), Annecy, France, 7–10 June 2010; pp. 147–157. [Google Scholar]
  2. Isenberg, T.; Neumann, P.; Carpendale, S.; Sousa, M.C.; Jorge, J.A. Non-Photorealistic Rendering in Context: An Observational Study. In Proceedings of the 4th International Symposium on Non-Photorealistic Animation & Rendering (NPAR 2006), Annecy, France, 5–7 June 2006; pp. 115–126. [Google Scholar]
  3. Haeberli, P. Paint by numbers: Abstract image representations. In Proceedings of the 17th Annual Conference on Computer Graphics and Interactive Techniques, Dallas, TX, USA, 6–10 August 1990; pp. 207–214. [Google Scholar]
  4. Hertzmann, A. Painterly rendering with curved brush strokes of multiple sizes. In Proceedings of the 25th Annual Conference on Computer Graphics and Interactive Techniques, Orlando, FL, USA, 19–24 July 1998; pp. 453–460. [Google Scholar]
  5. Jing, Y.; Yang, Y.; Feng, Z.; Ye, J.; Yu, Y.; Song, M. Neural Style Transfer: A Review. IEEE Trans. Vis. Comput. Graph. 2020, 26, 3365–3385. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  6. Gatys, L.A.; Ecker, A.S.; Bethge, M. A Neural Algorithm of Artistic Style. arXiv 2015, arXiv:1508.06576. [Google Scholar] [CrossRef]
  7. Huang, X.; Belongie, S. Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization. In Proceedings of the IEEE International Conference on Computer Vision, Venice, Italy, 22–29 October 2017; pp. 1510–1519. [Google Scholar]
  8. Li, Y.; Fang, C.; Yang, J.; Wang, Z.; Lu, X.; Yang, M.-H. Universal Style Transfer via Feature Transforms. In Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 4–9 December 2017; pp. 385–395. [Google Scholar]
  9. Zhang, Y.; Fang, C.; Wang, Y.; Wang, Z.; Lin, Z.; Fu, Y.; Yang, J. Multimodal Style Transfer via Graph Cuts. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV 2019), Seoul, Korea, 27 October–2 November 2019; pp. 5942–5950. [Google Scholar]
  10. Chen, C. Structure-Emphasized Multimodal Style Transfer; Master-Tokyo Institute of Technology: Tokyo, Japan, 2020. [Google Scholar]
  11. Ostagram. Available online: https://www.ostagram.me (accessed on 24 March 2021).
  12. Deepdreamgenerator. Available online: https://deepdreamgenerator.com (accessed on 24 March 2021).
  13. Prisma. Available online: https://prisma-ai.com (accessed on 24 March 2021).
  14. Joshi, B.; Stewart, K.; Shapiro, D. Bringing impressionism to life with neural style transfer in come swim. In Proceedings of the ACM SIGGRAPH Digital Production Symposium, Los Angeles, CA, USA, 29 July 2017; pp. 1–5. [Google Scholar]
  15. Fernie, E. Art History and Its Methods: A Critical Anthology; Phaidon: London, UK, 1995; pp. 27–28. [Google Scholar]
  16. Getlein, M. Living with Art, 11th ed.; McGraw-Hill Education: Singapore, 2015; p. 77. [Google Scholar]
  17. Chan, C.-S. Style and Creativity in Design; Springer International Publishing: New York, NY, USA, 2015; pp. 114–115. [Google Scholar]
  18. Solso, R.L. Cognition and the Visual Arts; MIT Press: Cambridge, MA, USA, 1996; pp. 34–36. [Google Scholar]
  19. Shamir, L.; Macura, T.; Orlov, N.; Eckley, D.M.; Goldberg, I.G. Impressionism, expressionism, surrealism: Automated recognition of painters and schools of art. ACM Trans. Appl. Percept. 2010, 7, 1–17. [Google Scholar] [CrossRef]
  20. Steenberg, E. Visual Aesthetic Experience. J. Aesthet. Educ. 2007, 41, 89–94. [Google Scholar] [CrossRef]
  21. Zeki, S. Inner vision: An exploration of art and the brain. Q. Rev. Biol. 2002, 6, 275–276. [Google Scholar]
  22. Golding, J. Cubism. In Concepts of Modern Art: From Fauvism to Postmodernism; Stangos, N., Ed.; Thames and Hudson: London, UK, 1994; pp. 50–77. [Google Scholar]
  23. Lynton, N. Expressionism. In Concepts of Modern Art: From Fauvism to Postmodernism; Stangos, N., Ed.; Thames and Hudson: London, UK, 1994; pp. 30–49. [Google Scholar]
  24. Whitfield, S. Fauvism. In Concepts of Modern Art: From Fauvism to Postmodernism; Stangos, N., Ed.; Thames and Hudson: London, UK, 1994; pp. 11–29. [Google Scholar]
  25. Beckett, W. The Story of Painting; DK Publisher: London, UK, 1994; pp. 330–346. [Google Scholar]
  26. Chilvers, I.; Zaczek, I.; Welton, J.; Bugler, C.; Mack, L. Art That Changed the World; DK Publisher: London, UK, 2013; p. 314. [Google Scholar]
  27. D’Elia, U.R. Painting in the Renaissance; Crabtree Publishing: New York, NY, USA, 2009; pp. 4–5. [Google Scholar]
  28. Barthes, R. Elements of Semiology; Hill and Wang: New York, NY, USA, 1977; pp. 39–42. [Google Scholar]
  29. Fiske, J. Introduction to Communication Studies, 3rd ed.; Routledge: London, UK, 2010; pp. 5–6. [Google Scholar]
  30. Jakobson, R. Language in Literature; Harvard University Press: Cambridge, MA, USA, 1987; pp. 100–101. [Google Scholar]
  31. Leder, H.; Nadal, M. Ten years of a model of aesthetic appreciation and aesthetic judgments: The aesthetic episode–Developments and challenges in empirical aesthetics. Br. J. Psychol. 2014, 105, 443–464. [Google Scholar] [CrossRef]
  32. Rebay, H. The beauty of non-objectivity. In Modern Art and Modernism: A Critical Anthology; Frascina, F., Harrison, C., Eds.; Routledge: Now York, NY, USA, 2018; p. 23. [Google Scholar]
  33. Dutton, D. The Art Instinct: Beauty, Pleasure, & Human Evolution; Oxford University Press: New York, NY, USA, 2009; pp. 96–97. [Google Scholar]
  34. Leder, H.; Bär, S.; Topolinski, S. Covert painting simulations influence aesthetic appreciation of artworks. Psychol. Sci. 2012, 23, 1479–1481. [Google Scholar] [CrossRef] [PubMed]
  35. McGuinness, D.E. Painting: Materials, Techniques, Styles, and Practice; Rosen Education Service: New York, NY, USA, 2016; pp. 1–17. [Google Scholar]
  36. Taylor, J.; Witt, J.; Grimaldi, P. Uncovering the connection between artist and audience: Viewing painted brushstrokes evokes corresponding action representations in the observer. J. Cogn. 2012, 125, 26–36. [Google Scholar] [CrossRef] [PubMed]
  37. Beatty, E.L.; Ball, L.J. Investigating exceptional poets to inform an understanding of the relationship between poetry and design. In Proceedings of the 2nd Conference on Creativity and Innovation in Design (ICCI 2014), Kuala Lumpur, Malaysia, 20–22 October 2014; pp. 157–165. [Google Scholar]
  38. Shannon, C.E. A mathematical theory of communication. Bell Syst. Tech. J. 1948, 27, 379–423. [Google Scholar] [CrossRef] [Green Version]
  39. Lin, R.; Qian, F.; Wu, J.; Fang, W.-T.; Jin, Y. A pilot study of communication matrix for evaluating artworks. In Proceedings of the International Conference on Cross-Cultural Design, Vancouver, BC, Canada, 9–14 July 2017; pp. 356–368. [Google Scholar]
  40. Shusterman, R. The end of aesthetic experience. J. Aesthet. Art Crit. 2010, 55, 29–41. [Google Scholar] [CrossRef]
  41. Bergeron, V.; Lopes, D.M. Aesthetic theory and aesthetic science. In Aesthetic Science: Connecting Minds, Brains, and Experience; Shimamura, A.P., Palmer, S.E., Eds.; Oxford University Press: New York, NY, USA, 2012; pp. 63–79. [Google Scholar]
  42. Mazzone, M.; Elgammal, A. Art, creativity, and the potential of artificial intelligence. Arts 2019, 8, 26. [Google Scholar] [CrossRef] [Green Version]
  43. Gao, Y.-J.; Chen, L.-Y.; Lee, S.; Lin, R.; Jin, Y. A study of communication in turning “poetry” into “painting”. In Proceedings of the International Conference on Cross-Cultural Design, Vancouver, BC, Canada, 9–14 July 2017; pp. 37–48. [Google Scholar]
  44. Gao, Y.; Wu, J.; Lee, S.; Lin, R. Communication between Artist and Audience: A Case Study of Creation Journey. In Proceedings of the International Conference on Human-Computer Interaction, Orlando, FL, USA, 26–31 July 2019; pp. 33–44. [Google Scholar]
Figure 1. The procedures for this study.
Figure 1. The procedures for this study.
Applsci 11 03290 g001
Figure 2. The example results of Gatys et al. [6], AdaIN [7], WCT [8], MST [9], and SEMST [10]. The most suitable method in each school was selected by experts and marked by a blue line.
Figure 2. The example results of Gatys et al. [6], AdaIN [7], WCT [8], MST [9], and SEMST [10]. The most suitable method in each school was selected by experts and marked by a blue line.
Applsci 11 03290 g002
Figure 3. Communication matrix for evaluating artistic style transfer.
Figure 3. Communication matrix for evaluating artistic style transfer.
Applsci 11 03290 g003
Figure 4. Participants’ most favorite paintings in each group.
Figure 4. Participants’ most favorite paintings in each group.
Applsci 11 03290 g004
Figure 5. Participants’ least favorite paintings in each group.
Figure 5. Participants’ least favorite paintings in each group.
Applsci 11 03290 g005
Table 1. The GitHub links of the five methods.
Table 1. The GitHub links of the five methods.
MethodsGitHub Links
Gatys et al. [6]https://github.com/ProGamerGov/neural-style-pt (accessed on 1 June 2020)
AdaIN [7]https://github.com/naoto0804/pytorch-AdaIN (accessed on 1 June 2020)
WCT [8]https://github.com/sunshineatnoon/PytorchWCT (accessed on 1 June 2020)
MST [9]https://github.com/irasin/Pytorch_MST (accessed on 1 June 2020)
AEMST [10]https://github.com/irasin/Structure-emphasized-Multimodal-Style-Transfer (accessed on 10 July 2020)
Table 2. Thirty-seven pairs of stimulus paintings.
Table 2. Thirty-seven pairs of stimulus paintings.
FauvismColorStrokeTexture
Technical level Applsci 11 03290 i001
A1 Complementary
Applsci 11 03290 i002
A2 Wantonly stroke
Applsci 11 03290 i003
A3 Decorative curve
Semantic level Applsci 11 03290 i004
B1 Strong
Applsci 11 03290 i005
B2 Chaos & power
Applsci 11 03290 i006
B3 Regular progress
Effectiveness level Applsci 11 03290 i007
C1 Unrestrained
Applsci 11 03290 i008
C2 Rough & natural
Applsci 11 03290 i009
C3 Cute Pattern
ExpressionismColorStrokeTexture
Technical level Applsci 11 03290 i010
A1 Contrasting
Applsci 11 03290 i011
A2 Solid and dashed
Applsci 11 03290 i012
A3 Regular
Semantic level Applsci 11 03290 i013
B1 Mysterious
Applsci 11 03290 i014
B2 Frantic
Applsci 11 03290 i015
B3 Wantonly
Effectiveness level Applsci 11 03290 i016
C1 Pessimist
Applsci 11 03290 i017
C2 Heartfelt
Applsci 11 03290 i018
C3 Vivid-generated
CubismColorStrokeTexture
Technical level Applsci 11 03290 i019
A1 Plane division
Applsci 11 03290 i020
A2 Short & powerful
Applsci 11 03290 i021
A3 Geometric
Semantic level Applsci 11 03290 i022
B1 Block structure
Applsci 11 03290 i023
B2 Bright & fast
Applsci 11 03290 i024
B3 Fragment space
Effectiveness level Applsci 11 03290 i025
C1 Space misplace
Applsci 11 03290 i026
C2 Manifest personality
Applsci 11 03290 i027
C3 Abstract beauty
RenaissanceColorStrokeTexture
Technical level Applsci 11 03290 i028
A1 Gradient
Applsci 11 03290 i029
A2 Delicate depiction
Applsci 11 03290 i030
A3 Stereoscopic
Semantic level Applsci 11 03290 i031
B1 Obvious levels
Applsci 11 03290 i032
B2 Realistic
Applsci 11 03290 i033
B3 Progressively
Effectiveness level Applsci 11 03290 i034
C1 Lively
Applsci 11 03290 i035
C2 Implicit
Applsci 11 03290 i036
C3 Natural
Reference sample Applsci 11 03290 i037
Mona Lisa
Table 3. Results of descriptive statistics and ANOVA analysis of four groups.
Table 3. Results of descriptive statistics and ANOVA analysis of four groups.
FauvismF1F2F3F4F5F6F7F8F9Reference SampleSignificance
Color64% c75% a82% a70% b83% a72% b72% b77% a78% a51% d***
Stroke53% b66% a76% a69% a74% a61% b74% a74% a75% a40% c***
Texture53% b61% b70% a70% a73% a62% b73% a74% a78% a38% c***
Overall54% c67% b76% a72% a79% a61% b73% a75% a77% a39% d***
ExpressionismE1E2E3E4E5E6E7E8E9Reference SampleSignificance
Color74% b59% c59% c84% a76% b86% a80% a74% b71% b59% c***
Stroke71% b54% c60% c84% a72% b84% a80% a73% b71% b50% d***
Texture70% b52% c57% c83% a73% b83% a79% a74% b69% b43% d***
Overall72% b52% c58% c84% a73% b84% a76% a74% b69% b46% d***
CubismC1C2C3C4C5C6C7C8C9Reference SampleSignificance
Color76% a67% b69% b75% a81% a81% a83% a66% b75% a54% c***
Stroke63% b64% b64% b71% a75% a73% a78% a64% b68% a47% c***
Texture60% b62% b62% b69% a74% a73% a77% a64% b67% a46% c***
Overall65% b63% b65% b72% a77% a76% a75% a64% b69% a47% c***
RenaissanceR1R2R3R4R5R6R7R8R9Reference SampleSignificance
Color80% a72% a76% a61% b68% b57% c64% b47% c47% c54% c***
Stroke70% a67% a70% a50% b60% a47% b60% a42% b42% b47% b***
Texture67% a64% a67% a48% b59% a47% b58% a41% b41% b45% b***
Overall71% a66% a71% a50% b61% a48% b59% a42% b41% b46% b***
*** p < 0.001; a, b, c and d are Duncan ex-post-test grouping results; the attribute with the highest score is marked in red.
Table 4. Result of average score and standard deviation for cognitive attributes.
Table 4. Result of average score and standard deviation for cognitive attributes.
FauvismP01P02P03P04P05P06P07P08P09P10
A1. Complementary3.583.844.033.513.973.483.903.774.192.10
B1. Strong3.553.743.973.613.943.263.743.944.292.10
C1. Unrestrained3.713.653.453.454.003.133.873.653.582.16
A2. Wantonly stroke3.713.453.423.583.453.133.943.363.422.32
B2. Chaos & power3.453.653.263.453.742.813.713.553.522.13
C2. Rough & natural3.553.583.523.653.813.033.453.293.422.26
A3. Decorative curve3.033.553.773.003.423.133.873.714.002.07
B3. Regular progress2.873.163.683.193.583.163.293.653.452.10
C3. Cute Pattern2.943.393.423.323.523.133.483.393.712.00
Average Score3.383.563.613.423.713.143.693.593.732.14
ExpressionismP01P02P03P04P05P06P07P08P09P10
A1. Contrasting3.472.72 3.003.69 3.413.813.283.693.092.63
B1. Mysterious3.19 3.313.313.033.193.19 3.193.723.00 3.41
C1. Pessimist2.81 3.25 2.72 2.78 2.75 2.472.883.092.842.72
A2. Solid and dashed2.382.722.50 2.942.562.842.812.56 2.532.31
B2. Frantic2.692.88 2.91 3.222.973.50 3.00 3.592.942.63
C2. Heartfelt2.692.59 2.72 3.223.093.223.223.342.412.16
A3. Rich in regular2.66 3.063.19 3.563.343.593.63 3.502.972.23
B3. Wantonly2.91 3.34 3.03 3.473.25 3.69 3.56 3.752.84 2.56
C3. Vivid-generated2.942.972.75 3.663.25 3.72 3.343.19 2.562.50
Average Score3.472.72 3.003.69 3.413.813.283.69 3.092.63
CubismP01P02P03P04P05P06P07P08P09P10
A1. Plane division2.633.003.003.163.283.283.883.133.002.00
B1. Block structure3.00 3.132.943.193.163.343.943.162.751.88
C1. Space misplace2.812.972.723.033.343.163.252.842.882.28
A2. Short & powerful3.193.252.843.133.593.593.593.283.062.06
B2. Bright & fast2.84 3.413.133.313.663.443.563.062.942.13
C2. Manifest personality2.88 3.383.063.593.813.783.813.193.092.25
A3. Geometric2.813.032.693.133.092.973.843.192.692.00
B3. Fragment space3.032.842.972.943.193.163.843.412.912.22
C3. Abstract beauty3.16 3.133.163.443.383.133.202.942.942.63
Average Score2.63 3.003.003.163.283.283.883.133.002.00
RenaissanceP01P02P03P04P05P06P07P08P09P10
A1. Gradient3.553.193.252.473.002.473.192.222.382.50
B1. Obvious levels3.063.253.752.473.092.442.782.162.222.38
C1. Lively3.093.223.692.192.972.252.722.062.002.28
A2. Delicate depiction2.972.883.311.942.691.972.382.031.912.09
B2. Realistic2.633.003.191.842.471.752.341.881.782.16
C2. Implicit3.092.883.002.502.722.312.752.192.092.28
A3. Stereoscopic2.662.723.282.032.722.092.281.911.841.94
B3. Progressively3.062.783.222.312.882.132.662.001.842.22
C3. Natural 2.972.723.162.062.471.782.441.941.752.28
Average Score3.553.193.252.473.002.473.192.222.382.50
The background color is used to emphasize the score of samples in their corresponding attributes; the attribute with the highest score is marked in red; the blue dotted line is used to mark the actual highest score that should obtain.
Table 5. Result of Pearson Correlation Analysis for the style attributes.
Table 5. Result of Pearson Correlation Analysis for the style attributes.
FauvismColorStrokeTextureOverallExpressionismColorStrokeTextureOverall
Color1 Color1
Stroke0.773 ***1 Stroke0.815 ***1
Texture0.695 ***0.894 ***1 Texture0.789 ***0.928 ***1
Overall0.759 ***0.909 ***0.902 ***1Overall0.821 ***0.876 ***0.899 ***1
CubismColorStrokeTextureOverallRenaissanceColorStrokeTextureOverall
Color1 Color1
Stroke0.793 ***1 Stroke0.846 ***1
Texture0.784 ***0.955 ***1 Texture0.822 ***0.969 ***1
Overall0.797 ***0.936 ***0.946 ***1Overall0.870 ***0.971 ***0.975 ***1
*** p < 0.001.
Table 6. The proportion of each painting being selected as a favorite.
Table 6. The proportion of each painting being selected as a favorite.
Fauvism
Order1234444888
No.P9P5P7P2P3P4P8P1P6P10
n (%)9
(29.0)
8
(25.8)
6
(19.4)
2
(6.5)
2
(6.5)
2
(6.5)
2
(6.5)
------
Expressionism
Order1234555888
No.P8P6P7P3P1P2P5P4P9P10
n (%)16
(50.0)
8
(25.0)
3
(9.4)
2
(6.3)
1
(3.1)
1
(3.1)
--------
Cubism
Order1224467799
No.P5P3P6P7P8P4P2P9P1P10
n (%)6
(19.4)
5
(16.1)
5
(16.1)
4
(12.9)
4
(12.9)
3
(9.7)
2
(6.5)
2
(6.5)
----
Renaissance
Order1234466666
No.P3P1P2P5P7P4P6P8P9P10
n (%)17
(54.8)
6
(19.4)
6
(19.4)
1
(3.2)
1
(3.2)
----------
Table 7. Proportion of each painting being selected as least favorite.
Table 7. Proportion of each painting being selected as least favorite.
Fauvism
Order1234466666
No.P10P4P5P3P6P1P2P7P8P9
n (%)24
(77.4)
3
(9.7)
2
(6.5)
1
(3.2)
1
(3.2)
----------
Expressionism
Order1234447777
No.P10P4P5P2P7P8P1P3P6P9
n (%)20
(64.5)
6
(19.4)
2
(6.5)
1
(3.2)
1
(3.2)
1
(3.2)
--------
Cubism
Order1234466666
No.P10P5P6P7P9P1P2P3P4P8
n (%)22
(71.0)
5
(16.1)
2
(6.5)
1
(3.2)
1
(3.2)
----------
Renaissance
Order1234466888
No.P9P8P6P2P3P5P10P1P4P7
n (%)11
(35.5)
10
(32.3)
4
(12.9)
2
(6.5)
2
(6.5)
1
(3.2)
1
(3.2)
------
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Lyu, Y.; Lin, C.-L.; Lin, P.-H.; Lin, R. The Cognition of Audience to Artistic Style Transfer. Appl. Sci. 2021, 11, 3290. https://0-doi-org.brum.beds.ac.uk/10.3390/app11073290

AMA Style

Lyu Y, Lin C-L, Lin P-H, Lin R. The Cognition of Audience to Artistic Style Transfer. Applied Sciences. 2021; 11(7):3290. https://0-doi-org.brum.beds.ac.uk/10.3390/app11073290

Chicago/Turabian Style

Lyu, Yanru, Chih-Long Lin, Po-Hsien Lin, and Rungtai Lin. 2021. "The Cognition of Audience to Artistic Style Transfer" Applied Sciences 11, no. 7: 3290. https://0-doi-org.brum.beds.ac.uk/10.3390/app11073290

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop