An Embodied Approach to Designing Meaningful Experiences with Ambient Media

Abstract

With the emerging trend in Human-Computer Interaction (HCI) shifting focus from usability to facilitating meaningful experiences, the notion of embodied cognition provides designers and researchers with valuable insight into how the body–mind interplay can influence meaning-making during embodied experiences. This paper presents an approach to designing embodied interactions with ambient media. Building on theories of embodied cognition and cognitive semantics, we developed our approach by conducting a series of studies, including an interpretive case analysis, empirical research into audience experience and design ideations, as well as designerly reflections on design. Our findings showed that an embodied approach is applicable for designing meaningful interactions, by coupling bodily engagement with metaphorical meanings. Design implications and future work are also presented.

1. Introduction

The vision of interaction design has been extended from digital artifacts to everyday “lively” environments, where interactive technologies are embedded to facilitate situated, contextual, meaningful interactions. While embodied cognition suggests that people’s cognitive processes are tightly related to the interaction between their body and the physical environment [1,2], growing research has centered on the role of bodily sensory-motor mechanisms in understanding languages [3], user experiences with digital media [4,5,6,7] and products [8], and movement-based interactions [9]. These studies substantiate and extend theories of embodied cognition and cognitive semantics. Much work has concentrated on the influence of sensory perceptions on emotional and cognitive experiences [6,10,11,12]. However, few studies have investigated how bodily interactions can facilitate meaningful experiences with ambient media, and how embodied theories can inform design. The main research question is thus: How can we develop an approach to supporting ambient media design regarding embodiment? Accordingly, the main objective of this paper is to explore an approach to informing the design development of ambient media works to facilitate embodied experiences. We will firstly summarize relevant theories and concepts on embodied interaction and ambient media, with the aim of establishing a theoretical foundation for our empirical explorations. Then, we will examine previous case analyses and empirical studies, to gain an interpretive and empirical understanding of audience experience. We will take the findings from the case and empirical studies and use these as guidance to explore a range of design experiments on ambient media works, following a research-through-design paradigm. Finally, we will discuss and reflect on the research process to develop an embodied approach to designing meaningful experiences with ambient media, and related design principles will also be formulated.

2. Theoretical Foundations

The exploration of design approach was positioned in a context of embodiment and experience with ambient media. Insights from embodied cognition and cognitive semantics helped construct a theoretical framework for this study.

2.1. Embodied Cognition and Embodied Interaction

Embodied cognition arose in the late 20th century with the critique that classical cognitive science typically centered on internal processes, regardless of the body’s role. On the contrary, embodied cognition believes that cognitive processes are largely shaped by the body’s states, and the mind is grounded in the body and bodily interactions with the world [2,13,14]. The philosophical root of embodied cognition dates back to a phenomenological tradition. Merleau-Ponty’s seminal work [15] put the body center stage, with sensory-motor activity as the vantage point. His term “intentional arc” denoted the inseparable relationship between the perception and action that underpins conscious thinking. Following Merleau-Ponty’s thoughts, Varela et al. [14] argued that cognition is shaped by sensory-motor interactions between the human body and the environment. Dreyfus [13] proposed three levels of embodiment: the body’s innate capacities (e.g., grip), acquired bodily skills through interacting with the world (e.g., walking on ground), and body actions related to the cultural environment (e.g., mailing a letter). Repeated practice enables the human body to “absorb” bodily skills in their interaction with environments, and these skills lead humans to act intuitively in certain tasks, without much cognitive effort. Wilson [2] pointed out six characteristics of embodied cognition, which claimed that cognition is formed through real-time interaction with the physical environment. To sum up, the mind is mutually intertwined with bodily actions.

Building on phenomenology, embodied interaction emphasizes that meaningful interactions between humans and machines are tightly coupled with both physical habits and social practices [16]. This coupling is actually in line with Dreyfus’s three ways of understanding embodiment: the body’s innate features, bodily skills, and cultural habits [13]. They determine how people perceive and act in different environments, and how people make sense of the world. Since then, growing research [5,9,17,18,19] has drawn attention to the bodily, social, and tangible nature of interactions with systems, as well as meaning-making and sharing through direct situated engagement. These studies, with various perspectives on embodiment, extend Dourish’s thought by focusing on the bodily experience in interaction design. For example, Svanæs [19] proposed two concepts related to the “lived body”: embodied perception and kinaesthetic creativity. This refers to how the body’s sensory perception is augmented by interactive technology and design related to the kinaesthetic senses of the body’s position and movement. Building on a theory from cognitive linguistics, Bakker et al. [17] investigated embodied interaction with tangible musical artifacts. Loke and Robertson [9] proposed the methodology “Moving and Making Strange” for the design and evaluation of movement-based interactions, believing that body movements play a central role in real-time cognition. Hornecker et al. [20] identified three distinctions in understanding the research on embodied interaction: subjective perspective vs. objective perspective, body vs. context, and practice vs. cognitive structure. This serves as clear guidance in developing embodied approaches. These embodiment-based studies indicate that embodied thinking provides an insightful lens in HCI research, especially for studies related to bodily movement and tangible interactions.

2.2. Embodiment in Cognitive Linguistics and Design Research

Embodiment in cognitive science has strongly influenced the emergence of cognitive linguistics, a modern field of language studies which believes that language reflects the way how people think [3,21,22,23,24]. The root of embodied cognition is central to research in cognitive linguistics, as it underlies the fundamental relationship between language, experience, and mental life [25]. Lakoff [26] summarized a number of research results regarding embodied cognition from various perspectives (e.g., experimental psychology, neuroscience), providing substantial evidence for the development of “a neural theory of thought and language (NTTL)”. These results also showed that abstract thinking is inherently embodied, and meaningful thought is grounded in the relationship between neural circuits and the human body. He also argued that conceptual systems are mainly embodied through two essential representations: embodied schemas (embodied cognitive primitives) and embodied metaphors. These have been applied to research in user experience with interactive systems [4,27,28] and products [8]. Johnson [21] argued that languages are rooted in bodily experience and structured by a range of “image schemas”. These schemas are the underlying cognitive patterns structuring human everyday understandings and meaning expressions. The term “embodied schema” emphasizes the embodied aspect of image schemas. “Embodied metaphors” are concrete extensions and applications of embodied schemas, especially referring to those conceptual metaphors clearly grounded in sensory-motor experiences. For example, the expression “HAPPY IS UP” is based on humans’ physical experience, as people feel that the body posture rises up when having a positive emotional state. When people talk about abstract concepts, they usually use expressions linked to bodily interactions with physical objects or environments to represent abstract concepts. In the phrase “grasp the meaning”, the meaning is like a physical object that can be held by hands. Through examining cases and empirical research on interactive experience, we propose that coupling affordances of a physical space and embodied schemas offers possibilities for designing embodied interaction with ambient media [29].

Myachykov et al. [30] proposed a framework called TEST theory to understand embodied knowledge. TEST includes three components: tropic (physical environment), embodied (bodily innate structure), and situated (cultural habits) characteristics. These can be used to eliminate the confusion between terms such as “situated”, “grounded”, and “embodied”. This framework partly echoes the three levels of embodiment described by Dreyfus (1996). Shapiro [31] critically reviewed the ideas of many major scholars (e.g., George Lakoff, Andy Clark) surrounding embodied cognition, aiming to clarify some of the diverse themes in this field and the relationship between embodied cognition and standard cognitive science. He believes that embodied cognition lacks well “defined theoretical commitments” and “uniform methodological practices”. A “Conceptualization Hypothesis” (p. 3) was used to denote the conceptual system that people use to make sense of the world, which is determined and constrained by the body’s properties. This is in line with the claim that “the body is our general medium for having a world” [15].

2.3. Experience with Ambient Media

In the Oxford dictionary, “ambient” means “relating to the immediate surroundings of something” [32]. In the authors’ prior studies, ambient media has been defined as a novel media form that integrates digital technologies into existing objects and environments in a public space for engaging people in embodied interaction. Three main properties have been identified; the physical environment in a public space (spatiality) can be transformed into an interactive media form that engages people in embodied interaction (engagement) and elicits unexpected responses (unexpectedness) [29,33]. Research into HCI regards ambient media as a combination of physical space and digital technology, which offers people with possibilities for natural interaction [34,35]. The research area most relevant to ambient media relates to “responsive environments”. Myron Krueger [36] initially explored interactive art in spaces by introducing the concept of a responsive environment that “perceives human behavior and responds with intelligent auditory and visual feedback”. Spatiality is the common property of the two concepts, as ambient media and responsive environments both emphasize public spaces as the basis for interaction that is augmented by digital technologies. However, ambient media focuses more on how people can be engaged in embodied interactions, by employing the existing affordances of the environment, while responsive environments pay more attention to artistic expressions and aesthetic experiences [37], and they may not be integrated into, but placed in, a public space. Furthermore, the spatial features of ambient media are closely related to the concept of affordances. The term “affordances” refers to the relation between the physical appearance of an environment or object and the potential actions of the organisms that are supported. For example, a horizontal surface (e.g., a rock, stone, or a stump) with appropriate height affords the possibility of sitting, although it is not a chair [1]. To some extent, affordances share similar ideas with embodiment in regard to the complementary relationship between the environment and people. As Heft [38] claimed, “Both Merleau-Ponty and Gibson emphasize that perceiving simultaneously entails an awareness of both the environment and the body”. Don Norman [39] introduced the concept of affordances to product and interface design by advocating that a product should provide perceivable clues to users for operating without confusion or difficulties. He also emphasized the role of “perceived affordances” in interaction design, as the feature of a product or interface should visibly indicate an actual operation. The perceivability of affordances is a key consideration in interaction design [40].

With the trend of designing for quality experience, considerable studies [6,41,42] have highlighted embodied, emotional, and meaningful properties in people’s experience with products and interactive systems. In general, experience is a relatively broad term, and there are various understandings in many research fields, such as user experience [41], brand experience [43], and audience experience [5]. However, most definitions of experience involve three basic psychological dimensions: perceptual, emotional, and cognitive responses. We generally conceptualize audience experience as sensory, emotional, kinesthetic, and cognitive responses that arise from interactions between the audience and ambient media [29] More specifically, this paper is concerned with the embodied experience with ambient media, focusing on the relationship between bodily interactions (bodily actions) and meaning-making (cognition).

In addition, regarding the term for the people experiencing ambient media, we use “participating audience” to refer to the persons who actively participate in interaction with ambient media. “Audience” has been widely used in studies on interaction in public spaces (e.g., [44,45]). Given that people have different roles (e.g., passerby, bystander, participant, and actor) in the interaction process, Wouters et al. [45] drew on “audience” to stand for those various roles. This interpretation is applicable to ambient media, and “audience experience” used in this article refers to the “participating audience’s experience”.

The theoretical framework is demonstrated as a diagram (Figure 1), which comprises three aspects. First, unexpectedness, engagement and spatiality can be regarded as the design strategies that open up new possibilities toward design creativity in ambient media design. Next, embodiment has been extensively discussed in the previous section, and as an approach and a strategy it provides phenomenological insights for ambient media design. Finally, facilitating audience experience is the major goal of ambient media design, and it needs empirical study to evaluate.

3. Methodology

In order to understand audience experience and explore an embodied approach to designing meaningful experience, this research followed a qualitative methodology, combining interpretive analysis and research through design. From a phenomenological perspective, understanding audience experience requires describing and interpreting meaning from subjective experiences, and a phenomenological approach that puts participants’ perceptions, feelings, and experiences center-stage in the qualitative research [46]. Interpretive analysis aims to understand phenomena and gather open-ended information through directly investigating participants’ experiences [47]. On the other hand, developing a design approach embodies the characteristics of exploratory research, such as Research through Design. Interaction design researchers contribute to new knowledge by engaging in iterative design development (e.g., prototyping and testing), which blurs the role between researcher and designer [48,49,50]. Research through Design is also a process of describing, interpreting, and reflecting on developing artifacts, and the reflections of researchers (designers) themselves can be a form of design evaluation [51]. This research was comprised of three progressive studies: a case study on ambient media works, an empirical study on audience experience, and a study on designing ambient media. Interpretive analysis was used in understanding experience with ambient media works, while thematic analysis was conducted for analyzing verbal reports in the empirical study. Interpretive and thematic analysis is inherently grounded in hermeneutic phenomenology and is a highly effective approach in eliciting personal experiences from a participant’s perspective [52].

The three studies constituted a tight and progressive research structure, as each study developed new models or instruments that aided the analysis and design of the next study, by following both deductive and inductive processes. Before reporting on the study on design ideation, we briefly review previous case analyses and empirical research.

3.1. Case Analyses

Based on the theoretical framework, we formulated an analytical model to analyze ten ambient media works [29]. The findings suggested that bodily engagement plays a central role in audience experience with ambient media. Interacting with ambient media is an active process, in which the audience may perceive affordances from an ambient medium that “invite” the audience to perform certain bodily actions. The interaction could directly generate embodied metaphorical expressions that reflect the structures of an audience’s past bodily experiences, and results in meaningful interaction and an engaging audience experience. For instance, with a Piano Staircase, passersby may notice the stairs designed as piano keys (affordance), and then they may step up or down (bodily actions), which triggers various musical sounds. The audience may try to get various musical pitches by stepping up and down (Scale schema), which maps the conceptual structure (musical pitches) on to another structure (heights of stairs). The interpretive analysis of cases provided a preliminary understanding of embodied experience with ambient media, informing further empirical study.

3.2. Empirical Study on the Piano Staircase

Previous empirical research [33,53] investigated embodied experience with ambient media through conducting experiments on a prototype. The main questions were: What major factors of audience experience can be identified? What mechanism can reflect the relationship between bodily states and cognitive experience? We firstly built an experience prototype piano staircase [54] as a test bed (Figure 2). Then, experimental tests were conducted to collect data about participants’ behaviors and verbal reports, which were analyzed qualitatively through thematic analysis. The findings included a coding scheme reflecting key factors of embodied experience, an engagement model, and embodied meaning-making model. This study provided empirical evidence that bodily action is not only an input behavior for triggering interaction, but also a major source of embodied conceptual mapping and facilitating meaningful experience.

Based on these findings, we developed a unified embodied engagement model (Figure 3), which integrates engagement processes (temporal dimension) with affordance-based experience. The horizontal dimension represents a temporal interaction process, comprising initial encounter, adaption, anticipation, exploration, and evaluation, in which audience experience evolves through the whole process. The vertical dimension refers to four types of responses (sensory, physical, cognitive, and emotional), which are based on four types of affordances [55]. The capability of audience engagement depends on the extent to which the work provides affordances to drive embodied interactions. Each type of affordance has various roles and intensity levels in different interaction phases. Three levels of intensity (high, medium, and low) stand for the major or minor effect of an affordance on experience. This model provides useful guidance for characterizing and designing experiences with ambient media.

4. Designing Meaningful Experience with Ambient Media

Based on the findings from past interpretive analyses and empirical research, we aimed to investigate how these insights can inform the design of ambient media. Following research regarding design [56], this study further explored the embodied approach by developing original designs that facilitate an engaging and meaningful experience. Through ideating, sketching, prototyping, experimenting, and analyzing, this study was able to evaluate, validate, and reframe previous models, and propose a feasible embodied design approach to designing ambient media. The researchers, as reflective practitioners, engaged in design activities to experience and understand the process of designing.

4.1. Pilot Study

We carried out a pilot design workshop to test whether the insights from previous studies could help designers to develop ambient media works towards meaningful experience. Four students (Lee Hoi Lam, Lam Hiu Tung, Chung King Tin, and Chau King Wing) from the interactive media design program participated in the workshop. The first author initially introduced the basic concepts of ambient media and embodied interaction, and emphasized the role of bodily engagement in creating meaningful experience. After introducing relevant concepts, some design strategies derived from the model (Figure 3) were provided to the students:

(1)

Incorporating the installation into an everyday public environment.

(2)

Eliciting unexpected experience by designing input–output relations.

(3)

Coupling the audience’s bodily actions with sensory feedbacks.

(4)

Creating embodied metaphorical mappings and considering the congruency between spatial structures (embodied schemas) and everyday bodily actions.

Once the participants had chosen their specific topics, they were advised to develop design ideas by considering these four guidelines. In this pilot study, some concepts (e.g., embodied schema) were difficult for students to understand, and general guidance was provided to see how students would employ the strategies to develop their designs.

The topic was about reducing paper use. Inspired by the design strategy of the embodied metaphorical mappings between bodily actions, they developed the design idea (Figure 4) that the bodily action of tearing a towel paper could metaphorically express destroying forest. The design prototype was proposed to be set up in the washing area of a public toilet. The interaction between the prototype and audience was expected to give rise to two metaphorical mappings: (1) When the audience teared (Splitting schema) a paper tower from the dispenser the video on the mirror (screen) displayed a tree being cut down by a saw and a forest disappearing; (2) Reminded by the video, the audience shook their hands in front of the washbasin to remove water, while a video showed trees growing (Scale schema), which metaphorically expresses that behavioral changes can contribute to sustainable communities.

The pilot design workshop indicated that the four strategies could guide the idea development of ambient media design, in regard to embodied meaning expression. Based on a certain embodied schematic structure, coupling bodily actions with metaphorical expression is a central starting point in designing embodied interaction with ambient media. Accordingly, a direct question is how embodied schemas can be linked to the various features of public spaces, in regard to people’s behaviors. Embodied schemas arise from long-term bodily experiences with the environmental properties that embody affordances for human acting [57]. After examining the previous studies and pilot study, we formulated a matrix to demonstrate the relationship between various spatial features in public spaces and schemas (

Table 1. Matrix of embodied schemas and spatial features.

Embodied Schemas and Spatial Features	Container	Path	Orientation (e.g., Up–Down)	Force (e.g., Blockage, Gravitation)
Staircase			•
Door	•
Corridor		•
Floor				•

), which indicates that the features of a public space and everyday objects provide corresponding action possibilities that are organized by a range of recurrent patterns. A black dot denotes a certain connection between a schema and a space, and that both dimensions can be extended. For example, a corridor may be related to a “Source-Path-Goal” (Path) schema. A staircase is about walking up and down, and forward and backward (orientation). The matrix provides a practical guide for designing ambient media, in regard to employing elements in public spaces as the interface for meaningful interactions.

4.2. Design Ideation

In order to further understand how to design meaningful interactions with ambient media, the first author conducted design ideation through the active participation of the researcher (also as a designer) in designing and inquiring. This was expected to open up new discussion on ambient media design and embodied design thinking. Before ideation, this study firstly prepared some topics as the thematic direction for development. The ideation started with the Global Goals for sustainable development proposed by the United Nations [58], which target enhancing life quality (e.g., good health and quality education) and reducing environmental pollution. Then, the researcher developed ten design ideas (Figure 5) by applying the embodied design approach, and the ideation process is described as follows.

(1) Basket Trash Bin

Throwing trash carelessly in public spaces is a common phenomenon in many developing countries. In order to encourage people to drop litter in trash bins rather than on the ground, the researcher tried to ideate with the schematic structure of trash bins and people’s everyday actions. A trash bin is a container that people constantly put something into or take something from. The bodily actions of dropping litter into a trash bin and throwing a basketball into a net have similar schematic patterns. They are both about exerting a force to put an object into a container. The difference is that playing basketball is more interesting than throwing trash. With regard to the same embodied schema, the researcher designed a basket trash bin by incorporating the backboard of a basketball game and a trash bin (Figure 5-1). When a passerby throws trash into the bin, the behavior is like throwing a basketball into a net, and the hidden speaker plays a cheering sound. The main design inspiration was bodily interaction with the schematic Container structure. This constructs the metaphorical mapping between the action of throwing a basketball and the action of dropping trash, which aims to motivate people to drop trash properly and accurately.

(2) Reading Door

This topic is about encouraging book reading through meaningful interaction, and it is related to the global goal of “Quality Education”. The design idea was inspired by the everyday activity of opening a book. There is a similar embodied schematic structure between opening a door and opening a book, since people constantly experience entering and opening bounded spaces (e.g., a room or box), which gives rise to an embodied Container schema. The design transforms the door of a building entrance into an interactive ambient medium (Figure 5-2). The surface of the door is decorated like a book cover. When a passerby pulls the handle, the separated layer is flipped over, but the door is not actually opened, and a quote appears on the white door: “A book is a gift you can open again and again—Garrison Keillor”, meanwhile, a sound like flipping book pages is played. The embodied metaphor is activated through the action of opening the “door” denoting a meaning of reading a book. This embodies the three attributes of ambient media.

(3) Social Bench

This design is concerned with a problem of social isolation caused by excessive use of smartphones or the fast pace of modern life. The ideation of this idea starts with the concept of “connecting people” in a public space, and how to facilitate people’s communication in a meaningful way. A bench in a public park was chosen as the medium for social interaction (Figure 5-3). When someone sits on the bench alone, a light effect will appear around the person, which may attract another passersby’s attention. Then, if another person sits on it, their positions will be connected by a glowing line, which may motivate them to communicate with each other. This schematic metaphor is triggered by the sensorimotor interaction and structured by the embodied Unity (Link) schema.

(4) Hug Pole

This design addresses the mental health issue that many people feel loneliness and coldness in winter (Figure 5-4). The idea was ideated from an opposite direction of coldness and aimed to afford people warmth and closeness of experience. Based on the Near–Far orientation schematic structure, the design aimed to provide a way in which people could engage in bodily interaction with sensory and physical warmth, giving rise to a psychological warmth and closeness. A heart-shaped cushion is installed on a street pole, which affords the possible actions of a hug or touch. If a passerby hugs the cushion tightly, he/she will gradually perceive warmth from the cushion that will simultaneously glow with red light. The embodied metaphorical mapping from physical warmth to psychological warmth and closeness is activated through bodily actions (i.e., approaching the pole and hugging the cushion), and the sensory feedbacks of physical warmth and light effects further amplify the embodied schematic metaphor.

(5) Book Staircase

This design’s topic is similar to “Reading Door”, as it also promotes book reading. The staircase of a public library is chosen as the medium for interaction. The form of the stairs resembles a book (Figure 5-5), and the arrangement is like a staircase composed of books. When people walk on a stair, it triggers a voice reading a famous quote from a book. Bodily engagement with the work is related to two metaphorical levels: the first is the visual metaphor resembling the shapes of two different objects (i.e., a stair and a book). The second is an embodied conceptual metaphor extended from the Orientation schema, which metaphorically means that a book is a “stair” that can bring people to a higher place.

(6) Lighting Corridor

This design idea is advocating a message about positive attitudes in university life. According to the conceptual metaphor “Life is a journey”, the temporal concept (life) can be understood in terms of a spatial concept (journey). In regard to the matrix of schemas and spatial features, the researcher drew upon a corridor as a medium to design embodied interaction, as a corridor potentially affords movement along a path. A corridor in the outdoor space of a university is chosen as the medium for interaction (Figure 5-6). When the passerby walks along the corridor, the two poles ahead of him light up, which metaphorically express that the future life (of students and staff) is lit up and people should look into the future with hope. The two schematic structures Path and Orientation are related to this embodied interaction.

(7) Water Body

Water Body is concerned with the UN Global Goal of “Good Health and Well-Being”. Many people forget to drink enough water in their daily lives. In order to raise awareness of keeping enough water in the body, the ideation started from the Container schematic structure, as the human body can be regarded as a container according to image schema theory. The design (Figure 5-7) projects a real-time body figure of the audience member on the wall, and at first the water level indicates that the body comprises about 70% water. With time passing by, the water level gradually declines, and in the meantime, a cup of water appears on the side. If the audience touches the cup by hand, the water level rises as water is being added to the body. The metaphorical meaning mainly pertains to the Container and Orientation (Up–Down) schemas.

(8) Smile Mirror

Smile Mirror pays attention to mental health problems (e.g., depression). Keeping smiling is often an important indicator of happiness and a way to maintain happiness. This design provides a medium by which people can experience their emotional changes. When a person goes to the public restroom (Figure 5-8), the mirror above the wash basin will reflect their face. If the facial expression is smile-less, a mask will hide the face. Conversely, when smiling, the mask disappears and the smiling face is shown. This facial interaction metaphorically expresses that people should change toward a positive mental state, which is structured by the Existence (Removal) schema.

(9) Forest Floor

This design is to address the environmental problem of forest degradation caused by human activities. As shown in Figure 5-9, the floor of an indoor public space is used as an interactive interface and ambient medium. A forest scene with a bird’s-eye view is projected onto the floor (what the forest looks like from sky). When the audience passes the forest, the area around them becomes grey and looks destroyed, but it recovers after they leave, which metaphorically expresses that human activities may lead to forest degradation. The image schema Removal, as a source domain in this embodied metaphor, is mapped onto the target domain “destroying forest”.

(10) Floating Belt

This design topic is concerned with increasingly severe marine pollution. In order to advocate awareness and action in protecting the marine environment, the ideation started from the Path schematic pattern, and the luggage belt in an airport is used as an ambient medium interface, as its motion is like the ocean flow. In addition, passengers in an airport have to experience a period of luggage waiting time. The design takes the waiting time as an opportunity to create embodied engagement with the ambient medium (Figure 5-10). Before luggage arrives, ocean waves are projected onto the belt containing floating garbage. Passengers can clean the garbage by waving their hands, and once the hand shadow hides garbage, it disappears. When their luggage arrives, the belt becomes a clean ocean with fish. The whole interaction is structured by the Path schema and Removal schema.

4.3. Ideation Analyses

These ten ideas were developed based on the embodied design approach. In the context of design ideation, these designs showed that embodied schematic thinking is workable, and the matrix of schemas and spatial features can provide a direct and fast guide to generate design ideas about interaction with ambient media. Although the current study did not build prototypes and conduct empirical experience tests, it is possible to analyze these designs in terms of the researchers’ self-evaluation, which will provide a reflective basis for the next phase of study.

4.3.1. Analysis Framework

Based on the theoretical framework established, the ten ideas were analyzed in terms of four criteria: metaphorical mapping, bodily engagement, unexpectedness, and public space.

Table 2. Analysis rubric.

Criteria	Rubric for Analysis	Level
Metaphorical mapping	Level 3: Embodied metaphor and visual metaphor are both constructed in the interaction process. Level 2: Embodied metaphor is constructed in the interaction process. Level 1: Visual metaphor is constructed in the interaction process.	1–3
Bodily engagement	Level 3: Participating in motor interaction through the whole body. Level 2: Participating in motor interaction through parts of the body (e.g., arms, hands). Level 1: Observing a situation by standing or sitting.
Unexpectedness	Level 3: All strategies are involved. Level 2: Two of the strategies are involved. Level 1: One of the strategies is involved. Three strategies for triggering unexpectedness: (1) Contrasting input/output (I/O) relations: When the audience performs an action, the output does not match the audience’s expectation (e.g., twisting a shoe results in the distortion of a building). (2) Contrasting physical and virtual elements: Digital media is embedded in the physical space, which leads to a mismatch between tangible reality and virtual reality (e.g., after a physical door is opened, a digital image emerges to show the scene from another city. (3) Visual elements mismatching the content or context (e.g., stairs resembling the shape of piano keys elicit a surprise.)
Public space	Level 3: The work is situated in an outdoor public space. Level 2: The work is situated in an indoor public space. Level 1: The work is situated in an indoor semi-public space.

, comprising four criteria (the first column), was formulated as a guideline for the qualitative analysis (the second column). As these design ideas targeted different topics (except Reading Door and Book Staircase), the analysis was not to evaluate which design was better than others for a certain design problem. Rather, it evaluated whether the embodied design approach could effectively guide the ideation process for ambient media design. The numbers one to three were used to represent the levels to which the design achieved the criteria (the third column).

We took Reading Door as an example for analysis, as shown in

Table 3. Analysis of Reading Door.

Criteria	Analysis	Level
Idea 2: Reading Door
Metaphorical mapping	The action of opening the door simultaneously triggers a sound like flipping book pages, and the sensory-motor interaction as a source domain is metaphorically mapped onto reading a book (target domain), which constructs a metaphor that opening a physical space is like entering a knowledge space. The door is decorated like a book cover, which is a visual metaphor.	3
Bodily engagement	Whole body is engaged in the interaction with the Reading Door.	3
Unexpectedness	Two strategies are used: The door resembles the visual features of a book cover and a hidden speaker plays a sound of opening book pages as feedback.	2
Public space	The entrance of the library is an outdoor public space.	3

. Admittedly, there are many ways to interact with a door in terms of the type of mechanism, as a door could be inward or outward for opening, automatic, or sliding. For example, Ju [59] investigated people’s perception and cognition about an automatic door’s movements, and the findings indicated that a door’s opening actions can influence people’s emotional appraisal, such as feeling welcome when the door is automatically opening. However, Reading Door focuses on the opening outward type. Although the bodily interactions involved with Reading Door may be related to other schemas (e.g., the Force schema in pulling the door and the Near-Far schema in approaching the door), we consider the metaphorical mapping between opening a door and reading a book is mainly structured by the Container schema, and the two domains are in line with the components of the Container schema: there is a boundary (door and book cover) that divides an interior space (room and knowledge) and an exterior space (outside of the door and book).

4.3.2. Overall Analysis Results

All above analyses were visualized as bar charts (Figure 6), indicating their level of variation according to four criteria (M stands for metaphorical mapping, B for bodily engagement, U for unexpectedness, and P for public space).

As these designs had different topics and scenarios, the comparison was not to judge which idea was best, but to discuss whether the embodied design approach is workable for generating design ideas. Idea 2 (Reading Door), Idea 4 (Hug Pole) and Idea 10 (Floating Belt) have a similar pattern, with relatively higher levels across the four aspects. They all engage whole-body action, while visual mapping and conceptual mapping both contribute to embodied meaning-making. Bodily movements, as a concrete embodiment of image schema, activate metaphorical mapping from source domain to target domain. In Reading Door, the action of opening the door, as the source domain, mapped onto the target domain of opening (reading) a book or entering a world of knowledge, which was structured by the Container schema. The action of opening the “book” was congruent with the feedback of seeing on the quote on the white board (page). Hug Pole had active bodily engagement by affording hugging actions, and the temperature feedback (warmth) aligns with the motor input and embodied meaning.

Idea 6 (Lighting Corridor) and Idea 9 (Forest Floor) have the same levels of the first three criteria. They only use conceptual metaphors, where spatial distance and forward lighting metaphorically express a sense of future hope, and walking on the floor (forest), as a source domain is mapped onto the target domain of “destroying forests”. They engage whole-body movements. One strategy for eliciting unexpectedness is used. The lights always appear about five meters ahead of the people who are walking (contrasting I/O relations). Disappearing and recovering of the forest may give rise to unexpectedness (contrasting physical and virtual elements). Idea 7 and Idea 8 have similar patterns. Metaphorical mappings in Water Body and Smile Mirror are embodied well and explicitly, but bodily actions are not actively engaged. They contrast I/O relations to elicit unexpectedness.

The above analysis indicates that the embodied design approach is applicable for generating design ideas for embodied experiences with ambient media. Although the approach has not yet been integrated as concrete design guidelines, four design strategies, and a matrix of schemas and spatial features have shown that they can effectively guide the ideation process.

5. Discussion

5.1. Understanding and Designing Embodied Experience

Referring to the main research question “How can we develop an approach to supporting ambient media design regarding embodiment?”, three progressive studies gave rise to two types of findings: models for understanding embodied experience and strategies for ideating the design of ambient media. These jointly constructed an embodied approach to designing ambient media. We synthesized a diagram to show the evolution of the approach (Figure 7).

Theoretical, interpretive, and empirical studies were conducted to gain a holistic and in-depth understanding of embodied experience with ambient media. First, the two models generated from the case studies could be used to analyze ambient media cases, while the embodied meaning-making model and the engagement model from the empirical study offered researchers guidance in understanding embodied experience (for both meaning-making and engagement levels) with interactive ambient media in physical spaces. Then, the two analytical models were further transformed into design strategies for guiding design ideation through a range of design experiments, where the researcher, as a reflective practitioner, engaged in design activities to experience and understand the process of designing. Finally, the process of designing was also a way to evaluate, validate, and reframe the previous models, and propose a feasible embodied design approach to designing ambient media.

5.2. Embodied Interaction with Ambient Media

This approach substantiated the theoretical presumptions about embodiment and the three properties of ambient media. The empirical findings showed that spatial features (spatiality) attract the attention of an audience, who may gain an unexpected experience (unexpectedness) while they are engaged in bodily interaction with the ambient media (engagement). Incorporating elements of physical public spaces with ambient media (see Section 3.1) exhibits the nature of embodied cognition: cognitive processing is situated in a real environment and ties to bodily engagement with the physical world (Wilson, 2002). Employing spatial features is in line with the theory of affordance (Gibson, 1979), and interacting with ambient media is an active process, in which audience perceive the affordances that “invite” them to perform certain bodily actions. Our findings extend studies on embodied metaphor theories, such as language comprehension (Gibbs and Wilson, 2002) and product expression (Van Rompay and Ludden, 2015). The embodied approach of this study considered the embodied metaphor as a design source rather than a mere expression of meaning, and the design strategy of coupling bodily actions with the potential schematic meaning of spatial features gave rise to endless possibilities in both design practice and research. On the other hand, we need to differentiate our embodied approach from metaphorical design (e.g., [60,61]). Compared to the metaphor concept in metaphorical design, embodied metaphors (see Section 2.2 and Section 3.2), as the key idea in the present study, have two characteristics. First, the metaphorical mapping of an embodied metaphor is mainly constructed by bodily interaction and the source domain is bodily action (e.g., opening the door), while the target domain is a conceptual meaning (entering a knowledge space). However, metaphorical design mainly regards metaphors as a design method, and designers create design forms by resembling everyday concrete things (e.g., the trash bin on a computer desktop denoting recycling deleted files) to increase the usability of a product or a digital interface, or thinking in terms of conceptual similarity (e.g., Bruce is a lion). Secondly, embodied metaphors are structured by embodied schemas that arise from long-term bodily experiences, while metaphorical design does not necessarily draw on the role of embodied schemas and bodily experiences in meaning-making.

This section focuses on discussing embodied interactions with ambient media, with regard to the relationship between meaning-making and the three properties of ambient media. Firstly, the argument that bodily engagement plays a central role in meaningful experience with ambient media resonates with previous studies on embodied interaction [5,9,16]. One important aspect of embodied interaction is coupling tangible interaction with people’s bodily skills, as “the intuition behind tangible computing is that, because we have highly developed skills for physical interaction with objects in the world…” [16]. The embodied approach to movement-based interaction [9] also recognizes that bodily engagement plays a central role in lived cognition and experience. The creative engagement model [5] depicts five cognitive engagement phases in a linear order to describe experiences with interactive art works in a museum. Compared with this model, our embodied engagement model (Figure 3) reflects the underlying relations between temporal engagement stages, experiential dimensions, as well as embodied metaphorical mapping, which construct a unified analytical tool for understanding embodied experience with ambient media.

Secondly, our findings regarding the relation between spatial features and embodied schemas extends our understanding of conceptual metaphors and image schemas in cognitive semantics and HCI. Related studies in HCI have applied theories of image schemas to investigate task-based interactions with digital interfaces [7] and tangible artifacts [4], which mainly focus on the correspondence between image schemas and metaphorical meanings, rather than potential schematic meanings generated from interactions with a certain physical space. In the ideation of ambient media design, the matrix of spatial features and embodied schemas, serving as a design instrument, provides direct guidance for ideating ideas.

In addition, eliciting unexpected responses also plays an important role in embodied interaction with ambient media. Based on our empirical research [33] and the analysis of design ideas (Section 4.3), unexpectedness is closely coupled with engagement with ambient media. When encountering an ambient medium, visual properties mismatching the physical space in function (e.g., the piano keyboard integrated with the staircase) can attract an audience’s attention and increase the perceivability of the affordance for engaging in interaction. The mismatch is embodied according to two aspects: location and arrangement. That is, when an object appears in an unexpected environment (e.g., a book cover appearing on a door), and the arrangement of the object is unconventional (e.g., a book cover enlarged to a door size or a piano keyboard arranged gradually in a vertical direction). Mismatching, which has been proposed as one of the key strategies for eliciting unexpectedness (Table 2), is basically a result of visual metaphors (see metaphorical mapping in Table 2 and Table 3) that refer to the resemblance in shape between two things and, thus, is also a means for meaning-making in metaphorical mapping. Unexpectedness is also elicited from bodily interactions, and the feedback triggered by an audience’s bodily action may be out of his or her expectation. For example, when someone sits on a bench and a light effect appears around them (see Section 4.2—Social Bench) or when opening a door triggers a sound like flipping book pages (see Section 4.2—Reading Door). Although embodied interaction is generally understood as intuitive in use [7] and based on familiarity with the real world [16], our emphasis on unexpectedness as a property of ambient media does not contradict the nature of embodied interaction, as ambient media design, to some extent, is not aimed to serve a practical function (e.g., a product), but rather, it is a way enriching people’s everyday experience. We argue that unexpectedness is not only one property of ambient media but also a factor for facilitating engagement and meaningful experience in interacting with ambient media.

5.3. Implications for Design

Based on the discussion, we summarize a set of design guidelines derived from the progressive studies described.

Guideline 1: Coupling the affordance of spatial features with embodied schemas

In the initial stage, one effective method is to ideate design ideas by coupling the physical affordance of a space with a corresponding embodied schema. Various spatial characteristics offer different action possibilities, which can be utilized as a design source for eliciting potential embodied metaphors, extended from a certain schema. For example, a trash bin, as a physical container, affords the action possibility for throwing objects, which may be mapped to another concept structured by the Container schema. As shown in Section 4, designers can connect the bodily actions solicited by physical spaces with various embodied schemas to design meaningful interactions.

Guideline 2: Creating unexpected responses to enhance engagement

As a main property of ambient media, eliciting surprising experiences has been considered as an experiential quality in designing meaningful interactions [62,63]. Three strategies for designing unexpectedness have been used in design ideation: contrasting input–output relations, contrasting physical and virtual elements, and mismatching visual elements. Referring to the case analyses and ideated ideas, these strategies can be used in different engagement phases. In an encounter, mismatching visual elements is a direct method for eliciting surprise and attracting an audience’s attention, while the other methods can help to engage people in exploring further possibilities.

Guideline 3: Aligning real-time feedback with embodied metaphors

Feedback is not only the effect of an audience’s behaviors, but also plays a key role in generating embodied metaphors, especially when body engagement is not very proactive (e.g., standing or sitting). Visual, auditory, tactile, and multimodal feedback can be used to jointly enhance embodied metaphorical mapping [4,5,40]. Facilitating meaningful interaction requires designers to maintain congruency between bodily actions (input) and feedback (output). In Social Bench, the seating areas are connected by a glowing line, which corresponds to the action of sitting down, and amplifies the metaphorical meaning of social connection. In Hug Pole, the physical warmth and red lighting as feedbacks are activated when a passerby hugs the heart-shaped cushion. The feeling of emotional warmth metaphorically results from the action of hugging, and the red lighting and heart-shape jointly enhance the embodied metaphor.

6. Conclusions

Building on a rigorous theoretical framework, this paper described a set of studies for understanding and designing meaningful experiences with ambient media. We propose a set of models and strategies that offer designers and researchers useful guidelines for analysis and design. These models exhibit the main factors of experience, meaning-making mechanisms, and engagement patterns. To explore the embodied approach, design strategies were applied in ideating a range of design ideas, and analyses on these ideas showed that the approach is promising and highly applicable. However, there were three main limitations to the present embodied design approach. First, despite the proposed matrix linking spatial features and embodied schemas (Table 1), this approach may not be easily applied when identifying appropriate embodied schemas in some public spaces (e.g., a square) or to express certain meanings (design topic). Second, these design ideas were generated from the authors’ perspective, and further study is needed to validate the approach in terms of accessibility to other designers. Additionally, there was a lack of empirical research evaluating the design ideas in terms of the design criteria (Table 2). In order to further evaluate our approach, we are currently working on an experimental study to prototype and test the design idea Reading Door, which is expected to substantiate the proposition. Future work will explore other modalities (e.g., tactile and kinesthetic senses) in addition to visual and auditory senses, and how these modalities influence meaning-making in interacting with Reading Door. For example, if the door is made of light and flexible materials, people can directly flip the door without holding the handle. This may be related to the Force schema and activate more metaphorical associations. Related design workshops and experiments will be conducted to apply and advance the embodied approach.