Next Article in Journal
Surface Modification of Dielectric Substrates by Broad Beams of High-Energy Atoms of Inert Gases
Previous Article in Journal
A Space-Time Correlation Model for MRC Receivers in Rayleigh Fading Channels
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Perceived Usefulness, Satisfaction, Ease of Use and Potential of a Virtual Companion to Support the Care Provision for Older Adults

1
Cáritas Diocesana de Coimbra, Rua Dom Francisco de Almeida 14, 3030-382 Coimbra, Portugal
2
Instituto Pedro Nunes, Rua Pedro Nunes, 3030-199 Coimbra, Portugal
3
IEETA-Institute of Electronics and Informatics Engineering of Aveiro, Department of Medical Sciences, University of Aveiro, Campo Universitário de Santiago, 3810-193 Aveiro, Portugal
*
Author to whom correspondence should be addressed.
Submission received: 24 June 2020 / Revised: 22 July 2020 / Accepted: 22 July 2020 / Published: 25 July 2020
(This article belongs to the Section Information and Communication Technologies)

Abstract

:
This article reports a study aiming to determine the perceptions of older adults needing formal care about the usefulness, satisfaction, and ease of use of CaMeLi, a virtual companion based on an embodied conversational agent, and the perceptions of formal caregivers about the potential of virtual companions to support care provision. An observational study involving older adults needing formal care was conducted to assess CaMeLi using a multi-method approach (i.e., an auto-reported questionnaire—the Usefulness, Satisfaction, and Ease of use questionnaire; a scale for the usability assessment based on the opinion of observers—the International Classification of Functioning Disability and Health-based Usability Scale; and critical incident registration). Moreover, a focus group was conducted to collect data regarding the perceived utility of virtual companions to support care provision. The observational study was conducted with 46 participants with an average age of 63.6 years, and the results were associated with a high level of usefulness, satisfaction, and ease of use of CaMeLi. Furthermore, the focus group composed of four care providers considered virtual companions a promising solution to support care provision and to prevent loneliness and social isolation. The results of both the observational study and the focus group revealed good perceptions regarding the role of virtual companions to support the care provision for older adults.

1. Introduction

The changes in population age pyramids that have occurred in recent decades, namely the increase of older age groups, impact provision of care for older adults. Solutions such as home care might contribute to addressing the challenges faced by health and social care systems concerning the current population evolution trend [1,2]. Most older adults desire to stay at home as long as possible, integrated in the community, if they can do it with independence and autonomy, rather than live in an institution [3,4]. However, the deterioration in mental and physical abilities resulting from the normal ageing process make this preference a challenge to be addressed. Therefore, integrated solutions based on information technologies are regarded as one of the possibilities to promote ageing in place, avoiding or delaying institutionalization, and preventing loneliness and social isolation [5].
Considering the current demographic challenges, new digital solutions are being developed [5]. Although these solutions cannot and should not replace the human role, they represent an additional support to human caregiving [6]. Examples of these digital solutions include virtual companions that provide practical tools to help older adults to maximize their independence and autonomy. Virtual companions might include embodied conversational agents [7] capable of receiving human voice communications from a user and selecting the appropriate responsive action to the query or instruction received [8].
In the scope of the CaMeLi European Project [9], funded by the Ambient Assisted Living Joint Program, a virtual companion, CaMeLi, was developed [10]. CaMeLi is grounded in an embodied conversational agent with a human-like figure with which older adults can speak and interact by asking it to perform actions such as reading the news [10].
The objectives of the study reported by this article were (i) to determine the perceptions of older adults needing formal care about the usefulness, satisfaction, and ease of use of CaMeLi, by conducting an observational study; and (ii) to analyze the potential of the embodied conversational agents for care provision, according to the perspective of formal caregivers, by conducting a focus group. The results of the study represent not only essential feedback for the technical improvement of CaMeLi, but also a contribution to the current knowledge of older adults’ experiences with virtual companions based on embodied conversational agents and how these digital solutions can be used to optimize care provision.
In the following sections, this article introduces (i) the state of the art of virtual companions based on embodied conversational agents, as well as the current trends of the assessment of usefulness, satisfaction, and ease of use; (ii) the materials and methods that were used; (iii) the results of the study; and (iv) discussion of the results and conclusion.

2. Related Work

2.1. Virtual Companions Based on Embodied Conversational Agents

Various researchers have developed embodied conversational agents to interact with users over multiple conversations, ranging from a handful of interactions to hundreds of interactions spanning long-term periods [11,12,13].
Embodied conversational agents are typically represented in the form of human or animal bodies that are specifically lifelike and believable in the way they behave. They simulate human-like properties in face-to-face conversation, including the ability to recognize and respond to verbal and non-verbal input, and generate verbal and non-verbal output (e.g., mouth, eye and head movements, hand gestures, facial expressions, or body posture), and can deal with conversational functions such as turn taking, feedback and repair mechanisms [7]. Due to these characteristics, embodied conversational agents provide familiar and non-threatening interfaces, especially useful for building systems that are easy to use and engaging, and can gain the trust of their users, even those with age-related or other cognitive impairments [14].
Studies suggest that virtual companions based on embodied conversational agents dealing with emotion and affection are particularly capable of capturing the users’ attention, engaging them in active tasks and entertaining them [15], leading to the development of affinity relationships with their human partners [16,17].
Different virtual companions based on embodied conversational agents have been developed aimed at a broad range of purposes, namely [18]: physical activity [11,12,19,20,21,22], medical treatment [20,23,24,25,26,27], mental health [28,29,30,31], nutrition [32,33,34,35], daily activities [14,36,37], or social support [17,38]. In some cases, virtual companions have been successfully developed to monitor, encourage, and assist older adults [11,14,17,20,22,31,36,38].
A variety of virtual companions has been developed aiming to provide social support to isolated older adults [17,38] and to address their daily needs for autonomous living [14]. Moreover, virtual companions have also been used as coaches and wellness counsellors in health behavior change interventions for older adults [11,22].
Embodied conversational agents supporting virtual companions to be used for extended periods of time require special design considerations compared to systems that are either only used for brief interactions or do not intend to engage the user in social interaction [22]. Relational agents, a term introduced and explored by Bickmore [11], are embodied conversational agents designed to form long-term relationships with their users. They are distinct from other types of virtual agents in their ability to imitate the way people incrementally get to know and trust each other through conversations [11]. They often maintain the memory of specific interactions, recalling and referring to them later to evolve relationships with their users [11]. In this respect, Kasap and colleagues [39] discuss an embodied conversational agent designed for repeated interaction. The agent maintains a relationship model of the user, which is updated based on the emotional content of events during a session. This model biases the agent’s mood and indirectly influences its behavior.
A handful of studies have been conducted in which autonomous embodied conversational agents were installed for prolonged periods in the daily living environments of older adults. For instance, in an exploratory pilot study by Ring and colleagues [38], a virtual companion designed to provide longitudinal social support to isolated older adults using empathetic feedback was placed in the houses of 14 older adults for a week. Results demonstrated significant reductions in loneliness based on self-reported mood. In turn, a randomized controlled trial was conducted by Bickmore and colleagues [22]: a virtual exercise coach designed to encourage sedentary older adults to walk more was installed in their houses for two months, followed by another period of ten months, so that the participants had the opportunity to continue the interaction in a kiosk in their clinic waiting room. The trial results showed that participants in the intervention group walked significantly more on average than participants from the control group.
Nonetheless, most of the developed virtual companions were designed for specific controlled environments and have rarely made the step out of the laboratory as autonomous applications in real-world settings. Consequently, there is a lack of knowledge about how autonomous virtual companions perform and which factors influence their acceptance and success in different contexts. To develop useful and successful virtual agents that engage their users in beneficial long-term relationships, there is the need to integrate these systems seamlessly in real-world environments and make them capable of interacting with humans autonomously, in an intuitive, natural and trouble-free way in everyday situations [40,41].

2.2. Usefulness, Satisfaction and Ease of Use

During the past two decades, in the context of human–computer interaction practices, the concept of user experience [42] has been widely used. The concept aims to embrace the total usage phenomenon of a system, product, or service, namely a digital solution, including the emotional impact that it can have in the users’ lives [43]. For the International Organization for Standardization, user experience is related to the users’ perceptions and responses (i.e., emotions, beliefs, preferences, perceptions, comfort, behaviors), that result from the use and/or anticipated use of a system, product, or service, which is dependent on each user personality and prior experiences, attitudes, skills, and abilities, as well as the context of use [44].
User experience is grounded mainly in intangible aspects, which means that different individuals and different usages of the same system, product, or service can result in different user experiences [45]. The user experience is, therefore, unique to each individual and for a specific moment in time, which means that the user experience is not able to be measured [45]. However, it is possible to operationalize this complex concept into different dimensions, which can then be characterized and assessed.
One of the most important components that contributes to and modulates the user experience of digital solutions is usability. According to ISO 9241-11, usability refers to the “extent to which a system, product or service can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [46] (pp. 1–2). In turn, assessment of usability is crucial to minimize the probability of errors and undesirable consequences, and to increase the probability of use by a large proportion of the target users [46,47]. In this respect, it should be noted that, when thinking about digital solutions to support healthcare, usability contributes to enhance patient safety and quality of care [48], although good assessment of usability is not a common practice [48,49,50].
Since there are different methods of usability assessment with different strengths and weaknesses, and varying abilities to analyze different aspects of digital solutions, it is recommended to use a multi-method approach for a specific assessment [51,52]. Certain methods rely on usability experts, while others rely on end-users; the first are known as analytical models [53,54,55], and the latter, known as empirical models, might contribute to highly usable digital solutions by capturing the perspective of future users [53], and include test and inquiry methods [56]. Test methods involve observing users while they perform predefined tasks and consists of collecting mostly quantitative data. Examples include performance evaluation (i.e., gathering data related to the execution of a task by the users) or observation (i.e., attentive visualization and systematic recording of a phenomenon, including people, artifacts, environments, behaviors, and interactions). In turn, inquiry methods aim to collect valuable information on the users’ opinions and expectations, namely by using focus groups, interviews, and scales or questionnaires. There are validated scales (i.e., focused on a unique dimension) and questionnaires (i.e., focused on multiple dimensions) to collect data on characteristics, thoughts, feelings, perceptions, behaviors, or attitudes of the users.
The System Usability Scale (SUS), a ten-statement scale [57], is considered a reliable tool for measuring usability and has been well-established and generalized, with a significant number of publications [58]. However, questionnaires, such as the Post-Study System Usability Questionnaire (PSSUQ) [59] or Usefulness, Satisfaction, and Ease of use (USE) [60], can complement the measurement of usability with other user experience dimensions.
In the literature, there are a few studies addressing the usability assessment of embodied conversational agents. For instance, two recent reviews of the literature [18,61] identified that the usability assessment was an objective of several primary studies. Although the use of SUS has been reported (e.g., to assess an embodied conversational agent to support diabetes treatment [62]) most studies do not report the use of validated instruments, which impairs the generalization of the results. In turn, semi-structured, individual interviews [63], observation of task completion [64], group discussions [65], or specific questionnaires [23,63] have been used to identify usability issues.
Although the use of scales and questionnaires has the advantage of providing quantitative results that can be generalized and compared, it should be noted that they must be validated to guarantee they have similar results in repeated measurements in similar circumstances. This means that the literature shows a lack of quantitative knowledge of both older adults’ experiences with virtual companions based on embodied conversational agents and the potential of virtual companions to support care provision. In this respect, the article reports an observational study using a multi-method approach, which includes the USE scale, to determine the perceptions of older adults needing formal care about the usefulness, satisfaction, and ease of use of a virtual companion based on an embodied conversational agent.

3. Materials and Methods

3.1. Research Questions

The objectives of the study reported by this article (i.e., to determine the perceptions of older adults needing formal care about the usefulness, satisfaction and ease of use of a virtual companion based on an embodied conversational agent, CaMeLi, and to analyze its potential, according to the perspective of the formal caregivers) informed the following research questions:
  • RQ1. What are the perceptions of older adults needing formal care about the usefulness, satisfaction, and ease of use of CaMeLi?
  • RQ2. What types of features of virtual companions based on embodied virtual agents might represent barriers to older adults?
  • RQ3. What are the perceptions of formal caregivers about potential contributions to care provision by virtual companions?
The usability assessment represents an important aspect during the development phase of technological solutions to optimize the user acceptance. However, in the study reported by the present article, the aim was not only to assess usability to improve a digital solution, but also to contribute to current understanding of the perceptions of older adults about the usefulness, satisfaction, and ease of use of virtual companions based on embodied conversational agents, and the perceptions of care providers about the potential of virtual companions to support care provision.

3.2. The CaMeLi Daily Life Companion for Older Adults

CaMeLi is a virtual companion created to support older adults to live independently and, consequently, to promote healthy and active ageing [9]. It works through an embodied conversational agent, able to interact with natural language, with which older adults can speak and interact by asking help to perform some daily life activities at home. Providing simple information about the daily schedule, contacting family or friends through Skype, consulting the news, choosing meals, or reminding users (e.g., about where they stored specific objects) are some of the functions provided by CaMeLi [10].
CaMeLi includes components for speech recognition, dialogue management, and synthesis of speech, sound, and movement. The virtual companion operates on an all-in-one stationary computer (i.e., Lenovo ThinkCentre Edge 93z All-in-One), mounted on the wall or placed on top of furniture, at a height ranging from 80 to 120 cm, allowing the user to interact from a standing or sitting position. A built-in microphone is used to enable speech as an input modality, and high definition speakers are used for producing audio output.
The virtual companion is presented through a 3D human, female gender (by default), called “Glória”. Its appearance presents a realistic and friendly face, and the voice is built to transmit a comfortable feeling.
Figure 1 presents the dashboard of CaMeLi. The embodied conversational agent is presented in the lower left corner, in the middle of the screen the main menus are displayed (i.e., web, calendar, menu, news, Skype, and games), and at the right side of the screen the weather forecast and a summary of the latest news are presented. On the top of the screen, information to orient the users is presented, namely date and hour. In addition, in the upper left corner of the screen are displayed the microphone button, mute button, and the help button (with instructions about voice commands and information about the project). It is also possible to alter the gender of the embodied conversational agent by changing to the male version (in the lower left corner of the screen).
CaMeLi closely simulates human conversational behavior using synthesized voice and synchronized non-verbal behaviors such as head nods and facial expressions.
The users interact with CaMeLi using a multimodal interface including automatic speech recognition and a graphical touch-based user interface. The design features of the speech interface and the user interface are consistent to simplify the learning process.
The overall architecture, which gives rise to the human-like abilities, was described in detail by Tsiourti and colleagues [9].

3.3. Study Design

The research study comprised two phases. During the first phase, an observational study was conducted. A multi-method approach comprising self-reported measures, measures reported by observers, and critical incident registration was implemented to capture the perceptions of a group of older adults needing formal care about the usefulness, satisfaction, and ease of use of CaMeLi.
The second phase comprised a focus group to capture the perceptions of the caregivers regarding potential contributions to the care provision of CaMeLi in particular, and virtual companions based on embodied conversational agents in general.

3.4. Observational Study

3.4.1. Ethical and Regulatory Considerations

The observational study was approved by the Portuguese Data Protection Commission and by the Ethical Committee of the University of Aveiro (process number 18/2018, July 2018) and was conducted according to the ethical principles that have their origin in the Declaration of Helsinki [66], the Good Epidemiological Practice guidelines [67], and all of the applicable laws and regulations.
Therefore, all participants signed an informed consent prior to their participation. Before providing informed consent, participants received an information sheet outlining the study objectives, duration, and methods. Elements of the research team explained that participants could request additional information about the study at any moment and abandon the study at any time without any explanation or personal prejudice.
All steps were taken to protect participants’ privacy and all relevant rules on data privacy were followed. Information allowing identification of participants was not captured in the study database. The participants were distinguished in the study documents by a unique running number. A document allowing establishment of a link between the running number and overt personal identifiers (e.g., name, address, or phone number) was locked with access limited to the principal investigator.
The researchers involved in the conduct and reporting of the study were obliged to professional secrecy. The principal investigator was responsible for making sure that all members of the team were aware they must not reveal information obtained by having access to research data to anyone outside the scientific research team.

3.4.2. Participants

Participants comprised a convenience sample of older adults and were recruited either by referral from caregivers or direct invitation by researchers, among those attending day care-centers of a non-profit social organization, Cáritas Diocesana de Coimbra (Cáritas).
Those willing to participate were eligible if they were aged over 60 years, were attending the Cáritas day care-centers, were able and willing to give their informed consent, and had experience in using CaMeLi (i.e., they have used CaMeLi for at least three months in activities they performed in Cáritas day care-centers). Participants were excluded from the study if they were unable to understand the study data collection forms, due to illiteracy (i.e., the participants were not able to explain in their own words the information provided by the evaluator about the study and the data collection form) or cognitive impairment (i.e., according the opinion of the head of the multidisciplinary team that assess and follows the older adults), or if they were not able to comply with the study protocol, due to health conditions or personal circumstances.

3.4.3. Observers

The five researchers involved in the conduct of the experiment (i.e., observers) were experienced in research related to older adults, namely, to assess the impact of digital solutions. In turn, none of the observers were involved in the technological development of CaMeLi.
For this specific observational study, an observation guide was prepared to ensure the reliability of the data collection. This guide was discussed in a meeting involving the research team. Then, a trial was conducted to determine the adequacy of the guide and the preparation of the observers. Finally, the performance of the observers during this trial was discussed by the team and some clarifications were introduced in the observation guide.

3.4.4. Assessment Procedures

The researchers conducted the experimental sessions with the participants in the premises of Cáritas. A single participant experienced the system each time, accompanied by three observers.
Each session was conducted by one of the observers and observed the following steps: (i) pre-test—the observer provided a basic introduction and the participants completed a sociodemographic questionnaire for sample characterization, with demographic information and information about the digital devices the participants were familiar with; (ii) test—the participants interacted with CaMeLi following the tasks described in a session script; simultaneously, two other observers took notes about the performance of the participants and recorded the critical incidents; (iii) post-test—the participants completed a self-reported questionnaire, the Portuguese version [68] of USE [60], and the observers completed the International Classification of Functioning Disability and Health-based Usability Scale (ICF-US) [69], according to their opinion about the interaction of the participant with CaMeLi.

3.4.5. Assessment Instruments

USE [60] allows the capture of perceptions of the participants (i.e., it is an auto-reported questionnaire) by identifying issues that may influence and determine frequency of use and user satisfaction.
The selection of USE to determine the perception of the older adults was due to the fact that the 30 items of this questionnaire were developed to assess multiple dimensions: (i) usefulness (items 1 to 7); (ii) ease of use (items 8 to 19); (iii) ease of learning (items 20 to 23); and (iv) satisfaction (items 24 to 30). Each item is scored in a seven-point Likert scale, ranging from 1 (strongly disagree) to 7 (strongly agree). The questionnaire analysis is based on the average of the users’ answers, and the average of each dimension can be analyzed in a disaggregated way [60].
The ICF-US [69] is a generic usability assessment scale that can be used for self-reported usability, as well as usability assessment based on the opinion of observers. This tool consists of two subscales: (i) ICF-US I, which allows a comprehensive usability assessment and the identification of participants that have difficulties in using the application being assessed (i.e., for them the application is a barrier) from the participants without significant difficulties (i.e., for them the application is a facilitator); and (ii) ICF-US II, which allows the classification of application components as barriers and facilitators, identifying the application strengths and weaknesses [69,70]. To fulfil the two subscales, observers must consider the performance of the participants during the evaluation. Both subscales score all items from −3 (less positive) to 3 (most positive) values.
The final score of the ICF-US I is calculated by the sum of the scores of all items. A value above 10 points indicates a good usability (i.e., facilitator) and less than 10 is considered a prototype with opportunities to improve (i.e., barrier).
The ICF-US II subscale is adaptable to address specific components of the application being assessed. Therefore, when an item is classified as a barrier, it is possible to identify the feature that is influencing that classification to determine what can be improved. On the other hand, the features identified as facilitators can be noted as good practices for future developments [56,71].

3.4.6. Critical Incident Registration

During each experimental session, while the participant was executing the tasks, two observers recorded the critical incidents to systematically identify behaviors that contribute to the success or failure of the participants in specific tasks. In this record some details were considered, such as easy/difficult interaction with CaMeLi or learning ability of the sequence needed to complete a task.

3.4.7. Tasks

The following tasks served as the base for the assessment of usefulness, satisfaction, and ease of use, as they are representative of CaMeLi’s functionalities and encompass various interaction mechanisms: (i) on the main screen, using the touch-based user interface, click on the “Games” menu; (ii) select the game “Letter Soup”; (iii) hide the virtual companion; (iv) identify at least three words on the game board; (v) view the list of available games again and select the “Game of the Rooster”; (vi) consult the “News” menu using a voice command (for example: “Open News” or “Read News”); (vii) select and read a news story of your choice; (viii) make the virtual companion visible again; (ix) select the “Agenda” menu and then the “Meals” menu, using touch-based commands; and (x) disconnect the microphone and reconnect again. Table 1 presents the type of command (i.e., touch or voice) related to each task.

3.4.8. Data Analysis

Descriptive data was analyzed in terms of mean ± standard deviation (SD) or frequency. Moreover, since ICF-US allows the identification of two different groups of participants (i.e., the barriers and facilitators groups) an independent samples t-test was used to compare the USE total score and the scores of the four USE dimensions between the two groups. The significance level was set at p < 0.05.
Statistical analyses were conducted with Statistical Package for the Social Sciences (SPSS) Statistics for Windows, version 23.0 (IBM Corp, Armonk, NY, USA).

3.5. Focus Group

The second phase of this study included an analysis of the perceived utility of virtual companions in terms of care provision. To achieve this, data was gathered from a focus group. The use of a focus group is a qualitative data collection technique that is highly popular in several contexts, which brings together a small number of people and promotes informal discussion on a specific topic [72]. This method aims to extract participants’ perceptions, feelings, attitudes, and ideas about a particular subject. One researcher should assume the moderator role, being responsible for introducing the topics, promoting participation, and maintaining the discussion [72].
The focus group comprised four participants. These participants were recruited by direct invitation by the research team. In terms of eligibility to participate in the focus group, the research team defined that the participants should be experienced caregivers with at least three years of experience in caring for older adults and be familiar with the potential of digital solutions to support older adults. Moreover, as exclusion criterion, it was defined that participants should not have been involved in the development of CaMeLi.
At the beginning of the focus group, all participants were introduced to the scope of the research and were informed that they had the right to discontinue their participation without any repercussions.
Then, the participants of the focus group were asked to give their opinion about, among other topics, whether the embodied virtual companions humanize contact with technology, the system usefulness for older adults living in their homes or living in institutions, or other functionalities that would be interesting to implement in future versions of CaMeLi.
The focus group was led by one member of the research team acting as a facilitator by providing instructions and guiding the discussion. Two other members of the research team were responsible for taking notes. Both the facilitator and the members of the research team responsible for taking notes had participated in research projects focusing older adults.
All the interventions were audio recorded and transcribed. Thematic analysis was then performed. Two researchers independently coded the focus group transcript and the results were interpreted and discussed by the research team.

4. Results

4.1. Observational Study

The observational study took place between September and October 2018 [73] in five different day care-centers of Cáritas.
The total sample consisted of 46 participants, 34 females (73.91%) and 12 males (26.09%), with an average age of 63.6 years (SD = 20.5). All the participants were retired and formally requested care services that Caritas provides to older adults due to physical or cognitive impairments; particularly, all of them were participating in the activities of Cáritas day care-centers. In terms of marital status, 19 participants were windowed (41.30%), 12 were married (26.09%), eight were divorced (15.22%) and seven were unmarried (17.39%). More than half of the sample had internet at home (60.87%), almost half used a computer or tablet (45.65%), and only 28.26% used a smartphone. All the participants had been using CaMeLi for at least three months.
In each session, the participant was accompanied by three observers. The overall experimental process lasted approximately 40 min for each participant and, on average, each one interacted with CaMeLi for approximately 20 min, varying between a minimum of 15 and a maximum of 25 min.
The results of USE and ICF-US I are presented in Table 2 and Table 3. Since the ICF-US I allows the classification of the participants into two groups (i.e., barriers and facilitators group), it was possible to conclude that, according ICF-US I, CaMeLi was a barrier for 11 participants and a facilitator for 35 participants. Therefore, the referred Table 2 and Table 3 present the mean value and SD of the scores of the different items of the scales for all the participants and for the two groups. In this respect, using an independent samples t-test, with a significance level set at p < 0.05, statistically significant differences were found between the two groups for the USE total score (p = 0.04) and for the scores of the usefulness (p = 0.014) and satisfaction (p = 0.041) dimensions of USE.
The average total score of the USE for all participants was 5.06 out of a maximum of 7 (SD = 1.10), which indicates that CaMeLi presents a good degree of usefulness, satisfaction, and ease of use. The results for the four dimensions associated with USE, listed in descending order of average scores, were: (i) satisfaction (mean value = 5.28; SD = 1.26); (ii) usefulness (mean value = 5.14; SD = 1.12); (iii) ease of learning (mean value = 5.02; SD = 1.35); and (iv) ease of use (mean value = 4.90; SD = 1.17).
Items with higher scores were related to the usefulness and satisfaction of the CaMeLi, namely, (i) “I would recommend it to a friend” (mean value = 5.70; SD = 1.33); (ii) “It is fun to use” (mean value = 5.65; SD = 1.22); and (iii) “It is useful” (mean value = 5.46; SD = 1.13). In turn, the items that obtained lower scores were related with the ease of use dimension: (i) “I can use it successfully every time” (mean value = 4.41; SD = 1.67); (ii) “I can use it without written instructions” (mean value = 4.59; SD = 1.71); and (iii) “It is user friendly” (mean value = 4.65; SD = 1.65).
Considering the results of ICF-US I, in a range from −30 to 30, the mean score for all participants was 15.93 (SD = 16.57), indicating that, in general, the application was a facilitator. The highest participant score was 30, and the lowest was −25.
Considering the average of the different items of ICF-US I, none of the items had scores lower than zero, which means that, on average, barriers were not identified. In turn, the facilitator results, in order of importance, were (i) “The degree of satisfaction with the use” (mean value = 2.02; SD = 1.51); and (ii) “The similarity of the way it works on different tasks (e.g., to confirm an action is always equal)” (mean value = 1.85; SD = 1.51). According to the identified facilitators, it seems that most participants were satisfied using CaMeLi and considered it to be consistent.
The ICF-US II subscale was applied to those participants of the group classified as barriers (i.e., 11 participants) and the detailed results are presented on Table 4.
The results of the parts associated with ICF-US II were analyzed according to the mean values of the items (range −3 to 3):
  • Application Components (three items)—average of the responses = −0.90 (SD = 0.62).
  • Detailed Usability (12 items)—average of the responses = −0.70 (SD = 1.19).
  • Overall Assessment (one item)—average of the responses = −1.23 (SD = 0.93).
The results of the application of ICF-US II indicate that several components of CaMeLi could be improved, particularly because the lowest results referred to the overall assessment. The identified barriers, in order of severity, were (i) the voice interaction (mean value = −2.46; SD = 0.78); (ii) the layout change (hide the virtual companion) (mean value = −1.69; SD = 1.32); and (iii) the virtual companion itself (mean value = −1.54; SD = 1.13). Thus, it seems that participants had difficulty in tasks that encompass the voice interaction and hiding the virtual companion. In turn, the items with the higher results identified by facilitators were all related to the design of the graphic interface, namely the icon specifications: (i) the contrast of the icons (mean value = 1.15; SD = 1.77); (ii) the color of the icons (mean value = 1.00; SD = 1.63); and (iii) the size of the icons (mean value = 0.62; SD = 1.56).
Regarding the critical incidents resulting from the observation process, 17 participants reported difficulties in obtaining a CaMeLi response to voice commands and, in 14 cases, the task had to be carried out by the observers or through the touch-based interface.
The requested tasks that raised problems for the participants were hiding the virtual companion and making it visible again (n = 19), turning off the microphone (n = 6), and identifying the return and locating it on the screen (n = 4).
Three participants suggested that these actions could be completed through voice commands. When the virtual companion is hidden, the layout of the elements on the screen change, spreading on the screen, and in these situations four participants revealed problems executing the tasks. Eight participants had difficulty interacting with the touch-based user interface due to their lack of manual dexterity, and five also revealed problems browsing the system pages using the scroll function to select element/buttons on the screen. Some of them, unintentionally, touched two points of the screen simultaneously, and indicated a lack of understanding of what was happening in the system (n = 8). These results are consistent with the outcomes of the application of the USE and ICF-US scale.
The existence of links for external sites, such as advertising or news sites, led to confusion among six participants when they inadvertently clicked on these options, without realizing what was happening.

4.2. Focus Group

The focus group sample consisted of four participants (three female and one male) with an average age of 38.1 with a minimum of 26 and a maximum of 45 years old. All the participants had been working directly with older adults for more than three years and were competent in using software applications, namely in their daily office activities. All the participants knew the CaMeLi and had already had the opportunity to interact with it.
The focus group lasted approximately 70 min and the respective guide included the following questions: (i) what impression do you have in relation to what users think about CaMeLi? (ii) do you consider that the avatar humanizes the interaction with technology? (iii) do you think that CaMeLi would be useful for older adults at home? (iv) and for institutionalized older adults? and (v) what is the added value of CaMeLi for your institution?
According to the results of the thematic analysis that was performed using the transcript of the focus group discussion, the participants of the focus group were unanimous about the potential of CaMeLi in the living environment of older adults. For instance, one participant stated that “CaMeLi should be used as a tool to encourage healthy and active ageing habits…”, while another focused on the possibilities in terms of cognitive and physical stimulation, arguing that CaMeLi “… has a great potential to be used for cognitive and physical stimulation…”. The group also recognized that the interaction mechanisms and iconography of CaMeLi seems to be perceptible by older adults, which might facilitate its dissemination. In this respect, a participant argued that “… the fact that CaMeLi is so simplistic, and the icons are large and familiar makes a difference when the target users are older adults”.
It was assumed that virtual companions represent an added value for combatting social isolation. Regarding CaMeLi, the participants highlighted its value in entertainment, which promotes socialization. In this respect a participant stated “… CaMeLi is another aid to fight against social isolation in institutions, since it can be used individually, as well as in pairs or groups”. Moreover, participants revealed that CaMeLi was important to diversify the offer of entertainment activities in the institution. For instance, the participants stated: “… CaMeLi turns out to be a great alternative in terms of activities…”); “… often, when users have no interest in participating in the activities of day care-centers, CaMeLi arises as an alternative source of entertainment”. In addition, the participants also envisaged the usefulness of CaMeLi to help older adults accomplish their daily routine activities (e.g., “CaMeLi could allow customization of users’ preferences and be tailored to their routines…”).
Moreover, according to the participants’ opinion, CaMeLi, and virtual companions in general, can contribute to the humanization of technological solutions, since, as one participant said, “the users interact with them as real persons”. They also considered that the opinions of older adults who had contact with the technology changed through time. They considered that there was some initial reluctance to use CaMeLi; however, the demystification of technological complexity, the continued use of CaMeLi, and the feeling of belonging to a group made it possible to overcome these barriers. In this respect, the participants stated: “… most users have verified the potential of the virtual companion for themselves and have expressed intention to use the system…”; “I found some resistance from the older adults to use the system in the first days, but as they were experimenting, they became more comfortable and independent using it…”; “… indeed, some older adults only started using it after seeing that their colleagues could use and liked it”; “In the beginning, they did not want to use CaMeLi, now there are discussions to dispute the CaMeLi…”.
Although the system has not reached a suitable maturity level in terms of speech recognition and tactile interactions, which can be reflected in the users’ performance when using these features, it is recognized that it is easier to perform activities on systems with tactile screens than traditional desktop systems (e.g., “CaMeLi is an application that makes the difference when dealing with older adults”). Participants of the focus group also referred the need to enhance several aspects of CaMeLi, namely, by diversifying the games that are available and promoting real time information updates. For instance, one participant recognized that “… it would be good to have more diversifying games”, while another stated “… the older adults that used to use CaMeLi to play games back in the institution were complaining about the poor variety of games”.
The possibility of having CaMeLi in the users’ homes was also discussed. The general opinion of the participants of the focus group was that the current maturity level of the CaMeLi makes it unsuitable for commercialization: “CaMeLi is great, and has a huge potential, but I do not think it is ready for commercialization yet”, stated a participant. When it comes to the use of CaMeLi at home, participants considered that security and privacy issues, for instance regarding sound collection for the voice interaction mechanism, should be completely solved.
Nevertheless, future developments to improve the potential of digital solutions such as CaMeLi at home, once all technical issues could be overcome, were debated. The integration of technologies such as artificial intelligence would allow customization of users’ preferences, characteristics, routine, and lifestyle, thereby improving the virtual companion. Another innovative idea was integration with home automation by using CaMeLi as an agent to manipulate the physical environment. In this respect the participants stated: “in homes of active and healthy older adults, the CaMeLi could be used to control the house, and manipulate the physical environment…”; “… it would be nice to ask the avatar to close the windows, for example…”; “… it could be linked to wearables…”; “… it could also be used to monitor physiologic aspects using data collected through sensors”. In this respect, the participants emphasized once more the need to guarantee the privacy of older adults. For instance, one participant said that he was “… concerned about security and privacy issues because the way CaMeLi works I am not sure if the protection and privacy of the older adults are fully considered”.
As future ideas for CaMeLi, the participants of the focus group also listed features to be implemented in a short-term period, namely, for the reinforcement and promotion of healthy and active ageing habits, such as physical exercise tutorials and cognitive stimulation exercises.

5. Discussion and Conclusions

The results collected during the observational study regarding the first research question (i.e., the perceptions of older adults needing formal care about the usefulness, satisfaction, and ease of use of a virtual companion based on an embodied conversational agent) made it clear that participants considered CaMeLi a useful tool that may represent an added value for their daily lives, both in institutional and home contexts. Participants enjoyed the available functionalities and activities, were satisfied with the application, and indicated that they would recommend it to friends.
Considering the results of the application of the USE questionnaire, the participants reported a good level of acceptance, specifically in the dimensions of usefulness, ease of learning, and satisfaction, which achieved the best results. These results were confirmed by the outcomes of the application of ICF-US I, with all the items identified as facilitators.
The positive results obtained by the application of USE and ICF-US may be a good predictor of a frequent and independent use of CaMeLi.
In terms of the features of a virtual companion based on an embodied virtual agent that might represent barriers to older adults (i.e., the second research question), the item with the lowest score in the USE questionnaire was related to the ability to use the system successfully, which indicates that participants had some difficulties in task completion. These problems were also identified in the ICF-US II and critical incident registration, which also indicated problems for a significant number of participants when using voice interaction for navigation. As a result, the introduction of an additional command (i.e., hide the virtual companion) to switch the navigation mode was suggested.
These results clearly showed that navigation using voice interaction deserves special attention in the development of virtual companions based on embodied conversational agents [65], together with other issues that are current objects of research, such as the limits of text-to speech [62], realism [20], facial expressions [36], and empathy [38] of the virtual companions.
The opinion of the caregivers involved in the focus group corroborated these results. Like in the case of the users, the experts also focused on the need to improve voice interaction, touch-based interaction, and integration of new features.
Considering the third research question (i.e., what are the perceptions of formal caregivers about potential contributions to the care provision by virtual companions based on embodied conversational agents?), ideas emerged for new phases of CaMeLi development, including the use of artificial intelligence, integration with home automation systems, and incorporation of new features, such as tutorials that aim to promote healthy and active ageing.
Additionally, the perspective of the caregivers related to the acceptance of virtual companions in the living environment of older adults was positive, which conforms with the evidence of other studies [6,14,50,63]. Moreover, when considering the possible applications of virtual companions, the participants of the focus group suggested daily schedule management, remote monitoring of physical and mental wellbeing, and socialization, which were identified previously by other studies [6,50]. Article [6] also refers to the usefulness of virtual companions in terms of the safety of older adults, which was not mentioned by the focus group.
Although some features need to be improved, it appears that virtual companions based on embodied conversational agents present a potential mean of providing support care to older adults, including the promotion of socialization and healthy and active ageing habits, such as physical activity and cognitive stimulation. Therefore, opportunities can be envisaged for the use and commercialization of virtual companions such as CaMeLi.
This study has a few limitations that must be considered in the interpretation of the results. First, the observational study featured a sample size that is adequate for a simple usability assessment, but is not large enough to draw definitive conclusions about perceived usefulness, satisfaction, ease of use and potential of virtual companions to support the care provision for older adults. Additionally, all the participants in the observational study had previous experience in using CaMeLi, which means that the inclusion of a group of participants without previous experience would be interesting, so that a comparation could be established. Moreover, since all the participants formally requested care services provided by Cáritas, the results cannot be generalized for other older adults’ population groups, namely healthy and active older adults that do not require formal care. Finally, still in terms of the observational study, the results show that CaMeLi presents some usability issues, which might had influenced the results.
Concerning the focus group, the number of participants is also small and only represents a partial perspective, which means that it would be interesting to organize additional focus groups to achieve a more general perspective (e.g., involving older adults, informal caregivers or healthcare professionals).
In terms of future work, CaMeLi is being object of further developments, namely, to solve usability issues, and it is planned the conduction of other observational studies to assess its acceptance by other older adults’ population groups and to identify personal characteristics and motivations that determine its acceptance. Moreover, an application of CaMeLi to support cognitive training is being studied in a European research project.

Author Contributions

Conceptualization, A.L.J., C.D., J.Q., N.P.R.; methodology, A.I.M., H.C., A.F.R., N.P.R.; Software, J.Q.; investigation, J.D., A.L.A., H.C., A.F.R., A.I.M.; writing—original draft preparation, H.C., A.F.R., A.I.M., N.P.R.; writing—review and editing, N.P.R. All authors have read and agreed to the published version of the manuscript.

Funding

This research was partially funded by the European Commission within the H2020-PHC-2014 GrowMeUp project, Grant Agreement 643647, and by European Ambient Assisted Living Joint Programme (Active and Assisted Living—ICT for ageing well, Call 2017) and Fundação para a Ciência e Tecnologia (FCT), Grant Agreement AAL/0005/2017.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Genet, N.; Boerma, W.G.; Kringos, D.S.; Bouman, A.; Francke, A.L.; Fagerström, C.; Devillé, W. Home care in Europe: A systematic literature review. BMC Health Serv. Res. 2011, 11, 207. [Google Scholar] [CrossRef] [Green Version]
  2. Santana, S.; Dias, A.; Souza, E.; Rocha, N. The Domiciliary Support Service in Portugal and the change of paradigm in care provision. Int. J. Integr. Care 2007, 7, e01. [Google Scholar] [CrossRef] [Green Version]
  3. Wiles, J.L.; Leibing, A.; Guberman, N.; Reeve, J.; Allen, R.E.S. The Meaning of “Ageing in Place” to Older People. The Gerontol. 2012, 52, 357–366. [Google Scholar] [CrossRef] [PubMed]
  4. Bedaf, S.; Gelderblom, G.J.; Syrdal, D.S.; Lehmann, H.; Michel, H.; Hewson, D.; de Witte, L. Which activities threaten independent living of elderly when becoming problematic: Inspiration for meaningful service robot functionality. Disabil. Rehabil. Assist. Technol. 2014, 9, 445–452. [Google Scholar] [CrossRef] [Green Version]
  5. Queirós, A.; Pereira, L.; Dias, A.; Pacheco Rocha, N. Technologies for Ageing in Place to Support Home Monitoring of Patients with Chronic Diseases. In Proceedings of the 10th International Joint Conference on Biomedical Engineering Systems and Technologies, Porto, Portugal, 21–23 February 2017. [Google Scholar] [CrossRef]
  6. Tsiourti, C.; Joly, E.; Wings, C.; Ben Moussa, M.; Wac, K. Virtual Assistive Companions for Older Adults: Qualitative Field Study and Design Implications. In Proceedings of the 8th International Conference on Pervasive Computing Technologies for Healthcare, Oldenburg, Germany, 20–23 May 2014. [Google Scholar] [CrossRef] [Green Version]
  7. Cassell, J. Embodied conversational interface agents. Commun. ACM 2000, 43, 70–78. [Google Scholar] [CrossRef]
  8. Cooper, R.S.; McElroy, J.F.; Rolandi, W.; Sanders, D.; Ulmer, R.M.; Peebles, E. Personal Virtual Assistant. U.S. Patent No. 6,757,362, 2008. Available online: https://patents.google.com/patent/US8000453B2/en (accessed on 16 June 2020).
  9. Tsiourti, C.; Ben Moussa, M.; Quintas, J.; Loke, B.; Jochem, I.; Lopes, J.A.; Konstantas, D. A virtual assistive companion for older adults: Design implications for a real-world application. In Proceedings of the 2016 SAI Intelligent Systems Conference, London, UK, 21–22 September 2016; Springer: Cham, Switzerland, 2016. [Google Scholar] [CrossRef]
  10. Cáritas Diocesana de Coimbra. International Project Cameli. Available online: https://www.caritascoimbra.pt/en/project/cameli/ (accessed on 16 June 2020).
  11. Bickmore, T.W. Relational Agents: Effecting Change through human-Computer Relationships; Massachusetts Institute of Technology: Cambridge, MA, USA, 2003. [Google Scholar]
  12. Bickmore, T.; Gruber, A.; Picard, R. Establishing the computer-patient working alliance in automated health behavior change interventions. Patient Educ. Couns. 2005, 59, 21–30. [Google Scholar] [CrossRef]
  13. Schulman, D. Embodied Agents for Long-Term Interaction; Northeastern University: Boston, MA, USA, 2013. [Google Scholar]
  14. Kramer, M.; Yaghoubzadeh, R.; Kopp, S.; Pitsch, K. A conversational virtual human as autonomous assistant for elderly and cognitively impaired users? In Proceedings of the Social acceptability and design considerations, Informatik 2013, Informatik angepasst an Mensch, Organisation und Umwelt, Koblenz, Germany, 16–20 September 2013. [Google Scholar]
  15. Klein, J.; Moon, Y.; Picard, R.W. This computer responds to user frustration. Interact. Comput. 2002, 14, 119–140. [Google Scholar] [CrossRef]
  16. Nijholt, A. Disappearing computers, social actors and embodied agents. In Proceedings of the 2003 International Conference on Cyberworlds, Singapore, 3–5 December 2003. [Google Scholar] [CrossRef] [Green Version]
  17. Vardoulakis, L.P.; Ring, L.; Barry, B.; Sidner, C.L.; Bickmore, T. Designing Relational Agents as Long-term Social Companions for Older Adults. In Proceedings of the 12th International Conference on Intelligent Virtual Agents (IVA’2012), Santa Cruz, CA, USA, 12–14 September 2012; Springer: Berlin/Heidelberg, Germany, 2012. [Google Scholar] [CrossRef] [Green Version]
  18. Ter Stal, S.; Kramer, L.L.; Tabak, M.; op den Akker, H.; Hermens, H. Design Features of Embodied Conversational Agents in eHealth: A Literature Review. Int. J. Hum. -Comput. Stud. 2020, 138, 102409. [Google Scholar] [CrossRef]
  19. Schmeil, A.; Suggs, S. “How am I doing?” Personifying health through animated characters. In Proceedings of the Third International Conference of Design, User Experience, and Usability, Heraklion, Greece, 22–27 June 2014; Springer: Cham, Switzerland, 2014. [Google Scholar] [CrossRef]
  20. Wissen, A.V.; Vinkers, C.; Halteren, A.V. Developing a virtual coach for chronic patients: A user study on the impact of similarity, familiarity and realism. In Proceedings of the 11th International Conference on Persuasive Technology, Salzburg, Austria, 5–7 April 2016; Springer: Cham, Switzerland, 2016. [Google Scholar] [CrossRef]
  21. Zhou, S.; Zhang, Z.; Bickmore, T. Adapting a persuasive conversational agent for the Chinese culture. In Proceedings of the 2017 International Conference on Culture and Computing (Culture and Computing 2017), Kyoto, Japan, 10–12 September 2017. [Google Scholar] [CrossRef]
  22. Bickmore, T.W.; Silliman, R.A.; Nelson, K.; Cheng, D.M.; Winter, M.; Henault, L.; Paasche-Orlow, M.K. A Randomized Controlled Trial of an Automated Exercise Coach for Older Adults. J. Am. Geriatr. Soc. 2013, 61, 1676–1683. [Google Scholar] [CrossRef]
  23. Martínez-Miranda JMartínez, A.; Ramos, R.; Aguilar, H.; Jiménez, L.; Arias, H.; Valencia, E. Assessment of users’ acceptability of a mobile-based embodied conversational agent for the prevention and detection of suicidal behaviour. J. Med Syst. 2019, 43, 246. [Google Scholar] [CrossRef]
  24. Ring, L.; Utami, D.; Bickmore, T. The right agent for the job? The effects of agent visual appearance on task domain. In Proceedings of the 2014 International Conference on Intelligent Virtual Agents (IVA’2014), Boston, MA, USA, 26–29 August 2014; Springer: Cham, Switzerland, 2014. [Google Scholar] [CrossRef]
  25. Zhou, S.; Bickmore, T.; Paasche-Orlow, M.; Jack, B. Agent-user concordance and satisfaction with a virtual hospital discharge nurse. In Proceedings of the 2014 International Conference on Intelligent Virtual Agents (IVA’2014), Boston, MA, USA, 26–29 August 2014; Springer: Cham, Switzerland, 2014. [Google Scholar] [CrossRef] [Green Version]
  26. Robertson, S.; Solomon, R.; Riedl, M.; Gillespie, T.W.; Chociemski, T.; Master, V.; Mohan, A. The visual design and implementation of an embodied conversational agent in a shared decision-making context (eCoach). In Proceedings of the 2015 International Conference on Learning and Collaboration Technologies, Las Vegas, NV, USA, 15–20 July 2015; Springer: Cham, Switzerland, 2015. [Google Scholar] [CrossRef] [Green Version]
  27. Parmar, D.; Olafsson, S.; Utami, D.; Bickmore, T. Looking the part: The effect of attire and setting on perceptions of a virtual health counselor. In Proceedings of the 18th International Conference on Intelligent Virtual Agents (IVA’2018), Sydney, Australia, 5–8 November 2018. [Google Scholar] [CrossRef]
  28. Alsharbi, B.; Richards, D. Using virtual reality technology to improve reality for young people with chronic health conditions. In Proceedings of the 9th International Conference on Computer and Automation Engineering (ICCAE’17), Sydney, Australia, 18–21 February 2017. [Google Scholar] [CrossRef]
  29. Tielman, M.L.; Neerincx, M.A.; van Meggelen, M.; Franken, I. How should a virtual agent present psychoeducation? Influence of verbal and textual presentation on adherence. Technol. Health Care 2017, 25, 1081–1096. [Google Scholar] [CrossRef] [Green Version]
  30. Philip, P.; Dupuy, L.; Auriacombe, M.; Serre, F.; de Sevin, E.; Sauteraud, A.; Micoulaud-Franchi, J.A. Trust and acceptance of a virtual psychiatric interview between embodied conversational agents and outpatients. npj Digital Med. 2020, 7, 1–7. [Google Scholar] [CrossRef] [Green Version]
  31. Wargnier, P.; Malaisé, A.; Jacquemot, J.; Benveniste, S.; Jouvelot, P.; Pino, M.; Rigaud, A.S. Towards attention monitoring of older adults with cognitive impairment during interaction with an embodied conversational agent. In Proceedings of the 2015 Virtual and Augmented Assistive Technology (VAAT’2015), Arles, France, 23 March 2015. [Google Scholar] [CrossRef] [Green Version]
  32. Creed, C.; Beale, R. User interactions with an affective nutritional coach. Interact. Comput. 2012, 24, 339–350. [Google Scholar] [CrossRef]
  33. Creed, C.; Beale, R.; Cowan, B. The impact of an embodied agent’s emotional expressions over multiple interactions. Interact. Comput. 2015, 27, 172–188. [Google Scholar] [CrossRef]
  34. Olafsson, S.; Kimani, E.; Asadi, R.; Bickmore, T.; Science, I. That’s a rap. In Proceedings of the 2017 International Conference on Intelligent Virtual Agents (IVA’2017), Stockholm, Sweden, 27–30 August 2017; Springer: Cham, Switzerland, 2017. [Google Scholar] [CrossRef]
  35. Das, K.S.; Beinema, T.; Op Den Akker, H.; Hermens, H. Generation of multi-party dialogues among embodied conversational agents to promote active living and healthy diet for subjects suffering from type 2 diabetes. In Proceedings of the 5th International Conference on Information and Communication Technologies for Ageing Well and e-Health (ICT4AWE’2019), Heraklion, Greece, 2–4 May 2019. [Google Scholar] [CrossRef]
  36. Malhotra, A.; Hoey, J.; König, A.; van Vuuren, S. A study of elderly people’s emotional understanding of prompts given by virtual humans. In Proceedings of the 10th EAI International Conference on Pervasive Computing Technologies for Healthcare, Cancun, Mexico, 16–19 May 2016. [Google Scholar]
  37. Llorach, G.; Agenjo, J.; Blat, J.; Sayago, S. Web-based embodied conversational agents and older people. In Perspectives on Human-Computer Interaction Research with Older People; Sayago, S., Ed.; Springer: Cham, Switzerland, 2019; pp. 119–135. [Google Scholar] [CrossRef]
  38. Ring, L.; Shi, L.; Totzke, K.; Bickmore, T. Social support agents for older adults: Longitudinal affective computing in the home. J. Multimodal User Interfaces 2015, 9, 79–88. [Google Scholar] [CrossRef]
  39. Kasap, Z.; Magnenat-Thalmann, N. Building long-term relationships with virtual and robotic characters: The role of remembering. Visual Comput. 2012, 28, 87–97. [Google Scholar] [CrossRef] [Green Version]
  40. Quintas, J.; Menezes, P.; Dias, J. Information Model and Architecture Specification for Context Awareness Interaction Decision Support in Cyber-Physical Human–Machine Systems. IEEE Trans. Hum. -Mach. Syst. 2017, 47, 323–331. [Google Scholar] [CrossRef]
  41. Quintas, J.; Martins, G.S.; Santos, L.; Menezes, P.; Dias, J. Toward a Context-Aware Human-Robot Interaction Framework Based on Cognitive Development. IEEE Trans. on Systems Man Cybern. Syst. 2018, 49, 227–237. [Google Scholar] [CrossRef]
  42. Norman, D.A. The design of everyday things: Revised and Expanded Edition; Basic Books: New York, NY, USA, 2013. [Google Scholar]
  43. Hartson, R.; Pyla, P. The UX Book: Process and Guidelines for Ensuring a Quality User Experience; Morgan Kaufmann: Burlington, MA, USA, 2012. [Google Scholar] [CrossRef]
  44. International Organization for Standardization. Ergonomics of Human-System Interaction—Part 210: Human-Centred Design for Interactive Systems (ISO 9241-210:2019); International Organization for Standardization, 2019. Available online: https://www.iso.org/obp/ui/#iso:std:iso:9241:-210:ed-2:v1:en (accessed on 16 June 2020).
  45. Kim, J.; Park, S.; Hassenzahl, M.; Eckoldt, K. The Essence of Enjoyable Experiences: The Human Needs A Psychological Needs-Driven Experience Design Approach. In Design, User Experience, and Usability; Marcus, A., Ed.; Springer: Berlin/Heidelberg, Germany, 2011; pp. 77–83. [Google Scholar] [CrossRef]
  46. International Standards Organisation. Ergonomics of Human-System Interaction—Part 11: Usability: Definitions and Concepts (ISO 9241-11: 2018); International Organization for Standardization: Geneva, Switzerland, 2008. [Google Scholar]
  47. Bevan, N.; Carter, J.; Earthy, J.; Geis, T.; Harker, S. New ISO standards for usability, usability reports and usability measures. In Proceedings of the 2016 International Conference on Human-Computer Interaction (HCI’2016), Toronto, ON, Canada, 17–22 July 2016; Springer: Cham, Switzerland, 2016. [Google Scholar] [CrossRef]
  48. Middleton, B.; Bloomrosen, M.; Dente, M.A.; Hashmat, B.; Koppel, R.; Overhage, J.M.; Payne, T.H.; Rosenbloom, S.T.; Weaver, C.; Zhang, J. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: Recommendations from AMIA. J. Am. Med. Inform. Assoc. 2013, 20, e2–e8. [Google Scholar] [CrossRef]
  49. Davis, T.L.; DiClemente, R.; Prietula, M. Taking mHealth forward: Examining the core characteristics. JMIR mHealth uHealth 2016, 4, e97. [Google Scholar] [CrossRef]
  50. Behler, R. Military Healthcare System (MHS) GENESIS Initial Operational test and Evaluation Report. Available online: https://www.nextgov.com/media/gbc/docs/pdfs_edit/051118letterng.pdf (accessed on 16 June 2020).
  51. Hanington, B.; Martin, B. Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions; Rockport Publishers: Beverly, MA, USA, 2012. [Google Scholar]
  52. Klaassen, B.; van Beijnum, B.J.; Hermens, H.J. Usability in telemedicine systems—A literature survey. Int. J. Med Inform. 2016, 93, 57–69. [Google Scholar] [CrossRef] [PubMed]
  53. Dix, A.; Dix, A.J.; Finlay, J.; Abowd, G.D.; Beale, R. Human-Computer Interaction; Pearson Education: London, UK, 2003. [Google Scholar]
  54. Cayola, L.; Macías, J.A. Systematic guidance on usability methods in user-centered software development. Inf. Softw. Technol. 2018, 97, 163–175. [Google Scholar] [CrossRef] [Green Version]
  55. Veral, R.; Macías, J.A. Supporting user-perceived usability benchmarking through a developed quantitative metric. Int. J. Hum.-Comput. Stud. 2019, 122, 184–195. [Google Scholar] [CrossRef]
  56. Martins, A.I.; Queirós, A.; Silva, A.G.; Rocha, N.P. Usability Evaluation Methods: A Systematic Review. In Human Factors in Software Development and Design; Saeed, S., Bajwa., I.S., Mahmood, Z., Eds.; IGI Global: Hershey, PA, USA, 2015. [Google Scholar] [CrossRef] [Green Version]
  57. Lewis, J.R. Usability Testing; IBM Software Group: Tempe, AZ, USA, 2006. [Google Scholar]
  58. Brook, J. SUS: A retrospective. J. Usability Stud. 2013, 8, 29–40. [Google Scholar]
  59. Lewis, J.R. Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies. Int. J. Human-Comput. Interac 2002, 14, 463–488. [Google Scholar] [CrossRef]
  60. Lund, A.M. Measuring Usability with the USE Questionnaire. Usability Interface 2001, 8, 3–6. [Google Scholar]
  61. Kramer, L.L.; Ter Stal, S.; Mulder, B.C.; de Vet, E.; van Velsen, L. Developing Embodied Conversational Agents for Coaching People in a Healthy Lifestyle: Scoping Review. J. Med Internet Res. 2020, 22, e14058. [Google Scholar] [CrossRef]
  62. Faddoul, G.; Chatterjee, S. The Virtual Diabetician: A Prototype for a Virtual Avatar for Diabetes Treatment Using Persuasion Through Storytelling. In Proceedings of the 25th Americas Conference on Information Systems (AMCIS’2019), Cancún, Mexico, 15–17 August 2019. [Google Scholar]
  63. Chi, N.C.; Sparks, O.; Lin, S.Y.; Lazar, A.; Thompson, H.J.; Demiris, G. Pilot testing a digital pet avatar for older adults. Geriatr. Nurs. 2017, 38, 542–547. [Google Scholar] [CrossRef]
  64. Wargnier, P.; Benveniste, S.; Jouvelot, P.; Rigaud, A.S. Usability assessment of interaction management support in LOUISE, an ECA-based user interface for elders with cognitive impairment. Technol. Disabil. 2018, 30, 105–126. [Google Scholar] [CrossRef] [Green Version]
  65. Tongpeth, J.; Du, H.Y.; Clark, R.A. Development and feasibility testing of an avatar-based education application for patients with acute coronary syndrome. J. Clin. Nurs. 2018, 27, 3561–3571. [Google Scholar] [CrossRef]
  66. General Assembly of the World Medical Association. World Medical Association Declaration of Helsinki: Ethical principles for medical research involving human subjects. J. Am. Coll. Dent. 2014, 81, 14. [Google Scholar]
  67. International Epidemiological Association. Good Epidemiological Practice (GEP). IEA Guidelines for Proper Conduct in Epidemiologic Research; International Epidemiological Association: Chicago, IL, USA, 2010. [Google Scholar]
  68. Dantas, C.; Jegundo, A.L.; Quintas, J.; Martins, A.I.; Queirós, A.; Rocha, N.P. European Portuguese Validation of Usefulness, Satisfaction and Ease of Use Questionnaire (USE). In Proceedings of the World Conference on Information Systems and Technologies (WCIST’2017), Porto Santo Island, Madeira, Portugal, 11–13 April 2017; Springer: Cham, Switzerland, 2017. [Google Scholar] [CrossRef]
  69. Martins, A.I.; Rosa, A.F.; Queirós, A.; Silva, A.; Rocha, N.P. Definition and Validation of the ICF—Usability Scale. Procedia Comput. Sci. 2015, 67, 132–139. [Google Scholar] [CrossRef] [Green Version]
  70. Martins, A.I.; Queirós, A.; Silva, A.G.; Rocha, N.P. ICF based Usability Scale: Evaluating usability according to the evaluators’ perspective about the users’ performance. In Proceedings of the 7th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (DSAI’2016), Vila Real, Portugal, 1–3 December 2016; ACM Press: New York, NY, USA, 2016. [Google Scholar] [CrossRef]
  71. Martins, A.I.; Queirós, A.; Silva, A.G.; Rocha, N.P. Usability evaluation of ambient assisted living systems using a multi-method approach. In Proceedings of the 7th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Info-exclusion (DSAI’2016), Vila Real, Portugal, 1–3 December 2016; ACM Press: New York, NY, USA, 2016. [Google Scholar] [CrossRef]
  72. Smith, J.A. Qualitative Psychology: A practical Guide to Research Methods; SAGE: Newcastle upon Tyne, UK, 2015. [Google Scholar]
  73. Jegundo, A.L.; Dantas, C.; Quintas, J.; Dutra, J.; Almeida, A.L.; Caravau, H.; Rocha, N.P. Usability Evaluation of a Virtual Assistive Companion. In Proceedings of the World Conference on Information Systems and Technologies (WCIST’2019), La Toja Island, Galicia, Spain, 16–19 April 2019; Springer: Cham, Switzerland, 2019. [Google Scholar] [CrossRef]
Figure 1. CaMeLi dashboard.
Figure 1. CaMeLi dashboard.
Technologies 08 00042 g001
Table 1. Type of command for each task.
Table 1. Type of command for each task.
TaskType of Command
In the main screen, through touch, access the “Games” menu.Touch
Select the game “Letter soup”.Touch
Hide the avatar.Touch
Identify at least three words on the game board.Touch
View the list of available games again and select the “Rooster Game”.Touch
Consult the “News” menu using a voice command (for example: “Open news” or “Read news”).Voice
Select and read a news story of your choice.Touch
Make the Avatar visible again.Touch
Select the “Agenda” menu and then the “Meals” menu, using touch commands.Touch
Disconnect the microphone and then reconnect it again.Touch
Table 2. Results of the application of USE.
Table 2. Results of the application of USE.
USEGlobal
(n = 46)
Barriers
(n = 11)
Facilitators
(n = 35)
Mean Value (SD)Mean Value (SD)Mean Value (SD)
Total Score *5.06 (1.10)4.87 (1.00)5.65 (1.20)
Usefulness *5.14 (1.12)4.91 (1.05)5.80 (1.05)
It helps me be more effective.5.22 (1.36)5.00 (1.28)5.91 (1.40)
It helps me be more productive.5.22 (1.46)4.91 (1.40)6.18 (1.19)
It is useful.5.46 (1.13)5.31 (1.16)5.91 (0.94)
It gives me more control over the activities in my life.5.17 (1.34)5.09 (1.27)5.45 (1.51)
It makes the things I want to accomplish easier to get done.5.07 (1.37)4.83 (1.27)5.82 (1.42)
It saves me time when I use it.4.98 (1.66)4.62 (1.52)6.09 (1.68)
It meets my needs.4.85 (1.41)4.60 (1.33)5.64 (1.38)
Ease of Use4.90 (1.17)4.73 (1.08)5.39 (1.32)
It does everything I would expect it to do.5.00 (1.58)4.66 (1.57)6.09 (1.05)
It is easy to use.5.24 (1.27)5.20 (1.18)5.36 (1.50)
It is simple to use.5.17 (1.51)5.14 (1.35)5.27 (1.91)
It is user friendly.4.65 (1.65)4.63 (1.50)4.73 (2.05)
It requires the fewest steps possible to accomplish what I want to do with it.4.87 (1.29)4.76 (1.21)5.18 (1.48)
It is flexible.4.76 (1.42)4.59 (1.23)5.27 (1.76)
Using it is effortless.5.35 (1.45)5.09 (1.34)6.18 (1.51)
I can use it without written instructions.4.59 (1.71)4.40 (1.58)5.18 (1.96)
I do not notice any inconsistencies as I use it.4.67 (1.63)4.51 (1.56)5.18 (1.78)
Both occasional and regular users would like it.5.33 (1.27)5.14 (1.19)5.91 (1.31)
I can recover from mistakes quickly and easily.4.72 (1.50)4.37 (1.24)5.82 (1.78)
I can use it successfully every time.4.41 (1.67)4.40 (1.65)4.45 (1.73)
Ease of Learning5.02 (1.35)4.90 (1.30)5.36 (1.44)
I learned to use it quickly.5.07 (1.44)5.03 (1.32)5.18 (1.75)
I easily remember how to use it.4.96 (1.43)4.80 (1.37)5.45 (1.51)
It is easy to learn to use it.5.17 (1.54)5.03 (1.48)5.64 (1.62)
I quickly became skillful with it.4.87 (1.50)4.77 (1.44)5.18 (1.66)
Satisfaction *5.28 (1.26)5.04 (1.14)6.05 (1.31)
I am satisfied with it5.37 (1.31)5.11 (1.23)6.18 (1.24)
I would recommend it to a friend.5.70 (1.33)5.49 (1.31)6.36 (1.15)
It is fun to use.5.65 (1.22)5.49 (1.17)6.18 (1.19)
It works the way I want it to work.4.83 (1.58)4.60 (1.35)5.55 (1.98)
It is wonderful.5.26 (1.60)4.97 (1.52)6.18 (1.60)
I feel I need to have it.4.78 (1.70)4.51 (1.46)5.64 (2.07)
It is pleasant to use.5.39 (1.37)5.11 (1.25)6.27 (1.36)
* p < 0.05.
Table 3. Results of the application of ICF-US I.
Table 3. Results of the application of ICF-US I.
ICF-US IGlobal (n = 46)Barriers (n = 11)Facilitators (n = 35)
Mean Value
(SD)
Mean Value
(SD)
Mean Value
(SD)
Total Score15.93 (16.57)−10.64 (8.90)24.29 (6.32)
The ease of use.1.61 (1.86)−1.27 (1.35)2.51 (0.70)
The degree of satisfaction with the use.2.02 (1.51)−0.36 (1.12)2.77 (0.49)
The ease of learning.1.57 (1.94)−1.36 (1.36)2.49 (0.89)
The achievement of expected results. 1.39 (1.79)−1.45 (1.04)2.29 (0.71)
The similarity of the way it works on different tasks.1.85 (1.51)−0.27 (1.49)2.51 (0.66)
The possibility to interact using different inputs1.41 (1.94)−1.45 (1.37)2.31 (0.96)
The understanding of the messages displayed.1.52 (1.88)−1.36 (1.03)2.43 (0.92)
The application responses to your actions.1.48 (1.71)−1.18 (0.98)2.31 (0.76)
The knowledge of what was happening in the application during its use.1.41 (1.84)−1.27 (1.35)2.26 (0.95)
Globally, I consider that the application was…1.59 (1.67)−1.00 (1.10)2.40 (0.69)
Table 4. Results of the application of ICF-US II.
Table 4. Results of the application of ICF-US II.
ICF-US IIMean Value (SD)
Application Components−0.90 (0.62)
Virtual companion.−1.54 (1.13)
News Page.−0.85 (1.68)
Games Page.−0.31 (1,89)
Detailed Usability−0.7 (1.19)
Touch interaction.−1.38 (0.96)
Voice interaction.−2.46 (0.78)
Subtitles.−0.46 (1.51)
Layout change (e.g., hiding the avatar).−1.69 (1.32)
System navigation (e.g., return).−1.31 (1.25)
Icons representativeness.−1.54 (1.27)
Icons size.0.62 (1.56)
Icons color.1.00 (1.63)
Icons contrast.1.15 (1.77)
Layout (elements distribution on the screen).0.31 (1.89)
Audio−1.15 (1.34)
Application functioning mode−1.46 (1.05)
Overall assessment −1.23 (0.93)

Share and Cite

MDPI and ACS Style

Jegundo, A.L.; Dantas, C.; Quintas, J.; Dutra, J.; Almeida, A.L.; Caravau, H.; Rosa, A.F.; Martins, A.I.; Pacheco Rocha, N. Perceived Usefulness, Satisfaction, Ease of Use and Potential of a Virtual Companion to Support the Care Provision for Older Adults. Technologies 2020, 8, 42. https://0-doi-org.brum.beds.ac.uk/10.3390/technologies8030042

AMA Style

Jegundo AL, Dantas C, Quintas J, Dutra J, Almeida AL, Caravau H, Rosa AF, Martins AI, Pacheco Rocha N. Perceived Usefulness, Satisfaction, Ease of Use and Potential of a Virtual Companion to Support the Care Provision for Older Adults. Technologies. 2020; 8(3):42. https://0-doi-org.brum.beds.ac.uk/10.3390/technologies8030042

Chicago/Turabian Style

Jegundo, Ana Luísa, Carina Dantas, João Quintas, João Dutra, Ana Leonor Almeida, Hilma Caravau, Ana Filipa Rosa, Ana Isabel Martins, and Nelson Pacheco Rocha. 2020. "Perceived Usefulness, Satisfaction, Ease of Use and Potential of a Virtual Companion to Support the Care Provision for Older Adults" Technologies 8, no. 3: 42. https://0-doi-org.brum.beds.ac.uk/10.3390/technologies8030042

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop