A Cloud-Based WEB Platform for Fall Risk Assessment Using a Therapist-Centered User Interface Which Enables Patients’ Tracking Remotely

Abstract

This work describes a system to help in the remote assessment of fall risk in elderly people. A portable hardware system equipped with an RGB-D sensor is used for motion capture. A set of anonymous frames, representing the process of skeleton tracking, and a collection of sequences of interesting features, obtained from body landmark evaluations through time, are stored in the Cloud for each patient. A WEB dashboard allows for tailored tests to be designed, which include the typical items within well-known fall risk evaluation tests in the literature. Such a dashboard helps therapists to evaluate each item from the analysis and observation of the sequences and the 3D representation of the body through time, and to compare the results of tests carried out in different moments, checking on the evolution of the fall risk. The software architecture that implements the system allows the information to be stored in a safe manner and preserves patients’ privacy. The paper shows the obtained results after testing an early prototype of the system, a discussion about its advantages, and the current limitations from the Human–Computer Interaction point of view, and a plan to deploy and evaluate the system from the usability perspective in the near future.

1. Introduction

The world population is progressively aging as a result of increasing life expectancy. According to the WHO’s (World Health Organization) projections [1], the number of people over 60 years of age will have doubled by 2050 and, in the case of those over 80, it will have tripled.

From the health systems’ perspective, this fact represents a great challenge both economically and socially; however, administrations could face it by implementing effective protocols of digital transformation in healthcare.

Many countries are committed to the process of health digitalization and are mainly focused on improving information management, such as medical appointments and reports, or digital prescriptions, among others [2]. However, once the COVID-19 pandemic ended, the relationship between patients and medical staff became presential again in most cases. This fact does not have to be a negative because human relations are necessary in the context of healthcare [3,4], but waiting lists and health system saturation make promoting a deep doctor–patient relation difficult. Such situations are even worse when the patients are elderly people [5]. Many senior citizens are dependent and, consequently, they need to receive special attention from informal and formal caregivers, both at home and in elderly institutions [6]. Moreover, many elderly people are not familiar with technology, and this is more problematic when their mental and/or physical condition is impaired [7].

On the other hand, multimorbidity and frailty are common in the elderly and these conditions negatively affect the quality of life (QoL). Frailty is often related to a diminished physical capacity which usually increases the fall risk [8].

Fall risk is an especially dangerous situation for the elderly. For example, in the literature, it is possible to find research works that relate the fear of falling with a low HRQoL (Health Related Quality of Life) rate, both physically and mentally [9]. In addition, the most frequent consequence of falls is usually a hip fracture, which is associated with a decrease in functional capacity and a high risk of subsequent death, as indicated by different studies [10,11,12]. The impact of falls on hospital pressure is also high, representing a high economic cost, not only due to hospitalization but also due to the consequent dependence generated in people [13]. Many elderly people have to be moved to elderly residences after suffering a fall that produces serious injuries. For people living in elderly homes, falls represent an increment in the degree of dependence [14]. Therefore, it is essential to properly assess fall risk because its consequences are very negative for patients, relatives, caregivers, health professionals and, in general, healthcare and social systems.

Therapists use different tests and protocols to evaluate an individual’s likelihood of falling. Due to the importance of preventing falls, an initiative for providing a coordinated approach for fall prevention has been designed by the CDC (Center for Disease Control and Prevention)—CDC’s STEADI (Stopping Elderly Accidents, Deaths, and Injures) [15]. Such initiative includes the following: (i) Educational materials for providers, patients, and caregivers; (ii) screening tools; (iii) standardized gait and balance assessment tests; (iv) information about those medications that increase the fall risk; and (v) clinical decision support for electronic health record systems, among others.

Specifically, the CDC has implemented the “STEADI algorithm for Fall Risk Screening, Assessment, and Intervention among Community-Dwelling Adults 65 years and older” [16], which is defined as a normalized protocol for therapists wanting to evaluate the fall risk through time. The following steps are considered:

• “SCREEN for fall risk yearly, or any time patient presents with an acute fall”; therapists apply the available risk screening tools as follows:

  ○

  “Stay Independent”, a 12-question tool for calculating the risk according to a score, which if it is equal or greater than four, the individual is at risk. If it is lower than four, but the patient fell in the past year, he/she is also at risk.

  ○

  “Three key questions for patients”, which obtains information about the patients’ feelings and worries regarding the perceived fall risk.

• If screened not at risk, future risk should be prevented by recommending “effective prevention strategies”. But if screened at risk, it is necessary to “assess patient’s modifiable risk factors and fall history”.
• Intervention for reducing those risk factors which have been identified using suitable strategies.

In Step 2, if screened at risk, the following common ways to assess fall risk factors are considered: (i) Gait, strength, and balance evaluation through common assessments, such as well-known balance tests (Time Up and Go, or 30-Second Chair Stand, among others); (ii) identification of medications that increase fall risk (such as Beers Criteria); (iii) asking about potential home hazards; (iv) measuring orthostatic blood pressure in lying and standing positions; (v) checking visual acuity; (vi) assessing feet/footwear; (vii) assessing vitamin D intake; and (viii) comorbidity identification.

Regarding tests for fall risk assessment that are well documented in the standard of care, the most used methods are the Tinetti test [17], the Berg Balance Scale (BBS) [18], the Dynamic Gait Index (DGI) [19], the test Time Up and Go (TUG) [20], the four-square test (FST) [21], the functional reach test (FRT) [22], and the single leg stance (SLS) test [23]. These methods are focused on the observation of the patient’s pose while he/she carries out specific short exercises, such as walking some meters to a specific goal (gait evaluation), getting up from a chair, or staying balanced without support (balance evaluation), among others.

The Tinetti test is the oldest clinical balance assessment tool and the widest used among older people [24] with a good inter-rater reliability (85% agreement between raters) and excellent sensitivity (93% of fallers are identified) [25,26]. The BBS (also developed for elderly people) has an excellent inter-rater reliability but with a sensitivity from poor to moderate [27,28] and it could be performed in only 10 or 15 min. The TUG test “is the shortest, simplest clinical balance test, and probably the most reliable because it uses agreement in stopwatch durations rather than rating scales” [29].

The application of such tests involves the presence of the therapist and the patient in the same space. In fact, the therapist subjectively assigns a score to each item after visually observing the exercise. The scores are manually annotated, and a very reduced number of digital tools are available for helping such information to be calculated or stored (mainly, smartphone applications or WEB forms).

There are certain types of academic research aimed at helping in fall detection or fall risk estimation using different kinds of sensors. Most of them have chosen to integrate IMUs (Inertial Measurement Units) to obtain signals and associate them with the score of one of the previously described tests [30,31,32,33,34]. However, it is very difficult to find a completely digital solution which allows therapists to carry out an integral fall risk assessment through time (that is, which enables fall risk evolution to be properly measured) by acquiring, processing, and showing the information in a usable way, from the Human–Computer Interaction (HCI) point of view.

After analyzing the lack of digital platforms to automate the fall risk assessment process, this work’s authors worked [35] on the design and development of a system “capable of enabling therapists to monitor and assess motion and gait difficulty, balance, and posture maintenance, even remotely, for preventing certain events related to falls in elderly people”. Such a system “consists of a hardware device equipped with an RGB-D sensor that allows the patients’ body pose landmarks to be captured, from which a wide number of useful parameters are calculated, which could be stored both locally and in a Cloud-based platform. Such parameters are closely related to the items used by many well-known clinical tests of gait already described in the standard of care”.

In [35], two limitations were detected as follows: (i) It is necessary to design a software architecture, preferably in the Cloud, for storing, processing, and presenting all the acquired data to help therapists track the evolution of each patient in an effective and usable way; and (ii) the system should be validated by a wide range of patients and therapists for testing its effectiveness from a significant statistical point of view.

In this paper, a Cloud-based platform for overcoming the first mentioned limitation is presented. The selected approach to develop such a platform is based on the application of the user-centered design (UCD) [36], which could be defined as an iterative design process in which designers focus on users and their needs in each step.

The outline of the rest of the paper is as follows: Section 2 describes the materials and methods used for developing the proposed digital platform for fall risk assessment in the Cloud. First, the iterative process based on UCD is presented, then the components of the implemented distributed software architecture are detailed. The portable device designed for facilitating the remote application of balance and gait tests is also described. Section 3 shows the tests carried out in the laboratory context. Section 4 shows the results, which are then properly analyzed to detect the limitations of the system and propose future solutions. Finally, Section 5 presents the main conclusions of the work.

2. Materials and Methods

In [35], the system that helps therapists to remotely evaluate the fall risk, by applying tests defined in the standard of care, is described in detail. In summary, such a system is briefly outlined next.

The hardware components are a mini personal computer (or mini-PC), a touchscreen, and an RGB-D sensor. This device is properly located in front of the patient whose motion and posture should be evaluated.

Figure 1 summarizes how therapists select tests consisting of items that should be executed by patients during fall risk assessment trials. Each item involves measuring one or more parameters (for example, “Trunk tilt”, “Trunk swing”, or others). Such parameters are calculated from the data obtained after extracting certain significant landmarks (for instance, “Right shoulder”, “Left elbow”, or others) using the RGB and depth images as inputs. The set of software modules that allow those parameters to be computed are integrated as nodes in the ROS (Robotic Operation System) framework. Such modules calculate distances and angles (for example, the angle of inclination of the shoulder allows “Trunk swing” to be estimated). For acquiring frames (RGB and depth ones), from the RGB-D sensor, the ROS package provided by the manufacturer of the sensor is used, exposing each synchronized and registered pair of images in two ROS topics. A client node is subscribed to the mentioned topics to obtain both images. The RGB image is then sent as a parameter of a request made to a service node, which applies the MediaPipe Pose solution for obtaining 33 landmarks of the human body representative of their corresponding human body joints [37]. The landmarks from 11 to 28 are relevant for parameters’ calculation. Since such points are not referenced to any three-dimensional coordinate system, each point is transformed to the RGB-D sensor’s reference system. From the 3D position of the joints calculated with respect to such reference system, multiple measurements (also named parameters), which will be used to evaluate the different items of the gait and balance tests, are computed.

We note that developing a system for remotely assessing fall risk tests could be based on recording a video for each item, to store it, and to reproduce it for a therapist’s remote evaluation. However, this solution could give rise to privacy issues, and multimedia storage requires a great capacity of storing and “uploading” and “streaming” in the Cloud, which is highly time and energy consuming. Additionally, scoring is completely subjective and hardly comparable between different trials. Therefore, the system implemented in [35] would overcome the previously mentioned disadvantages, since evaluating a specific item could be carried out by observing the temporal evolution of the parameters associated to such item. For example, the fifth item in the Tinetti test (standing balance), could be assessed through the sequences of “trunk swing”, “separation of the heels”, and “separation of the arms”. Such data are actually represented as a time-series in a 2D graphic, where the X-axis is “time” and the Y-axis is “angle”, “distance” or another similar factor, depending on the measurement type. On the other hand, it is possible to also store the set of landmarks calculated through time by allowing therapists to watch a reconstruction of the patient’s motions, but only centered in the representative links and joints of the human body. For health professionals, this kind of representation (time-series) is familiar. For example, electrocardiograms (ECGs), electroencephalograms (EEGs), or other similar measurements are very common.

As aforementioned, a limitation of the work carried out in [35] is that a Cloud-based platform for integral fall risk assessment is missing. Such a limitation is overcome in the system presented in this paper.

This section explains the procedure followed for designing and implementing such a Cloud-based platform, which uses the UCD as main design strategy. Next, the subsequent subsections detail the following: (i) how the requirements of the system have been captured by putting users in the center of the design process; (ii) how the information is selected and processed by the different components of the distributed software architecture, and then stored in the Cloud according to a data model represented by an “entity-relation” diagram; (iii) how the user interfaces (UIs) have been designed using UI mockups; and (iv) how the backend and the frontends have been implemented according to such mockups.

2.1. Application of the UCD Lifecycle for Requirements and User Detection and Software Design

The UCD lifecycle consists of several stages where the user’s needs are in the center [38]. Therefore, it is necessary to appropriately define who the users are. In this case, the main direct users are therapists and caregivers (both formal and informal, including relatives), and indirectly, elderly people (patients). The context of use of the system is different for each kind of user as follows:

• Elderly people do not appreciate the differences between doing the tests in a traditional way or with the new system, since they should be guided by someone (for example, a caregiver or a relative) in both cases. The main advantage of using this remote system is that it is not necessary for them to go to a hospital or specialized center. Furthermore, they do not need to wear any specific sensor. Consequently, the solution is comfortable and accessible.
• Caregivers should only use the hardware device and a suitable user interface for applying the tests designed by therapists.
• Therapists should use the Cloud-based platform for the following: (i) designing the tests and prescribing them to each patient; (ii) observing the results of the application of each test for each patient at a given time; and (iii) comparing the evolution of fall risk between different instances of a test applied in different moments.

Figure 2 summarizes the process of applying the UCD lifecycle. Once the users and their contexts are defined in the first stage, it is necessary to define all the requirements that the system should meet in the second stage. The solution is designed and developed in the third stage as follows: (i) the data model is defined; (ii) the Cloud-based infrastructure is selected, together with the model used for software development—Platform as a Service (PaaS) in this case; (iii) the Cloud provider is chosen—Google Cloud Tools (GCT) are used in this case; (iv) the backend services are developed; and finally, (v) the frontends (UIs) are designed and implemented. The fourth stage allows the system to be deployed in real settings with real users.

The released system could be validated through usability tests. Moreover, the whole process is iterative and the stages are usually connected with the previous ones because feedback is essential in this paradigm.

As the system has not been deployed in a real elderly institution yet, only three of the UCD’s steps have been carried out. The “solution’s evaluation” stage will be accomplished after real deployments, which are programmed during the second semester of 2024.

However, informal contact with the institutions’ workers were carried out for the purpose of improving specification of the requirements. In particular, the following clinical specialists have participated: two psychologists, one occupational therapist, two physiotherapists, and the main persons responsible for two elderly homes. During the “Analysis of user’s needs and context” stage, informal interviews were carried out to acquire the following information: schedule of activities in the residences, frequency of application of fall risk tests and their evaluations, classification of residents according to illness and comorbidities, type of medical reports obtained in the institutions, and kind of digital tools for storing and processing the medical information. Moreover, stakeholders were asked if they are applying the STEADI algorithm, what kind of tests are often used, and which items of such tests are frequently included. Once all this information was obtained and all the steps of the STEADI algorithm were analyzed in detail, the information to be saved and processed was selected, and then several early prototypes of the system (using storyboards) were presented to the stakeholders. A set of new interviews were carried out to evaluate the prototypes and slight changes were incorporated into the design. Finally, with all this acquired information, the “designing solution” stage started.

2.2. Portable Hardware Device and Software Architecture in the Cloud

The mini-PC, the touchscreen, and the RGB-D sensor have been encapsulated in a portable device with an ergonomic design to help users apply the prescribed tests (see Figure 3). The RGB-D sensor is located at the top of the device at a 22.75 cm height from the base of the device, which weighs 2.7 kg, similar to any laptop.

Figure 4 shows the outline of the designed software architecture. It integrates the ROS-based components developed in [35] and adds all the software modules included in the backend and in the frontends, namely one for presenting a Graphical User Interface (GUI), adapted to the portable device’s touchscreen, which allows caregivers/relatives to apply a prescribed test to a specific patient; and another one for enabling therapists to design such tests and to evaluate the results through a WEB application.

2.2.1. Data Model

The STEADI algorithm details which information therapists should collect as follows:

• Personal information for patients’ identification; answers to the “Stay Independent” 12-question tool and the corresponding scores, and the date when the test was conducted on; and a report with information about the answers that patients gave to “Three key questions”—(i) “Feels unsteady when standing or walking?”; (ii) “Worries about falling?”; and (iii) “Has fallen in past year?”
• Information about medication that increases the fall risk; relevant information about medical parameters, such as orthostatic blood pressure, visual acuity, or detected comorbidities; tests defined in the standard of care for assessing fall risk (they consist of items whose evaluation could be made from the parameters described in Figure 1); and landmarks recorded with the MediaPipe Pose solution.

The proposed system allows all these data to be stored on the Google Cloud using Firestore in Datastore mode (a NoSQL document database with automatic scaling, ease of application development, and high performance [39,40]). Implementation is performed using the Java API (Application Programing Interface) Objectify for inserting and retrieving information, specifically designed for the Firestore in Cloud Datastore mode. This API allows the creation of entities with one or more named properties, each of which can have one or more values. Each entity has a kind, which categorizes it for the purpose of queries. Compared to a classical SQL relational database, the concept of kind is linked to the concept of a table, an entity is similar to a table’s row and a property is similar to each column of a table. Therefore, by considering all the previously mentioned issues, the designed data model used in this work is presented as an E–R model in Figure 5, whose entities and relationships are detailed next.

The entities in the E–R model are not all transformed into Datastore entities since such a diagram is a simplified version, which could be extended during the deployment of the system in different real settings in the near future. Currently, the grey-marked part of the diagram is directly transformed into Datastore entities, together with the entities “Elderly patient”, “Therapist”, “Caregiver”, and “Interview”. The patient’s medical history is modelled as a different entity, “History”, splitting the patients’ data into the following two groups of information to help the developers ensure data privacy: “Elderly patient” with attributes for the univocal identification of each individual in the database and the profile with summarized anonymized information about certain interesting features (biological sex, age, or comorbidities); and “History”, which includes the personal information of the individual which should be protected, and it will only be accessible to accredited health professionals. The entity “Interview” allows the questionaries used in the STEADI algorithm to be stored, together with the answers given by patients in a specific time during presential consultations.

The set of entities in the grey-marked part of the diagram enables the storage of the information related to the design and application of tests for fall risk assessment well-known in the standard of care. The description of each entity is detailed as follows:

• “Type of Test”: It stores information about a specific test designed by a therapist.
• “Type of Item”: It stores information about an item included in a test, which could be created by a therapist.
• “Type of Parameter”: Each “Type of Item” includes a set of parameters automatically calculated by the system using the set of landmarks provided by the MediaPipe Pose solution. The available types of parameters are those defined in Figure 1.
• “Test”: The difference between the entities “Test” and “Type of Test” is that while a “Type of Test” describes the characteristics of a test defined in the standard of care, a “Test” is an instance of a specific “Type of Test” executed by the patient in a given moment.
• “Item”: As it occurs with “Type of Test” and “Test”, the relationship between “Type of Item” and “Item” is similar. The “Item” could be considered as a specific instance of “Type of Item” executed by the patient in a given moment.
• “Parameter”: The relationship between “Type of Parameter” and “Parameter” represents that the latter is an instance of the former. When a patient executes a specific item, a set of specific parameters should be calculated during an interval of time, and they should be stored as sequences of values through time. However, it is not possible to store any array of values in the Datastore because there is a limitation related to the length of such an array. Thus, each specific entity “Parameter”, generated as a consequence of executing a specific item, will store a list of identifiers, where each identifier will identify a specific package of measurements implemented by the entity “Package”.
• “Package”: One package of measurements (parameters) is implemented as a “Package” entity. There exists a relationship between “Package” and “Measurement” because each package consists of a bounded number of measurements with an order number “N”.
• “Measurement”: This entity represents an individual parameter measured in a specific time “ts”.
• “Skeleton samples”: For each “Item”, a sequence of timestamped set of landmarks, calculated by using the Mediapipe Pose solution, should also be stored. As with the measurements, it is necessary to split the collection of frames, which represent the landmarks associated to the skeleton in a given time, into packages.
• “Skeleton Package”: Each entity of this type is ordered by a number “N”. There exists a relationship between the “Skeleton Package” and “Skeleton Sample” because each package consists of a bounded number of samples.
• “Skeleton Sample”: This entity represents an individual frame consisting of the landmarks calculated by the MediPipe Pose solution at a specific time “ts”.
• “3D MediaPipe Landmarks”: It is included as an attribute of the “Skeleton Sample”, and it stores the 33 landmarks provided by the MediaPipe Pose solution at a given time.

For a better understanding of how the Datastore stores the set of entities represented in the grey-marked area of the E–R model in Figure 5, an example is shown in Figure 6.

In this case, a therapist has created a simplified variation of the Tinetti test (type of test), which consists of the following three items (type of item) to be evaluated: “Sitting Balance”, where the parameter Trunk Tilt (TT) is relevant; “Standing Balance”, where the parameters (type of parameter) Trunk Swing (TS), Separation of the Heels (SH), and Separation of the Arms (SA) are relevant; and “Eyes Closed”, where TT, TS, and SA are also relevant, because it is the same task as “Standing Balance” but with the eyes closed. Thereafter, an instance of the Tinetti test is prescribed to a specific patient. The caregiver selects it in the portable device and the patient executes the exercises or items remotely. The information captured by the RGB-D sensor is properly processed and a sequence of skeleton samples consisting of 3D MediaPipe landmarks is stored, together with several time-series with information about TT, TS, SH, and SA through time obtained after processing the mentioned sequence of landmarks. As the starting date and time for the test is 4 April 2024 at 10:00, the test is timestamped with such data. The first executed item is “Sitting Balance”; thus, its timestamp is also 4 April 2024 at 10:00. The array of measurements and skeleton samples contains elements with different timestamps, generating the mentioned time-series, which can be graphically represented.

2.2.2. Google App Engine and Google Cloud Endpoints

As Figure 4 shows, both the application in the portable device and the WEB application have to share information with the Google Cloud. For this reason, the Google App Engine (GAE) has been selected for the backend implementation, since it enables monolithic server-side rendered websites to be built, freeing up the developers of the server management and configuration deployments. In addition, GAE helps the Cloud-based application to be safeguarded by defining access rules with an App Engine firewall and leveraging managed SSL/TLS certificates by default on a custom domain at no additional cost.

On the other hand, the access to the GAE application is carried out using calls to an API REST (Representational State Transfer) under HTTPS (Hypertext Transfer Protocol Secure). The application has been developed in such a way that calls are only accepted if users are registered in the system (such information is also stored in the Datastore) and they are previously logged in with a username and a password. Notably, all the information saved in the Datastore is encrypted and the sensitive personal information of elderly users (patients) could be stored in other private servers, if needed, to protect their privacy. Thus, information anonymization on the Google side is ensured.

For implementing the services, which are called from the frontends, Google Cloud Endpoints are used. This component allows a custom API to be implemented, secured, monitored, analyzed, and managed using the same infrastructure Google uses for its own APIs. In particular, the Cloud Endpoints Frameworks for the App Engine standard has been selected, since it provides the tools and libraries that allow REST APIs and client libraries for an application to be generated. Endpoints Frameworks handle the low-level communication details of HTTP requests and responses for the application. When a client sends a request to the developed API, Endpoints Frameworks route the request’s URL (Uniform Resource Locator) to the function or method in the API code that processes the request. Endpoints Frameworks then convert the return value to JSON (JavaScript Object Notation) and it sends the response.

Java has been selected for implementing all the code in the backend, and the suitable metadata (Java annotations) have been added. JSON–Java and Java–JSON conversions are automatically conducted, making codification easier. The combination of GAE and Google Cloud Endpoints facilitates the deployment of the application in the standard App Engine environment.

2.3. User Interface Design

According to UCD, users should be taken into account during the process of the user interface’s design. As explained in Section 2.1, potential users have been informally interviewed in two nursing homes, and the STEADI algorithm has been carefully analyzed to capture the requirements that should be considered for designing the frontends.

As mentioned before, patients (elderly people) are the indirect system users who only follow the caregivers’ orders and/or recommendations when they are executing the exercises (items) included in a specific test. Caregivers (formal and informal ones) and relatives could be users with a very low or even no training in technology and usage of digital tools. However, it is expected that therapists have a minimum training in the use of health digital tools (for example, WEB applications commonly used for data management in healthcare environments).

Considering all these issues, two GUIs have been designed, one for the portable device (as a part of a Qt-based application that runs on such a device), and the other one for the WEB application. The following common aspects have been considered in designing both of them:

• The “Material Design” metaphor is the preferred option. Material is a design system created by Google, widely used in Android-based smartphones. Most people are familiar with the use of smartphones; therefore, using Material Design is a good option for ensuring usability and reducing the learning curve of the applications.
• Partial dark mode is the preferred display setting, since it is known for its ability to reduce eye strain. The selected themes (combination of shapes and background and foreground colors, among others) were configured after considering that some individuals could also suffer mild visual impairments. Thus, the traditional dark mode was modified to include some colors and enough contrast. Notably, certain informal caregivers or relatives that care elderly people at home could be elderly persons too. On the other hand, using GUIs in dark mode could help to reduce the digital emissions, contributing to further alignment with the Green Computing paradigm. Devices with OLED (Organic Light-Emitting Diode) and LED (Light-Emitting Diode) screens can save energy using the dark mode because they turn off individual pixels to show darker colors. That means that using dark mode on these kinds of screens consumes less energy compared to light mode, since less pixels need to be lit up.

UI mockups (static high-fidelity visual representations of the final interface’s prototypes without functionality) were designed before the final implementation of the applications. This facilitated early corrections of the user interfaces where necessary.

2.3.1. Description of the UI Designed for the Application Running in the Portable Device

As aforementioned, the Qt-based application (see Figure 7) executed in the portable device allows caregivers/relatives to download the tests prescribed for a specific patient. The access to the system is protected by using a login ID and password (see Figure 7a). Therefore, caregivers/relatives should be previously registered in the system through the WEB application. At this point, therapists would be responsible for adding the patients and their caregivers. After authentication, a list of patients could be shown since a caregiver could be caring for several individuals (see Figure 7b). Once an individual is chosen, a test could be selected from a list (see Figure 7c), and the items of the selected test will be shown (see Figure 7d). For each item, a description with instructions on how to carry out the item will be presented (see Figure 7e). Once the patient is correctly located in front of the RGB-D sensor, the application of the item starts. The sensor captures both RGB and depth images and they are used as inputs of the MediaPipe Pose solution that provides the landmarks from which parameters are calculated through time. Caregivers stop the recording process, and the results are shown on the screen (see Figure 7f). If the item is not correctly executed, the process could be repeated as many times as needed. Finally, when the test is completed, it is possible to synchronize the results in the Cloud by uploading the registered information.

The UI that helps caregivers/relatives to apply the tests is based on the use of a touchscreen; consequently, the components of the UI and the interaction mode is similar to using a common smartphone/tablet application.

2.3.2. Description of the UI Designed for the WEB Application

The WEB application’s GUI has been designed considering therapists as potential users because they are responsible for the patients’ data management and the registration of the caregivers who are in charge of helping the elderly users apply the tests. The WEB application is hosted as a part of the GAE application, and it could be accessed through typical authentication based upon the input of the corresponding username and password.

Once the authentication process is passed, therapists work with a complete dashboard (see Figure 8), which enables them to manage all the relevant information as follows: patients’ personal data (see Figure 8a), medical history, application of questionnaires according to the STEADI algorithm (see Figure 8b), tests’ design and prescription (see Figure 8c), and evaluation of results for each particular patient (see Figure 8d).

3. Results

The system has been tested in a laboratory context. Several trials have also been carried out with real patients in a nursing home for capturing motion, as fully described in [35], but the final applications have not been deployed there yet. However, the information acquired in these real tests are suitable for uploading such information to the Cloud and they help the WEB application GUI to be evaluated.

This section describes one of the trials carried out in the laboratory to test the functionality of the developed distributed software architecture, with the aim of validating all the software components, both in the backend and in the frontends.

In particular, the Tinetti test version shown in Figure 1 and Figure 6 was designed using the WEB application described in Section 2.3.1.

First, the authors uploaded information about four “fake” patients, whose information had been stored in Google Cloud, using the services provided by the developed Endpoint API, according to the data model which Figure 5 and Figure 6 illustrate. Therapists are capable of designing the specific items (activities) that patients should remotely carry out. Such items are grouped by specific types of tests. First, each activity is added, and it is described with a name and a description. Then, it is necessary to add which parameters (measurements) will be used for assessment. Such parameters are only selectable by the therapist, and they were previously uploaded in the Cloud when the system was deployed for the first time in this laboratory trial.

In this case, the item “Sitting Balance”, present in Tinetti tests, was added since it enables therapist to evaluate how the patient stays seated in a chair from the balance point of view. If the patient leans or slides in the chair (this could be observed through the Trunk Tilt value), the score should be 0, and if the patient is steady and safe, the score should be 1. As the Trunk Tilt observation is enough, only such a parameter could be selected from the list, while the items “Standing Balance” and “Eyes Closed” were also added (see Figure 9).

Once the required activities are stored, it is possible to create a specific test (type of test) which includes them. When the test is carried out using the portable device remotely, the results are uploaded to the Cloud, and they are automatically available for the therapist, who is able to evaluate them as Figure 10 shows. The layout is split into several areas as follows: one with the information of the patient, an area showing the time-series for each parameter which is measured for a specific item (activity), and the representation of the 3D landmarks calculated using the MediaPipe Pose solution. Such representation is made according to the RGB-D system reference, and the therapist could then reproduce the patient’s motions, as many times as necessary.

In the example shown in Figure 10, the selected parameter is “Separation of the Arms”. In this case, the X-axis represents time, and the Y-axis represents the angles corresponding to “Right arm” and “Left arm”. If the position of both arms is similar and the value of the angle is low, it means that the patient could be standing in balance without using the arms. It would be necessary to observe the “Trunk Swing” and the “Separation of the Heels” to complete the information needed to give a score in this item. The corresponding control “Select the parameter” allows therapists to change the parameter to be observed.

The current version of the system only facilitates therapists to write scores manually. Such scores could be compared using the layout shown in Figure 11. Thus, the process of carrying out the fall risk assessment through time is facilitated, since the evolution of the score could give an idea about if a therapy is working or if it is necessary to apply another kind of intervention.

Furthermore, the WEB application also helps therapists register answers to questionnaires proposed by the STEADI algorithm (see Figure 12).

4. Discussion

All the software components and GUIs have been tested in a laboratory context. Several tests with different items have been created and executed using the portable device, and the results have been suitably uploaded to the Cloud.

Furthermore, in [35], the authors proved that the system is capable of capturing human motion for real elderly users (six real patients selected by clinical specialists), demonstrating that the sequence of parameter values through time are representative, and the time-series allow therapists to identify normal and pathological motions.

This section summarizes the benefits and limitations of the presented system and shows the future work planned to help overcome such limitations.

4.1. Benefits of the Proposed System

From the usability point of view, both UIs satisfied the captured requirements. The selected colors and shapes are adapted to the potential users, considering that dark mode as preferred, allowing the system to be lower in energy consumption.

Concerning the sustainability of the system, as Google Cloud is used, it is possible to select datacenters to host the GAE application with low CO₂ emissions. Google offers different regions for deploying the application. The metrics CFE% (average percentage of time an application is running on carbon-free energy) and grid carbon intensity (gCO₂eq/kWh, average operational gross emissions per unit of energy from the grid) are used for distinguishing regions according to CO₂ emissions. In this work, as the system is being deployed in Spain, the “europe-west6” region (located in Zurich) has been selected, with 85% and 118 in CFE% and gCO₂eq/kWh, respectively.

Regarding the interpretation of the information captured and processed by the portable device and then uploaded to the Cloud, the results demonstrate that therapists have several tools to assess each item (activity) of a test. It is possible to analyze the time-series or to reproduce the recording of a set of 3D landmarks calculated by the MediaPipe Pose solution. Moreover, in [35], the authors demonstrated that the MediaPipe solution is capable of identifying the human joints with high precision, since it has been tested by the MediaPipe’s developers using a Pose Validation dataset, yielding a PDJ (Average Percentage of Detected Joints) of 97.5%, which is a strong indicator of precise matches between predicted keypoints and ground truth keypoints.

Concerning how the system helps therapists to identify normal and pathological human motion, Figure 13 shows some demonstrative time-series taken from the previous experiments carried out in one of the mentioned nursing homes [35]. In particular, in such trials, six elderly users were selected by the clinical specialists, and they carried out several activities related to balance and gait, namely sitting and standing up of a chair, staying balanced with their eyes open and closed, and walking straight for 3 m. In this paper, it is interesting to analyze if the time-series themselves help therapists identify abnormal behaviors.

Figure 13a,b show how Patient 1 stays balanced, but slightly swings their trunk, while Patient 3 does not show such swinging. The separation of the heels for Patient 2 (see Figure 13c) is also slightly higher than for Patient 4 (see Figure 13d) when they stay balanced, but both of them maintain such stable separation, indicating that they are not moving the feet. Finally, in the experiment for evaluating issues related to gait, Figure 13e,f show how Patient 2 and Patient 4 move their feet and hands because the stride and the separation of the arms are measured. Patient 2 clearly moves their feet and hands in a correct manner while walking, while Patient 4 maintains their hands close to the body. This could indicate that there is a problem with the coordination between hands and feet when walking straight.

These results demonstrate that the proposed system is a wide extension of the work presented in [35], and the limitations found in such work, related to the presentation of data to users, have been successfully overcome by applying the UCD paradigm.

Now, it is possible for therapists to create different tests tailored for each patient, and for caregivers and relatives to help the elderly users easily carry out such tests using the portable device. The solution accomplishes the objectives related to usability and ergonomics, and it is as inclusive as possible, while also considering issues related to sustainability and low energy consumption.

It helps a better fall risk assessment to be carried out because elderly patients could repeat the tests remotely, and the results are synchronized in the Cloud. Thus, such results are available for the therapist, who can make decisions about additional tests, interviews, consultations, or prescriptions, if necessary. Moreover, the WEB application allows therapists to apply all the stages of the STEADI algorithm using only one digital tool, and this is a novel contribution to the field of study.

4.2. Limitations

However, this work also has some limitations. The most important one is related to the lack of application of usability tests, since the final system has not yet been deployed in a real setting for a long period of time. Furthermore, the following points should be considered:

• Validation and testing: Conducting further validation studies with a larger and more diverse sample of patients and therapists to ensure the effectiveness and reliability of the system is essential.
• Usability and user experience: Although laboratory tests demonstrate that the system is usable and accessible (both the WEB application and the portable device), it is necessary to compile information about the final users’ perception. A study about the relationship between digital knowledge and usability in caregivers is crucial to validate if the portable device and its GUI are suitable even for people with low technological background. On the other hand, it is necessary to obtain therapists’ feedback about the WEB application usage, considering the possibility of using different computational devices (PCs or tablets, for example).
• Data presentation: It is also essential to analyze if 3D representation of the skeleton and time-series shown in the WEB application are clear and usable from a clinical point of view, and if this data representation helps therapists to better evaluate the fall risk through time and properly make decisions about treatments and rehabilitation therapies.
• Security and privacy: The system has been designed by splitting data into anonymized information and personal information. The latter can only be accessed by authorized therapists and stakeholders. However, it is necessary to analyze the behavior of the system in real environments to ensure robust security measures are in place to protect the patient data stored in the Cloud and maintain patient privacy throughout the assessment process.
• Integration and compatibility: The current system stores information according to the designed data model using Google Cloud Tools. However, it is necessary to ensure seamless integration of the system with existing healthcare technologies and workflows to facilitate its adoption and implementation in clinical settings.

4.3. Future Work

The main future work would be deploying the system in the two nursing homes previously mentioned, applying the system over several months with at least 20 elderly users (to obtain significant statistical results), observing how the remote application of the STEADI algorithm could improve fall risk assessment, and carrying out a full usability analysis using questionnaires with each kind of user to objectively measure the acceptation of the system in terms of usability, accessibility, and ergonomics.

Furthermore, the authors are currently working closely with therapists to evaluate if the 3D representation of the skeleton and time-series shown in the WEB application are clear and usable from a clinical point of view.

Finally, the integration of the data model with a standard representation of the information in clinical contexts is also being addressed, by taking into account the feasibility of importing/exporting FHIR (Fast Healthcare Interoperability Resources) data using the Google Cloud Healthcare API.

5. Conclusions

The described system is a novel proposal in the field of fall risk assessment, since therapists could apply the STEADI algorithm and track the evolution of patients remotely. It consists of a portable device that enables caregivers and relatives to help elderly people to apply tests well known in the standard of care for fall risk assessment. The results are easily shared on the Cloud with therapists, who are in charge of designing, prescribing, and evaluating such tests. They carry out these tasks through a WEB application which calls the services hosted in Google Cloud by deploying a GAE application that uses Google Cloud Endpoints. All the information is properly saved in the Firestore on Datastore mode, which stores the information in an encrypted form. The access to the software architecture, both through the Qt-based application running in the portable device and through the WEB application, is made using typical authentication based on the usage of a username and password.

Therapists are in charge of managing the information related to their patients and only they can access the medical history.

The system has been validated in a laboratory, and only the last stage of the UCD lifecycle is missing. However, final deployment of the system in two nursing homes and its corresponding evaluation with real users from the usability point of view is already planned for the first half of 2024.