Multimodal Technol. Interact., Volume 6, Issue 1 (January 2022) – 7 articles

Ideally, language and reading skills in bilingual children are assessed in both languages spoken in order to avoid misdiagnoses of communication or learning disorders. Due to limited capacity of clinical and educational staff, computerized screenings that allow for automatic evaluation of the children’s performance on reading tasks (accuracy and speed) might pose a useful alternative in clinical and school settings. In this study, a novel web-based screening platform for language and reading assessment is presented. This tool has been preliminarily validated with monolingual Italian, Mandarin–Italian and English–Italian speaking primary school children living and schooled in Italy. Their performances in the screening tasks in Italian and—if bilingual—in their native language were compared to the results of standardized/conventional reading assessment tests as well as parental and teacher questionnaires. Correlations revealed the tasks that best contributed to the identification of risk for the presence of reading disorders and showed the general feasibility and usefulness of the computerized screening. In a further step, both screening administrators (Examiners) and child participants (Examinees) were invited to participate in usability studies, which revealed general satisfaction and provided suggestions for further improvement of the screening platform. Based on these findings, the potential of the novel web-based screening platform is discussed. Full article

Virtual reality devices are used for several application domains, such as medicine, entertainment, marketing and training. A handheld controller is the common interaction method for direct object manipulation in virtual reality environments. Using hands would be a straightforward way to directly manipulate objects in the virtual environment if hand-tracking technology were reliable enough. In recent comparison studies, hand-based systems compared unfavorably against the handheld controllers in task completion times and accuracy. In our controlled study, we compare these two interaction techniques with a new hybrid interaction technique which combines the controller tracking with hand gestures for a rigid object manipulation task. The results demonstrate that the hybrid interaction technique is the most preferred because it is intuitive, easy to use, fast, reliable and it provides haptic feedback resembling the real-world object grab. This suggests that there is a trade-off between naturalness, task accuracy and task completion time when using these direct manipulation interaction techniques, and participants prefer to use interaction techniques that provide a balance between these three factors. Full article

The article presents a contextual survey of eight contributions in the special issue Musical Interactions (Volume I) in Multimodal Technologies and Interaction. The presentation includes (1) a critical examination of what it means to be musical, to devise the concept of music proper to MTI as well as multicultural proximity, and (2) a conceptual framework for instrumentation, design, and assessment of musical interaction research through five enabling dimensions: Affordance; Design Alignment; Adaptive Learning; Second-Order Feedback; Temporal Integration. Each dimension is discussed and applied in the survey. The results demonstrate how the framework provides an interdisciplinary scope required for musical interaction, and how this approach may offer a coherent way to describe and assess approaches to research and design as well as implementations of interactive musical systems. Musical interaction stipulates musical liveness for experiencing both music and technologies. While music may be considered ontologically incomplete without a listener, musical interaction is defined as ontological completion of a state of music and listening through a listener’s active engagement with musical resources in multimodal information flow. Full article

Numerous statistics show that cyclists are often involved in road traffic accidents, often with serious outcomes. One potential hazard of cycling, especially in cities, is “dooring”—passing parked vehicles that still have occupants inside. These occupants could open the vehicle door unexpectedly in the cyclist’s path—requiring a quick evasive response by the cyclist to avoid a collision. Dooring can be very poorly anticipated; as a possible solution, we propose in this work a system that notifies the cyclist of opening doors based on a networked intelligent transportation infrastructure. In a user study with a bicycle simulator (N = 24), we examined the effects of three user interface designs compared to a baseline (no notifications) on cycling behavior (speed and lateral position), perceived safety, and ease of use. Awareness messages (either visual message, visual message + auditory icon, or visual + voice message) were displayed on a smart bicycle helmet at different times before passing a parked, still-occupied vehicle. Our participants found the notifications of potential hazards very easy to understand and appealing and felt that the alerts could help them navigate traffic more safely. Those concepts that (additionally) used auditory icons or voice messages were preferred. In addition, the lateral distance increased significantly when a potentially opening door was indicated. In these situations, cyclists were able to safely pass the parked vehicle without braking. In summary, we are convinced that notification systems, such as the one presented here, are an important component for increasing road safety, especially for vulnerable road users. Full article

The ubiquity of mobile devices in peoples’ everyday life makes them a feasible tool for language learning. Learning anytime and anywhere creates great flexibility but comes with the inherent risk of infrequent learning and learning in interruption-prone environments. No matter the length of the learning break, it can negatively affect knowledge consolidation and recall. This work presents the design and implementation of memory cues to support task resumption in mobile language learning applications and two evaluations to assess their impact on user experience. An initial laboratory experiment ($N=15$) revealed that while the presentation of the cues had no significant effect on objective performance measures (task completion time and error rate), the users still perceived the cues as helpful and would appreciate them in a mobile learning app. A follow-up study ($N=16$) investigated revised cue designs in a real-world field setting and found that users particularly appreciated our interactive test cue design. We discuss strengths and limitations of our concept and implications for the application of task resumption cues beyond the scope of mobile language learning. Full article

The ubiquity of multimodal smart devices affords new opportunities for eyes-free applications for conveying graphical information to both sighted and visually impaired users. Using previously established haptic design guidelines for generic rendering of graphical content on touchscreen interfaces, the current study evaluates the learning and mental representation of digital maps, representing a key real-world translational eyes-free application. Two experiments involving 12 blind participants and 16 sighted participants compared cognitive map development and test performance on a range of spatio-behavioral tasks across three information-matched learning-mode conditions: (1) our prototype vibro-audio map (VAM), (2) traditional hardcopy-tactile maps, and (3) visual maps. Results demonstrated that when perceptual parameters of the stimuli were matched between modalities during haptic and visual map learning, test performance was highly similar (functionally equivalent) between the learning modes and participant groups. These results suggest equivalent cognitive map formation between both blind and sighted users and between maps learned from different sensory inputs, providing compelling evidence supporting the development of amodal spatial representations in the brain. The practical implications of these results include empirical evidence supporting a growing interest in the efficacy of multisensory interfaces as a primary interaction style for people both with and without vision. Findings challenge the long-held assumption that blind people exhibit deficits on global spatial tasks compared to their sighted peers, with results also providing empirical support for the methodological use of sighted participants in studies pertaining to technologies primarily aimed at supporting blind users. Full article