Research

16 pages, 2398 KiB

Open AccessArticle

Research on Robust Audio-Visual Speech Recognition Algorithms

by Wenfeng Yang, Pengyi Li, Wei Yang, Yuxing Liu, Yulong He, Ovanes Petrosian and Aleksandr Davydenko

Mathematics 2023, 11(7), 1733; https://0-doi-org.brum.beds.ac.uk/10.3390/math11071733 - 05 Apr 2023

Cited by 2 | Viewed by 2221

Abstract

Automatic speech recognition (ASR) that relies on audio input suffers from significant degradation in noisy conditions and is particularly vulnerable to speech interference. However, video recordings of speech capture both visual and audio signals, providing a potent source of information for training speech [...] Read more.

Automatic speech recognition (ASR) that relies on audio input suffers from significant degradation in noisy conditions and is particularly vulnerable to speech interference. However, video recordings of speech capture both visual and audio signals, providing a potent source of information for training speech models. Audiovisual speech recognition (AVSR) systems enhance the robustness of ASR by incorporating visual information from lip movements and associated sound production in addition to the auditory input. There are many audiovisual speech recognition models and systems for speech transcription, but most of them have been tested based in a single experimental setting and with a limited dataset. However, a good model should be applicable to any scenario. Our main contributions are: (i) Reproducing the three best-performing audiovisual speech recognition models in the current AVSR research area using the most famous audiovisual databases, LSR2 (Lip Reading Sentences 2) LSR3 (Lip Reading Sentences 3), and comparing and analyzing their performances under various noise conditions. (ii) Based on our experimental and research experiences, we analyzed the problems currently encountered in the AVSR domain, which are summarized as the feature-extraction problem and the domain-generalization problem. (iii) According to the experimental results, the Moco (momentum contrast) + word2vec (word to vector) model has the best AVSR effect on the LRS datasets regardless of whether there is noise or not. Additionally, the model also produced the best experimental results in the experiments of audio recognition and video recognition. Our research lays the foundation for further improving the performance of AVSR models. Full article

(This article belongs to the Special Issue Application of Mathematical Methods in Artificial Intelligence)

► Show Figures

Figure 1

22 pages, 4208 KiB

Open AccessArticle

An Intelligent Edge-as-a-Service Framework to Combat COVID-19 Using Deep Learning Techniques

by Mohammad Mehedi Hassan, Mabrook S. AlRakhami, Amerah A. Alabrah and Salman A. AlQahtani

Mathematics 2023, 11(5), 1216; https://0-doi-org.brum.beds.ac.uk/10.3390/math11051216 - 01 Mar 2023

Cited by 1 | Viewed by 1350

Abstract

This study proposes and develops a secured edge-assisted deep learning (DL)-based automatic COVID-19 detection framework that utilizes the cloud and edge computing assistance as a service with a 5G network and blockchain technologies. The development of artificial intelligence methods through services at the [...] Read more.

This study proposes and develops a secured edge-assisted deep learning (DL)-based automatic COVID-19 detection framework that utilizes the cloud and edge computing assistance as a service with a 5G network and blockchain technologies. The development of artificial intelligence methods through services at the edge plays a significant role in serving many applications in different domains. Recently, some DL approaches have been proposed to successfully detect COVID-19 by analyzing chest X-ray (CXR) images in the cloud and edge computing environments. However, the existing DL methods leverage only local and small training datasets. To overcome these limitations, we employed the edges to perform three tasks. The first task was to collect data from different hospitals and send them to a global cloud to train a DL model on massive datasets. The second task was to integrate all the trained models on the cloud to detect COVID-19 cases automatically. The third task was to retrain the trained model on specific COVID-19 data locally at hospitals to improve and generalize the trained model. A feature-level fusion and reduction were adopted for model performance enhancement. Experimental results on a public CXR dataset demonstrated an improvement against recent related work, achieving the quality-of-service requirements. Full article

(This article belongs to the Special Issue Application of Mathematical Methods in Artificial Intelligence)

► Show Figures

Figure 1

17 pages, 5869 KiB

Open AccessArticle

Enhanced Evaluation Method of Musical Instrument Digital Interface Data based on Random Masking and Seq2Seq Model

by Zhe Jiang, Shuyu Li and Yunsick Sung

Mathematics 2022, 10(15), 2747; https://0-doi-org.brum.beds.ac.uk/10.3390/math10152747 - 03 Aug 2022

Cited by 3 | Viewed by 1488

Abstract

With developments in artificial intelligence (AI), it is possible for novel applications to utilize deep learning to compose music by the format of musical instrument digital interface (MIDI) even without any knowledge of musical theory. The composed music is generally evaluated by human-based [...] Read more.

With developments in artificial intelligence (AI), it is possible for novel applications to utilize deep learning to compose music by the format of musical instrument digital interface (MIDI) even without any knowledge of musical theory. The composed music is generally evaluated by human-based Turing test, which is a subjective approach and does not provide any quantitative criteria. Therefore, objective evaluation approaches with many general descriptive parameters are applied to the evaluation of MIDI data while considering MIDI features such as pitch distances, chord rates, tone spans, drum patterns, etc. However, setting several general descriptive parameters manually on large datasets is difficult and has considerable generalization limitations. In this paper, an enhanced evaluation method based on random masking and sequence-to-sequence (Seq2Seq) model is proposed to evaluate MIDI data. An experiment was conducted on real MIDI data, generated MIDI data, and random MIDI data. The bilingual evaluation understudy (BLEU) is a common MIDI data evaluation approach and is used here to evaluate the performance of the proposed method in a comparative study. In the proposed method, the ratio of the average evaluation score of the generated MIDI data to that of the real MIDI data was 31%, while that of BLEU was 79%. The lesser the ratio, the greater the difference between the real MIDI data and generated MIDI data. This implies that the proposed method quantified the gap while accurately identifying real and generated MIDI data. Full article

(This article belongs to the Special Issue Application of Mathematical Methods in Artificial Intelligence)

► Show Figures

Figure 1

21 pages, 5214 KiB

Open AccessArticle

Singular Spectrum Analysis of Tremorograms for Human Neuromotor Reaction Estimation

by Olga Bureneva, Nikolay Safyannikov and Zoya Aleksanyan

Mathematics 2022, 10(11), 1794; https://doi.org/10.3390/math10111794 - 24 May 2022

Cited by 3 | Viewed by 1761

Abstract

Singular spectrum analysis (SSA) is a method of time series analysis and is used in various fields, including medicine. A tremorogram is a biological signal that allows evaluation of a person’s neuromotor reactions in order to infer the state of the motor parts [...] Read more.

Singular spectrum analysis (SSA) is a method of time series analysis and is used in various fields, including medicine. A tremorogram is a biological signal that allows evaluation of a person’s neuromotor reactions in order to infer the state of the motor parts of the central nervous system (CNS). A tremorogram has a complex structure, and its analysis requires the use of advanced methods of signal processing and intelligent analysis. The paper’s novelty lies in the application of the SSA method to extract diagnostically significant features from tremorograms with subsequent evaluation of the state of the motor parts of the CNS. The article presents the application of a method of singular spectrum decomposition, comparison of known variants of classification, and grouping of principal components for determining the components of the tremorogram corresponding to the trend, periodic components, and noise. After analyzing the results of the SSA of tremorograms, we proposed a new algorithm of grouping based on the analysis of singular values of the trajectory matrix. An example of applying the SSA method to the analysis of tremorograms is shown. Comparison of known clustering methods and the proposed algorithm showed that there is a reasonable correspondence between the proposed algorithm and the traditional methods of classification and pairing in the set of periodic components. Full article

(This article belongs to the Special Issue Application of Mathematical Methods in Artificial Intelligence)

► Show Figures

Figure 1

13 pages, 4610 KiB

Open AccessArticle

Determination of Significant Parameters on the Basis of Methods of Mathematical Statistics, and Boolean and Fuzzy Logic

by Yulia Shichkina, Mikhail Petrov and Fatkieva Roza

Mathematics 2022, 10(7), 1133; https://0-doi-org.brum.beds.ac.uk/10.3390/math10071133 - 01 Apr 2022

Cited by 2 | Viewed by 1614

Abstract

Among the set of parameters for which data are collected for decision-making based on artificial intelligence methods, often only some of the parameters are significant. This article compares methods for determining the significant parameters based on the theory of mathematical statistics, and fuzzy [...] Read more.

Among the set of parameters for which data are collected for decision-making based on artificial intelligence methods, often only some of the parameters are significant. This article compares methods for determining the significant parameters based on the theory of mathematical statistics, and fuzzy and boolean logic. The testing was conducted on several test data sets with a different number of parameters and different variability of parameter values. It was shown that for data sets with a small number of parameters (<5), the most accurate result was given for a method based on the theory of mathematical statistics and boolean logic. For a data set with a large number of parameters—the most suitable is the method of fuzzy logic. Full article

(This article belongs to the Special Issue Application of Mathematical Methods in Artificial Intelligence)

► Show Figures

Figure 1

18 pages, 2850 KiB

Open AccessArticle

Multi-Drone 3D Building Reconstruction Method

by Anton Filatov, Mark Zaslavskiy and Kirill Krinkin

Mathematics 2021, 9(23), 3033; https://0-doi-org.brum.beds.ac.uk/10.3390/math9233033 - 26 Nov 2021

Cited by 4 | Viewed by 3014

Abstract

In the recent decade, the rapid development of drone technologies has made many spatial problems easier to solve, including the problem of 3D reconstruction of large objects. A review of existing solutions has shown that most of the works lack the autonomy of [...] Read more.

In the recent decade, the rapid development of drone technologies has made many spatial problems easier to solve, including the problem of 3D reconstruction of large objects. A review of existing solutions has shown that most of the works lack the autonomy of drones because of nonscalable mapping techniques. This paper presents a method for centralized multi-drone 3D reconstruction, which allows performing a data capturing process autonomously and requires drones equipped only with an RGB camera. The essence of the method is a multiagent approach—the control center performs the workload distribution evenly and independently for all drones, allowing simultaneous flights without a high risk of collision. The center continuously receives RGB data from drones and performs each drone localization (using visual odometry estimations) and rough online mapping of the environment (using image descriptors for estimating the distance to the building). The method relies on a set of several user-defined parameters, which allows the tuning of the method for different task-specific requirements such as the number of drones, 3D model detalization, data capturing time, and energy consumption. By numerical experiments, it is shown that method parameters can be estimated by performing a set of computations requiring characteristics of drones and the building that are simple to obtain. Method performance was evaluated by an experiment with virtual building and emulated drone sensors. Experimental evaluation showed that the precision of the chosen algorithms for online localization and mapping is enough to perform simultaneous flights and the amount of captured RGB data is enough for further reconstruction. Full article

(This article belongs to the Special Issue Application of Mathematical Methods in Artificial Intelligence)

► Show Figures

Figure 1

19 pages, 533 KiB

Open AccessArticle

Deep Gene Networks and Response to Stress

by Sergey Vakulenko and Dmitry Grigoriev

Mathematics 2021, 9(23), 3028; https://0-doi-org.brum.beds.ac.uk/10.3390/math9233028 - 26 Nov 2021

Cited by 2 | Viewed by 1107

Abstract

We consider systems of differential equations with polynomial and rational nonlinearities and with a dependence on a discrete parameter. Such systems arise in biological and ecological applications, where the discrete parameter can be interpreted as a genetic code. The genetic code defines system [...] Read more.

We consider systems of differential equations with polynomial and rational nonlinearities and with a dependence on a discrete parameter. Such systems arise in biological and ecological applications, where the discrete parameter can be interpreted as a genetic code. The genetic code defines system responses to external perturbations. We suppose that these responses are defined by deep networks. We investigate the stability of attractors of our systems under sequences of perturbations (for example, stresses induced by environmental changes), and we introduce a new concept of biosystem stability via gene regulation. We show that if the gene regulation is absent, then biosystems sooner or later collapse under fluctuations. By a genetic regulation, one can provide attractor stability for large times. Therefore, in the framework of our model, we prove the Gromov–Carbone hypothesis that evolution by replication makes biosystems robust against random fluctuations. We apply these results to a model of cancer immune therapy. Full article

(This article belongs to the Special Issue Application of Mathematical Methods in Artificial Intelligence)

► Show Figures

Figure 1

12 pages, 4412 KiB

Open AccessArticle

Comparison and Explanation of Forecasting Algorithms for Energy Time Series

by Yuyi Zhang, Ruimin Ma, Jing Liu, Xiuxiu Liu, Ovanes Petrosian and Kirill Krinkin

Mathematics 2021, 9(21), 2794; https://0-doi-org.brum.beds.ac.uk/10.3390/math9212794 - 04 Nov 2021

Cited by 9 | Viewed by 2778

Abstract

In this work, energy time series forecasting competitions from the Schneider Company, the Kaggle Online platform, and the American society ASHRAE were considered. These competitions include power generation and building energy consumption forecasts. The datasets used in these competitions are based on reliable [...] Read more.

In this work, energy time series forecasting competitions from the Schneider Company, the Kaggle Online platform, and the American society ASHRAE were considered. These competitions include power generation and building energy consumption forecasts. The datasets used in these competitions are based on reliable and real sensor records. In addition, exogenous variables are accurately added to the dataset. All of these ensure the richness of the information contained in the dataset, which is crucial for energy management. Therefore, (1) We choose to study forecast models suitable for energy management on these energy datasets; (2) Forecast models including popular algorithm structures such as neural network models and ensemble models. In addition, as an innovation, we introduce the Explainable AI method (SHAP) to explain models with excellent performance indicators, thereby strengthening its trust and transparency; (3) The results show that the performance of the integrated model in these competitions is more stable and efficient, and in the integrated model, the advantages of LightGBM are more obvious; (4) Through the interpretation of SHAP, we found that the lagging characteristics of the building area and target variables are important features. Full article

(This article belongs to the Special Issue Application of Mathematical Methods in Artificial Intelligence)

► Show Figures

Figure 1

20 pages, 56766 KiB

Open AccessArticle

Cancer Cell Profiling Using Image Moments and Neural Networks with Model Agnostic Explainability: A Case Study of Breast Cancer Histopathological (BreakHis) Database

by Dmitry Kaplun, Alexander Krasichkov, Petr Chetyrbok, Nikolay Oleinikov, Anupam Garg and Husanbir Singh Pannu

Mathematics 2021, 9(20), 2616; https://0-doi-org.brum.beds.ac.uk/10.3390/math9202616 - 17 Oct 2021

Cited by 9 | Viewed by 2973

Abstract

With the evolution of modern digital pathology, examining cancer cell tissues has paved the way to quantify subtle symptoms, for example, by means of image staining procedures using Eosin and Hematoxylin. Cancer tissues in the case of breast and lung cancer are quite [...] Read more.

With the evolution of modern digital pathology, examining cancer cell tissues has paved the way to quantify subtle symptoms, for example, by means of image staining procedures using Eosin and Hematoxylin. Cancer tissues in the case of breast and lung cancer are quite challenging to examine by manual expert analysis of patients suffering from cancer. Merely relying on the observable characteristics by histopathologists for cell profiling may under-constrain the scale and diagnostic quality due to tedious repetition with constant concentration. Thus, automatic analysis of cancer cells has been proposed with algorithmic and soft-computing techniques to leverage speed and reliability. The paper’s novelty lies in the utility of Zernike image moments to extract complex features from cancer cell images and using simple neural networks for classification, followed by explainability on the test results using the Local Interpretable Model-Agnostic Explanations (LIME) technique and Explainable Artificial Intelligence (XAI). The general workflow of the proposed high throughput strategy involves acquiring the BreakHis public dataset, which consists of microscopic images, followed by the application of image processing and machine learning techniques. The recommended technique has been mathematically substantiated and compared with the state-of-the-art to justify the empirical basis in the pursuit of our algorithmic discovery. The proposed system is able to classify malignant and benign cancer cell images of 40× resolution with 100% recognition rate. XAI interprets and reasons the test results obtained from the machine learning model, making it reliable and transparent for analysis and parameter tuning. Full article

(This article belongs to the Special Issue Application of Mathematical Methods in Artificial Intelligence)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Application of Mathematical Methods in Artificial Intelligence

Share This Special Issue

Special Issue Editors

Special Issue Information

Keywords

Published Papers (9 papers)

Research

Further Information

Guidelines

MDPI Initiatives

Follow MDPI