Turkish lip-reading using Bi-LSTM and deep learning models

Simple item page
dc.authoridATILA, UMIT/0000-0002-1576-9977
dc.contributor.authorAtila, Uemit
dc.contributor.authorSabaz, Furkan
dc.date.accessioned2024-09-29T15:57:36Z
dc.date.available2024-09-29T15:57:36Z
dc.date.issued2022
dc.departmentKarabük Üniversitesien_US
dc.description.abstractIn recent years, lip-reading has been one of the studies whose importance has increased considerably, especially with the spread of deep learning applications. In this topic, researchers try to detect what a person says from video frames without sound. When the previous studies are analysed, it is seen that automatic lip-reading systems have been developed for various languages such as Chinese, Korean, English and German. However, these studies reveal that the development of the system is difficult because lip-reading from video frame images without audio data depends on many parameters such as light, shooting distance, and the gender of the person. Lip-reading systems were first developed using classical machine learning methods. However, especially in recent years, with the popularity of deep learning applications, this subject has started to be studied more than before and studies reveal that in general, deep learning-based lip-reading gives more successful results.Even though there are studies in this field in different languages, there is no current study and dataset in Turkish. Therefore, this study aims to investigate the performances of the state-of-the art deep learning models on Turkish lip-reading. To this aim, two new datasets, one with 111 words and other with 113 sentences were created using image processing techniques. The model used in this study to perform lip-reading extracts features from video frames using CNN based models and performs classification using Bidirectional Long Short-Term Memory (Bi-LSTM). Results of experiments reveal that, ResNet-18 and Bi-LSTM pair gives the best results in both word and sentence datasets with accuracy values 84.5% and 88.55%, respectively. It is also observed that, better performances are obtained in sentence recogni-tion than word recognition in almost every model implemented.(c) 2022 Karabuk University. Publishing services by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).en_US
dc.identifier.doi10.1016/j.jestch.2022.101206
dc.identifier.issn2215-0986
dc.identifier.scopus2-s2.0-85133350584en_US
dc.identifier.scopusqualityQ1en_US
dc.identifier.urihttps://doi.org/10.1016/j.jestch.2022.101206
dc.identifier.urihttps://hdl.handle.net/20.500.14619/4911
dc.identifier.volume35en_US
dc.identifier.wosWOS:000892526300009en_US
dc.identifier.wosqualityQ1en_US
dc.indekslendigikaynakWeb of Scienceen_US
dc.indekslendigikaynakScopusen_US
dc.language.isoenen_US
dc.publisherElsevier - Division Reed Elsevier India Pvt Ltden_US
dc.relation.ispartofEngineering Science and Technology-An International Journal-Jestechen_US
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/openAccessen_US
dc.subjectLip-readingen_US
dc.subjectBi-lstmen_US
dc.subjectDeep learningen_US
dc.subjectDataseten_US
dc.subjectTurkishen_US
dc.titleTurkish lip-reading using Bi-LSTM and deep learning modelsen_US
dc.typeArticleen_US

Files

Collections

WoS İndeksli Yayınlar Koleksiyonu
Scopus İndeksli Yayınlar Koleksiyonu