BenAV: A Bengali audio-visual corpus for visual speech recognition

Ashish Pondit; Muhammad Eshaque Ali Rukon; Anik Das; Ashad Kabir

doi:10.1007/978-3-030-92270-2_45

BenAV: A Bengali audio-visual corpus for visual speech recognition

Ashish Pondit, Muhammad Eshaque Ali Rukon, Anik Das, Ashad Kabir

Research output: Book chapter/Published conference paper › Conference paper › peer-review

Abstract

Visual speech recognition (VSR) is a challenging task. It has many applications such as facilitating speech recognition when the acoustic data is noisy or missing, assisting hearing impaired people, etc. Modern VSR systems require a large amount of data to achieve a good performance. Popular VSR datasets are mostly available for the English language and none in Bengali. In this paper, we present a large-scale Bengali audio-visual dataset, named "BenAV". To the best of our knowledge, BenAV is the first publicly available large-scale dataset in the Bengali language. BenAV contains a lexicon of 50 words from 128 speakers with a total number of 26,300 utterances. We have also applied three existing deep learning based VSR models to provide a baseline performance of our BenAV dataset. We run extensive experiments in two different configurations of the dataset to study the robustness of those models and achieved 98.70% and 82.5% accuracy, respectively. We believe that this research provides a basis to develop Bengali lip reading systems and opens the doors to conduct further research on this topic.

Original language	English
Title of host publication	Neural Information Processing
Subtitle of host publication	28th International Conference, ICONIP 2021, Proceedings, Part II
Editors	Teddy Mantoro, Minho Lee, Media Anugerah Ayu, Kok Wai Wong, Achmad Nizar Hidayanto
Place of Publication	Cham, Switzerland
Publisher	Springer
Pages	526-535
Number of pages	10
Volume	13109
ISBN (Electronic)	9783030922702
ISBN (Print)	9783030922696
DOIs	https://doi.org/10.1007/978-3-030-92270-2_45
Publication status	E-pub ahead of print - 07 Dec 2021
Event	The 28th International Conference on Neural Information Processing (ICONIP 2021): ICONIP 2021 - Online, Bali, Indonesia Duration: 08 Dec 2021 → 12 Dec 2021 https://web.archive.org/web/20220108032705/https://iconip2021.apnns.org/ (Conference website)

Publication series

Name	Lecture Notes in Computer Science
Publisher	Springer, Cham
Volume	13109
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	The 28th International Conference on Neural Information Processing (ICONIP 2021)
Country/Territory	Indonesia
City	Bali
Period	08/12/21 → 12/12/21
Other	The 28th International Conference on Neural Information Processing (ICONIP2021) aims to provide a leading international forum for researchers, scientists, and industry professionals who are working in neuroscience, neural networks, deep learning, and related fields to share their new ideas, progresses and achievement, through its regular sessions, special sessions, tutorials, and workshops. ICONIP 2021 will be held in online mode during December 8-12, 2021.
Internet address	https://web.archive.org/web/20220108032705/https://iconip2021.apnns.org/ (Conference website)

Access to Document

10.1007/978-3-030-92270-2_45

Cite this

Pondit, A., Rukon, M. E. A., Das, A., & Kabir, A. (2021). BenAV: A Bengali audio-visual corpus for visual speech recognition. In T. Mantoro, M. Lee, M. A. Ayu, K. W. Wong, & A. N. Hidayanto (Eds.), Neural Information Processing: 28th International Conference, ICONIP 2021, Proceedings, Part II (Vol. 13109, pp. 526-535). (Lecture Notes in Computer Science; Vol. 13109). Springer. https://doi.org/10.1007/978-3-030-92270-2_45

Pondit, Ashish ; Rukon, Muhammad Eshaque Ali ; Das, Anik et al. / BenAV : A Bengali audio-visual corpus for visual speech recognition. Neural Information Processing: 28th International Conference, ICONIP 2021, Proceedings, Part II. editor / Teddy Mantoro ; Minho Lee ; Media Anugerah Ayu ; Kok Wai Wong ; Achmad Nizar Hidayanto. Vol. 13109 Cham, Switzerland : Springer, 2021. pp. 526-535 (Lecture Notes in Computer Science).

@inproceedings{76c343ad5f124ed78d9b5b8ccf770a3f,

title = "BenAV: A Bengali audio-visual corpus for visual speech recognition",

abstract = "Visual speech recognition (VSR) is a challenging task. It has many applications such as facilitating speech recognition when the acoustic data is noisy or missing, assisting hearing impaired people, etc. Modern VSR systems require a large amount of data to achieve a good performance. Popular VSR datasets are mostly available for the English language and none in Bengali. In this paper, we present a large-scale Bengali audio-visual dataset, named {"}BenAV{"}. To the best of our knowledge, BenAV is the first publicly available large-scale dataset in the Bengali language. BenAV contains a lexicon of 50 words from 128 speakers with a total number of 26,300 utterances. We have also applied three existing deep learning based VSR models to provide a baseline performance of our BenAV dataset. We run extensive experiments in two different configurations of the dataset to study the robustness of those models and achieved 98.70% and 82.5% accuracy, respectively. We believe that this research provides a basis to develop Bengali lip reading systems and opens the doors to conduct further research on this topic.",

keywords = "Visual speech recognition, Audio-visual dataset, Lip reading, Corpus, Bengali, Deep learning",

author = "Ashish Pondit and Rukon, {Muhammad Eshaque Ali} and Anik Das and Ashad Kabir",

note = "Includes bibliographical references.; The 28th International Conference on Neural Information Processing (ICONIP 2021) : ICONIP 2021 ; Conference date: 08-12-2021 Through 12-12-2021",

year = "2021",

month = dec,

day = "7",

doi = "10.1007/978-3-030-92270-2_45",

language = "English",

isbn = "9783030922696",

volume = "13109",

series = "Lecture Notes in Computer Science",

publisher = "Springer",

pages = "526--535",

editor = "Teddy Mantoro and Minho Lee and Ayu, {Media Anugerah} and Wong, {Kok Wai} and Hidayanto, {Achmad Nizar}",

booktitle = "Neural Information Processing",

address = "United States",

url = "https://web.archive.org/web/20220108032705/https://iconip2021.apnns.org/",

}

Pondit, A, Rukon, MEA, Das, A & Kabir, A 2021, BenAV: A Bengali audio-visual corpus for visual speech recognition. in T Mantoro, M Lee, MA Ayu, KW Wong & AN Hidayanto (eds), Neural Information Processing: 28th International Conference, ICONIP 2021, Proceedings, Part II. vol. 13109, Lecture Notes in Computer Science, vol. 13109, Springer, Cham, Switzerland, pp. 526-535, The 28th International Conference on Neural Information Processing (ICONIP 2021), Bali, Indonesia, 08/12/21. https://doi.org/10.1007/978-3-030-92270-2_45

BenAV : A Bengali audio-visual corpus for visual speech recognition. / Pondit, Ashish; Rukon, Muhammad Eshaque Ali; Das, Anik et al.

Neural Information Processing: 28th International Conference, ICONIP 2021, Proceedings, Part II. ed. / Teddy Mantoro; Minho Lee; Media Anugerah Ayu; Kok Wai Wong; Achmad Nizar Hidayanto. Vol. 13109 Cham, Switzerland : Springer, 2021. p. 526-535 (Lecture Notes in Computer Science; Vol. 13109).

Research output: Book chapter/Published conference paper › Conference paper › peer-review

TY - GEN

T1 - BenAV

T2 - The 28th International Conference on Neural Information Processing (ICONIP 2021)

AU - Pondit, Ashish

AU - Rukon, Muhammad Eshaque Ali

AU - Das, Anik

AU - Kabir, Ashad

N1 - Includes bibliographical references.

PY - 2021/12/7

Y1 - 2021/12/7

N2 - Visual speech recognition (VSR) is a challenging task. It has many applications such as facilitating speech recognition when the acoustic data is noisy or missing, assisting hearing impaired people, etc. Modern VSR systems require a large amount of data to achieve a good performance. Popular VSR datasets are mostly available for the English language and none in Bengali. In this paper, we present a large-scale Bengali audio-visual dataset, named "BenAV". To the best of our knowledge, BenAV is the first publicly available large-scale dataset in the Bengali language. BenAV contains a lexicon of 50 words from 128 speakers with a total number of 26,300 utterances. We have also applied three existing deep learning based VSR models to provide a baseline performance of our BenAV dataset. We run extensive experiments in two different configurations of the dataset to study the robustness of those models and achieved 98.70% and 82.5% accuracy, respectively. We believe that this research provides a basis to develop Bengali lip reading systems and opens the doors to conduct further research on this topic.

AB - Visual speech recognition (VSR) is a challenging task. It has many applications such as facilitating speech recognition when the acoustic data is noisy or missing, assisting hearing impaired people, etc. Modern VSR systems require a large amount of data to achieve a good performance. Popular VSR datasets are mostly available for the English language and none in Bengali. In this paper, we present a large-scale Bengali audio-visual dataset, named "BenAV". To the best of our knowledge, BenAV is the first publicly available large-scale dataset in the Bengali language. BenAV contains a lexicon of 50 words from 128 speakers with a total number of 26,300 utterances. We have also applied three existing deep learning based VSR models to provide a baseline performance of our BenAV dataset. We run extensive experiments in two different configurations of the dataset to study the robustness of those models and achieved 98.70% and 82.5% accuracy, respectively. We believe that this research provides a basis to develop Bengali lip reading systems and opens the doors to conduct further research on this topic.

KW - Visual speech recognition

KW - Audio-visual dataset

KW - Lip reading

KW - Corpus

KW - Bengali

KW - Deep learning

U2 - 10.1007/978-3-030-92270-2_45

DO - 10.1007/978-3-030-92270-2_45

M3 - Conference paper

SN - 9783030922696

VL - 13109

T3 - Lecture Notes in Computer Science

SP - 526

EP - 535

BT - Neural Information Processing

A2 - Mantoro, Teddy

A2 - Lee, Minho

A2 - Ayu, Media Anugerah

A2 - Wong, Kok Wai

A2 - Hidayanto, Achmad Nizar

PB - Springer

CY - Cham, Switzerland

Y2 - 8 December 2021 through 12 December 2021

ER -

Pondit A, Rukon MEA, Das A, Kabir A. BenAV: A Bengali audio-visual corpus for visual speech recognition. In Mantoro T, Lee M, Ayu MA, Wong KW, Hidayanto AN, editors, Neural Information Processing: 28th International Conference, ICONIP 2021, Proceedings, Part II. Vol. 13109. Cham, Switzerland: Springer. 2021. p. 526-535. (Lecture Notes in Computer Science). Epub 2021 Dec 7. doi: 10.1007/978-3-030-92270-2_45

BenAV: A Bengali audio-visual corpus for visual speech recognition

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this