Fall 2021 Electrical Engineering and Computer Science (EECS) Seminar Series

"Lip Reading as an Active Mode of Interaction with Computer Systems"

Laxmi Pandey

UC Merced

Faculty Host: Prof. Ahmed Arif

Abstract:
Silent speech that converts lip movements into text can mitigate many challenges of speech and traditional input methods. Yet, existing silent speech recognition models are slow and error-prone, or use impractical and invasive setups, impeding their scalability in real-world scenarios. In this talk, I will present the findings of studies involving silent speech input. First, a social study exploring the social acceptance of silent speech input in different social contexts. Second, two empirical studies investigating user tolerance of recognition errors in silent speech input and identifying suitable feedback mechanism for real-time silent speech input on mobile devices. Third, a new end-to-end deep neural network that can automatically segment lip sequence videos and classify them into text. Fourth, an independent repair model which is a multi-stage pipeline that processes video input for poor lighting conditions, when applicable, and corrects potential errors in output for increased accuracy.

Speaker Bio:
Laxmi Pandey is a Ph.D. student of EECS at UC Merced, where she works at the Inclusive Interaction Lab. Her research focuses on multiple areas, including natural language processing, audio-visual speech recognition, and applied artificial intelligence. Her aim is to design and develop effective and accessible communication technologies for mobile devices. She has received many awards for her research activities, including the Fred and Mitzie Ruiz Fellowship, the Hatano Cognitive Development Research Fellowship, ACM-W Scholarship, and the SIGIR Travel Grants. She received her Master’s in Electrical Engineering from IIT Kanpur, India, and Bachelor’s in Electronics and Communication Engineering from Uttar Pradesh Technical University, India.