Anuvadak: An Indian Sign language translator to text phrase and audio

Abstract: Objectives: To develop a versatile real-time sign language recognition system for individuals with hearing, visual, and speech impairments. Thus bridging the communication gap and enabling their interaction with general populace. Methods: Utilized deep learning concepts like transfer learning with Inceptionv3 CNN architecture and NLP models Pegasus paraphraser and Gramformer for gesture recognition and text generation. Applied transfer learning for efficient training on a custom dataset, ensuring high accuracy and performance. Findings: The Inceptionv3-based system achieved 92% training accuracy and 95% validation accuracy. The integration of Gramformer enabled the generation of coherent text from recognized gestures. The system's output includes Kannada text and audio, providing accessibility for diverse disabled communities. The findings align with existing research on deep learning for sign language recognition, but our system uniquely offers curated image processing techniques, real-time multilingual support and audio synthesis, enhancing communication inclusivity. Novelty: First to provide real-time sign language translation into Kannada text and audio, combining advanced CNN(InceptionV3) and NLP models. The novel integration of these technologies allows for seamless communication across different disabilities, fostering greater inclusivity.