Real time Indian sign language recognition using transfer learning with VGG16

Abstract

Normal people’s interaction and communication are easier than those with disabilities such as hearing and speech, which are very complicated; hence, the use of sign language plays a crucial role in bridging this gap in communication. While previous attempts have been made to solve this problem using deep learning techniques, including convolutional neural networks (CNNs), support vector machine (SVM), and K-nearest neighbours (KNN), these have low accuracy or may not be employed in real time. This work addresses both issues: improving upon prior limitations and extending the challenge of classifying characters in Indian sign language (ISL). Our system, which can recognize 23 hand gestures of ISL through a purely camera-based approach, eliminates expensive hardware like hand gloves, thus making it economical. The system yields an accuracy of 97.5% on the training dataset, utilizing a pre-trained VGG16 CNN optimized by the Adam optimizer and cross-entropy loss function. These results clearly show how effective transfer learning is in classifying ISL and its possible real-world applications.