Enhancing Indonesian Sign Language Recognition Accuracy Using Transfer Learning for Deaf Learners
Abstract
The recognition of Indonesian Sign Language remains challenging because of limited annotated datasets and the complexity of distinguishing similar hand gestures, reducing the effectiveness of automated communication systems for deaf learners. This study aims to improve the accuracy of Indonesian Sign Language recognition by applying transfer learning to deep learning models. An experimental approach was employed by fine-tuning two pre-trained convolutional neural network architectures, MobileNetV2 and EfficientNetB0, using a dataset of Indonesian Sign Language hand gestures. Model performance was evaluated using accuracy, precision, recall, and F1-score to measure classification effectiveness. The experimental results demonstrate that MobileNetV2 achieved a test accuracy of 95.8%, considerably outperforming EfficientNetB0, which obtained 3.85% accuracy. Most gesture classes were classified with high precision and recall, although several visually similar gestures remained more difficult to distinguish, indicating the influence of subtle variations in hand configuration. These findings show that lightweight transfer learning models can provide robust performance for Indonesian Sign Language recognition when properly adapted to local datasets. This study concludes that MobileNetV2 is a more suitable architecture for Indonesian Sign Language recognition and has strong potential to support accessible communication technologies and promote more inclusive learning environments for deaf students in Indonesia.
References
Abdullahi, S. B., & Chamnongthai, K. (2022). American Sign Language Words Recognition of Skeletal Videos Using Processed Video Driven Multi-Stacked Deep LSTM. Sensors, 22(4), 1406. https://doi.org/10.3390/s22041406
Agustiani, S., Haryani, H., Junaidi, A., Putri, R. R., & Adam Z, M. G. (2025). Comparative Optimization of EfficientNetB3, MobileNetV2, and ResNet50 for Waste Classification. Jurnal Informatika, 12(2), 131–137. https://doi.org/10.31294/inf.v12i2.27533
Agustiansyah, Y., & Kurniadi, D. (2025). Indonesian Sign Language Alphabet Image Classification using Vision Transformer. Journal of Intelligent Systems Technology and Informatics, 1(1), 1–9. https://doi.org/10.64878/jistics.v1i1.5
Alaftekin, M., Pacal, I., & Cicek, K. (2024). Real-time sign language recognition based on YOLO algorithm. Neural Computing and Applications, 36(14), 7609–7624. https://doi.org/10.1007/s00521-024-09503-6
Balat, M., Awaad, R., Adel, H., Zaky, A. B., & Aly, S. A. (2024). Advanced Arabic Alphabet Sign Language Recognition Using Transfer Learning and Transformer Models. https://doi.org/10.48550/ARXIV.2410.00681
Baumgartl, H., & Buettner, R. (2021). Developing efficient transfer learning strategies for robust scene recognition in mobile robotics using pre-trained convolutional neural networks (Version 1). arXiv. https://doi.org/10.48550/ARXIV.2107.11187
Briones Cerquín, A. D., Tumay Guevara, J. A., & Ovalle, C. (2025). Mobile Application for Continuous Recognition and Classification of Sign Language Images through Deep Learning. International Journal of Interactive Mobile Technologies (iJIM), 19(07), 4–21. https://doi.org/10.3991/ijim.v19i07.52853
Daniels, S., Suciati, N., & Fathichah, C. (2021). Indonesian Sign Language Recognition using YOLO Method. IOP Conference Series: Materials Science and Engineering, 1077(1), 012029. https://doi.org/10.1088/1757-899X/1077/1/012029
Das, S., Biswas, S. Kr., & Purkayastha, B. (2023). A deep sign language recognition system for Indian sign language. Neural Computing and Applications, 35(2), 1469–1481. https://doi.org/10.1007/s00521-022-07840-y
Dwijayanti, S., -, H., Taqiyyah, S. I., Hikmarika, H., & Suprapto, B. Y. (2021). Indonesia Sign Language Recognition using Convolutional Neural Network. International Journal of Advanced Computer Science and Applications, 12(10). https://doi.org/10.14569/IJACSA.2021.0121046
Fauzi, D. R., & Haqdu D, G. A. (2025b). Comparison of CNN Models Using EfficientNetB0, MobileNetV2, and ResNet50 for Traffic Density with Transfer Learning. Journal of Intelligent Systems Technology and Informatics, 1(1), 22–30. https://doi.org/10.64878/jistics.v1i1.6
Hugar, G., & Kagalkar, R. M. (2025). Hybrid Dual-Stream Deep Learning Approach for Real-Time Kannada Sign Language Recognition in Assistive Healthcare. Journal of Information Systems Engineering and Business Intelligence, 11(3), 393–406. https://doi.org/10.20473/jisebi.11.3.393-406
Iqbal, S., N. Qureshi, A., Li, J., & Mahmood, T. (2023). On the Analyses of Medical Images Using Traditional Machine Learning Techniques and Convolutional Neural Networks. Archives of Computational Methods in Engineering, 30(5), 3173–3233. https://doi.org/10.1007/s11831-023-09899-9
Joan, D., Vincent, V., Daniel, K. J., Achmad, S., & Sutoyo, R. (2023). BISINDO Hand-Sign Detection Using Transfer Learning. 2023 IEEE 8th International Conference on Recent Advances and Innovations in Engineering (ICRAIE), 1–7. https://doi.org/10.1109/ICRAIE59459.2023.10468194
Kelana, E. L., Anshori Prasetya, M. R., . M., & Zulfadhilah, M. (2025). Integrating the CNN Model with the Web for Indonesian Sign Language (BISINDO) Recognition. Journal of Applied Informatics and Computing, 9(3), 883–896. https://doi.org/10.30871/jaic.v9i3.9345
Kindiroglu, A. A., Kara, O., Ozdemir, O., & Akarun, L. (2024). Transfer Learning for Cross-dataset Isolated Sign Language Recognition in Under-Resourced Datasets (Version 2). arXiv. https://doi.org/10.48550/ARXIV.2403.14534
Kingma, D. P., & Ba, J. (2014). Adam: A Method for Stochastic Optimization (Version 9). arXiv. https://doi.org/10.48550/ARXIV.1412.6980
Lamaakal, I., Yahyati, C., Maleh, Y., Makkaoui, K. E., & Ouahbi, I. (2026). An explainable hybrid CNN–transformer model for sign language recognition on edge devices using adaptive fusion and knowledge distillation. Scientific Reports, 16(1), 7143. https://doi.org/10.1038/s41598-026-38478-8
Miah, A. S. M., Hasan, Md. A. M., Tomioka, Y., & Shin, J. (2024). Hand Gesture Recognition for Multi-Culture Sign Language Using Graph and General Deep Learning Network. IEEE Open Journal of the Computer Society, 5, 144–155. https://doi.org/10.1109/OJCS.2024.3370971
Papatsimouli, M., Sarigiannidis, P., & Fragulis, G. F. (2023). A Survey of Advancements in Real-Time Sign Language Translators: Integration with IoT Technology. Technologies, 11(4), 83. https://doi.org/10.3390/technologies11040083
Podder, K. K., Ezeddin, M., Chowdhury, M. E. H., Sumon, Md. S. I., Tahir, A. M., Ayari, M. A., Dutta, P., Khandakar, A., Mahbub, Z. B., & Kadir, M. A. (2023). Signer-Independent Arabic Sign Language Recognition System Using Deep Learning Model. Sensors, 23(16), 7156. https://doi.org/10.3390/s23167156
Rachmawati, I. D. A., Yunanda, R., Hidayat, M. F., & Wicaksono, P. (2023). Deep Transfer Learning for Sign Language Image Classification: A Bisindo Dataset Study. Engineering, MAthematics and Computer Science Journal (EMACS), 5(3), 175–180. https://doi.org/10.21512/emacsjournal.v5i3.10621
Ridwan, A. E. M., Chowdhury, M. I., Mary, M. M., & Abir, M. T. C. (2024b). Deep Neural Network-Based Sign Language Recognition: A Comprehensive Approach Using Transfer Learning with Explainability (arXiv:2409.07426). arXiv. https://doi.org/10.48550/arXiv.2409.07426
Rizka Fadhillah, I., Muharrom Al Haromainy, M., & Maulana, H. (2024a). Implementasi Model Transfer Learning Efficientnet Untuk Pendeteksian Bahasa Isyarat Indonesia (Bisindo) Pada Perangkat Android. JATI (Jurnal Mahasiswa Teknik Informatika), 8(4), 7816–7822. https://doi.org/10.36040/jati.v8i4.10463
Sadik, R., Majumder, A., Biswas, A. A., Ahammad, B., & Rahman, Md. M. (2023). An in-depth analysis of Convolutional Neural Network architectures with transfer learning for skin disease diagnosis. Healthcare Analytics, 3, 100143. https://doi.org/10.1016/j.health.2023.100143
Sutjiadi, R. (2023). Android-Based Application for Real-Time Indonesian Sign Language Recognition Using Convolutional Neural Network. TEM Journal, 1541–1549. https://doi.org/10.18421/TEM123-35
Tan, M., & Le, Q. V. (2021). EfficientNetV2: Smaller Models and Faster Training. Proceedings of the 38th International Conference on Machine Learning, PMLR.
TensorFlow Developers. (2024). Transfer Learning and Fine-tuning Guide [Computer software]. https://www.tensorflow.org/tutorials/images/transfer_learning
Töngi, R. (2021). Application of Transfer Learning to Sign Language Recognition using an Inflated 3D Deep Convolutional Neural Network (Version 1). arXiv. https://doi.org/10.48550/ARXIV.2103.05111
Wahyudi, I. P. A. T., Sudipa, I. G. I., Libraeni, L. G. B., Radhitya, M. L., & Asana, I. M. D. P. (2025a). Performance Comparison of MobileNet and EfficientNet Architectures in Automatic Classification of Bacterial Colonies. Indonesian Journal of Data and Science, 6(2), 333–342. https://doi.org/10.56705/ijodas.v6i2.218
Xia, K., Lu, W., Fan, H., & Zhao, Q. (2022). A Sign Language Recognition System Applied to Deaf-Mute Medical Consultation. Sensors, 22(23), 9107. https://doi.org/10.3390/s22239107
Zaelani, F., & Miftahuddin, Y. (2022). Perbandingan Metode EfficientNetB3 dan MobileNetV2 Untuk Identifikasi Jenis Buah-buahan Menggunakan Fitur Daun: Metode EfficientNetB3 dan MobileNetv2. Jurnal Ilmiah Teknologi Infomasi Terapan, 9(1). https://doi.org/10.33197/jitter.vol9.iss1.2022.911
Zufria, I., Harumy, T. H. F., & Efendi, S. (2025). Identifying the best models for BISINDO alphabet gesture classification to support the communication needs of the deaf community. Eastern-European Journal of Enterprise Technologies, 6(2 (138)), 26–41. https://doi.org/10.15587/1729-4061.2025.332096
Copyright (c) 2026 Yuvi Darmayunata, Lucky Lhaura Van FC, Veby, Didik Siswanto, Devia Kartika
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors retain the copyright and full publishing rights of their articles without restrictions. Authors grant Moestopo International Review on Social, Humanities, and Sciences (MIRSHuS) the non-exclusive right of first publication.
