Machine Learning-Based Sentiment Classification of Reviews from Indonesian Mobile Applications Using TF-IDF

Tuti Handayani; Sri Mardiyati

doi:10.59431/jms.v4i2.972

Quick Links

Journal

Home: Journal Home
Current: Current Issue
Archives: All Issues
News: Announcements
Search: Search Articles

About

About: About Journal
Team: Editorial Team
Reviewers: Reviewer Team
Indexing: Indexed In

For Authors

Guidelines: Author Guidelines
Submit: Submit Article
Fees: Publication Fees

Policies

Ethics: Publication Ethics
Review: Reviewer Guide
Access: Open Access
Contact: Contact Us

081377790601
journal@msti-indonesia.com
Mon - Sat: 08:00 - 15:00

Articles

PDF

Articles

Machine Learning-Based Sentiment Classification of Reviews from Indonesian Mobile Applications Using TF-IDF

Tuti Handayani, Sri Mardiyati

View Author Affiliations

Tuti Handayani: Universitas Indraprasta PGRI, Indonesia
Sri Mardiyati: Universitas Indraprasta PGRI, Indonesia

DOI:

https://doi.org/10.59431/jms.v4i2.972

Published:

September 10, 2026

Pages:

95–105

16

Views

2

Downloads

Article Metrics

Abstract

The increasing volume of mobile application reviews has created a need for automated sentiment classification capable of handling large-scale and imbalanced user-generated data. This study aimed to compare classical machine learning algorithms and evaluate the effectiveness of class-weighted learning in improving minority-class recognition in Indonesian mobile application reviews. The dataset was cleaned, deduplicated, and controlled for conflicting labels and data leakage, resulting in 357,327 unique reviews across Negative, Neutral, and Positive sentiment classes. Textual features were represented using Term Frequency-Inverse Document Frequency (TF-IDF) with 50,000 features, and the data were divided using an 80:20 stratified train-test split. Logistic Regression, Multinomial Naive Bayes, Linear Support Vector Machine (SVM), and Random Forest were evaluated alongside class-weighted Logistic Regression and Linear SVM. Logistic Regression achieved the highest accuracy of 83.47%, whereas Balanced Linear SVM achieved the highest Macro F1-score of 62.87% and improved the Neutral F1-score from 8.70% to 21.18%, while maintaining an accuracy of 79.09%. The findings demonstrate that the highest overall accuracy does not necessarily indicate the most balanced classifier under class imbalance. Balanced Linear SVM provided the most favorable trade-off between overall performance and minority-class recognition, highlighting the importance of class-sensitive evaluation and class-weighted learning for imbalanced sentiment classification.

Keywords

Class Imbalance; Class-Weighted Learning; Indonesian Mobile Application Reviews; Sentiment Analysis; TF-IDF

Author Biographies

Tuti Handayani

Universitas Indraprasta PGRI

Department of Informatics Engineering, Universitas Indraprasta PGRI, South Jakarta City, Special Capital Region of Jakarta, Indonesia

Sri Mardiyati

Universitas Indraprasta PGRI

Department of Informatics Engineering, Universitas Indraprasta PGRI, South Jakarta City, Special Capital Region of Jakarta, Indonesia

Article Identifiers

Article Title: Machine Learning-Based Sentiment Classification of Reviews from Indonesian Mobile Applications Using TF-IDF
DOI: 10.59431/jms.v4i2.972
Publication Date: 2026-09-10

Journal: Journal Mobile Technologies (JMS)
Volume: 4
Issue: 2
Pages: 95–105

References

Abbas, S., Boulila, W., Driss, M., Sampedro, G. A., Abisado, M. B., & Almadhor, A. S. (2024). Active learning empowered sentiment analysis: An approach for optimizing smartphone customer's review sentiment classification. IEEE Transactions on Consumer Electronics. https://doi.org/10.1109/tce.2023.3328878
Allam, H., Makubvure, L., Gyamfi, B., Graham, K. N., & Akinwolere, K. (2025). Text classification: How machine learning is revolutionizing text categorization. Information, 16(2), 130. https://doi.org/10.3390/info16020130
Assi, M., Hassan, S., & Zou, Y. (2025). LLM-Cure: LLM-based competitor user review analysis for feature enhancement. ACM Transactions on Software Engineering and Methodology. https://doi.org/10.1145/3744644
Birjali, M., Kasri, M., & Beni-Hssane, A. (2021). A comprehensive survey on sentiment analysis: Approaches, challenges and trends. Knowledge-Based Systems, 226, 107134. https://doi.org/10.1016/j.knosys.2021.107134
Cunha, W., Canuto, S., Viegas, F., Salles, T., Gomes, C., Mangaravite, V., Resende, E., Rosa, T., Gonçalves, M. A., & Rocha, L. (2020). Extended pre-processing pipeline for text classification: On the role of meta-feature representations, sparsification and selective sampling. Information Processing & Management, 57(4), 102263.
Dabrowski, J., Letier, E., Perini, A., & Susi, A. (2022). Analysing app reviews for software engineering: A systematic literature review. Empirical Software Engineering, 27(2), 1–63. https://doi.org/10.1007/s10664-021-10065-7
Das, S., Mullick, S. S., & Zelinka, I. (2022). On supervised class-imbalanced learning: An updated perspective and some key challenges. IEEE Transactions on Artificial Intelligence, 3(6), 973–993. https://doi.org/10.1109/TAI.2022.3160658
Desuky, A. S., & Hussain, S. (2021). An improved hybrid approach for handling class imbalance problem. Arabian Journal for Science and Engineering, 46(4), 1–12. https://doi.org/10.1007/S13369-021-05347-7
Hardiansyah, D., Aziz, R. Z. A., & Hasibuan, M. A. (2024). The classification method is used for sentiment analysis in My Telkomsel. International Journal of Artificial Intelligence Research, 8(2), 169. https://doi.org/10.29099/ijair.v8i2.1229
Hassan, S. U., Ahamed, J., & Ahmad, K. (2022). Analytics of machine learning-based algorithms for text classification. Sustainable Operations and Computers, 3, 238–248. https://doi.org/10.1016/j.susoc.2022.03.001
Imran, M., Mahmood, A. M., & Qyser, A. A. M. (2014). An empirical experimental evaluation on imbalanced data sets with varied imbalance ratio. International Conference on Computer Communications, 1–7. https://doi.org/10.1109/ICCCT2.2014.7066742
Kaope, C., & Pristyanto, Y. (2023). The effect of class imbalance handling on datasets toward classification algorithm performance. MATRIK: Jurnal Manajemen, Teknik Informatika dan Rekayasa Komputer, 22(2), 227–238. https://doi.org/10.30812/matrik.v22i2.2515
Kholwal, R. (2023). Text-classify: A comprehensive comparative study of logistic regression, random forest, and KNN models for enhanced text classification performance. Zenodo. https://doi.org/10.5281/zenodo.10148008
Liu, T., Wang, C., Huang, K., Liang, P., Zhang, B., Daneva, M., & van Sinderen, M. (2023). RoseMatcher: Identifying the impact of user reviews on app updates. Information and Software Technology, 161, 107261. https://doi.org/10.1016/j.infsof.2023.107261
Liu, Y., Du, G., Yin, C., Zhang, H., & Wang, J. (2024). Clustering-based incremental learning for imbalanced data classification. Knowledge-Based Systems, 292, 111612. https://doi.org/10.1016/j.knosys.2024.111612
Mahmud, Md. S., Bonny, A. J., Saha, U., Jahan, M., Tuna, Z. F., & Marouf, A. Al. (2022). Sentiment analysis from user-generated reviews of ride-sharing mobile applications. International Conference Computing Methodologies and Communication, 738–744. https://doi.org/10.1109/ICCMC53470.2022.9753947
Memon, S. A., Mahar, M. A., Kehar, A., Oad, S., & Ahmed, I. (2025). User experience enhancement through sentiment analysis: A machine learning approach to app reviews. International Journal of Information Systems and Computer Technologies. https://doi.org/10.58325/ijisct.004.02.00131
Mondal, A. S., Zhu, Y., Bhagat, K. K., & Giacaman, N. (2022). Analysing user reviews of interactive educational apps: A sentiment analysis approach. Interactive Learning Environments, 1–18. https://doi.org/10.1080/10494820.2022.2086578
Mortaz, E. (2020). Imbalance accuracy metric for model selection in multi-class imbalance classification problems. Knowledge-Based Systems, 210, 106490. https://doi.org/10.1016/j.knosys.2020.106490
Niaz, N. U., Shahariar, K. M. N., & Patwary, M. J. A. (2022). Class imbalance problems in machine learning: A review of methods and future challenges. Proceedings of the 2nd International Conference on Computing Advancements, 485–490.
Rekha, G., & Tyagi, A. K. (2020). Necessary information to know to solve class imbalance problem: From a user's perspective (pp. 645–658). https://doi.org/10.1007/978-3-030-29407-6_46
Robertson, S. (2004). Understanding inverse document frequency: On theoretical arguments for IDF. Journal of Documentation, 60(5), 503–520. https://doi.org/10.1108/00220410410560582
Solovyeva, E. B., & Abdullah, A. S. (2022). Comparison of different machine learning approaches to text classification. 1427–1430. https://doi.org/10.1109/ElConRus54750.2022.9755806
Tang, Z., Li, W., & Li, Y. (2022). An improved supervised term weighting scheme for text representation and classification. Expert Systems with Applications, 189, 115985. https://doi.org/10.1016/j.eswa.2021.115985
Varghese, L., Pai, R. R., Savitha, G., Girisha, S., & Chetty, N. (2025). Manual annotation based sentiment analysis of user feedback in health and wellness app reviews. Scientific Reports, 15(1), 44766. https://doi.org/10.1038/s41598-025-28799-5
Vluymans, S. (2019). Dealing with imbalanced and weakly labelled data in machine learning using fuzzy and rough set methods (Vol. 807). Springer International Publishing. https://doi.org/10.1007/978-3-030-04663-7
Wang, L., Wang, H., & Fu, G. (2021). Multiple kernel learning with minority oversampling for classifying imbalanced data. IEEE Access, 9, 565–580. https://doi.org/10.1109/ACCESS.2020.3046604
Wankhade, M., Rao, A. C. S., & Kulkarni, C. (2022). A survey on sentiment analysis methods, applications, and challenges. Artificial Intelligence Review, 55(7), 5731–5780. https://doi.org/10.1007/s10462-022-10144-1
Zhao, L., Han, F., Ling, Q., Han, H., Yao, Z., Liu, W., & Zhou, Z. (2025). A survey on class imbalance learning algorithms in complex scenarios. IEEE Access, 13, 180799–180833. https://doi.org/10.1109/access.2025.3618909

Article Statistics

Downloads

Download data is not yet available.

Issue Information

Article Details

Volume:	4
Issue:	2
Year:	2026
Published:	2026-09-10
Pages:	95–105
Section:	Articles

View Full Issue

Additional Information

How to Cite

Handayani, T., & Mardiyati, S. (2026). Machine Learning-Based Sentiment Classification of Reviews from Indonesian Mobile Applications Using TF-IDF. Journal Mobile Technologies (JMS), 4(2), 95–105. https://doi.org/10.59431/jms.v4i2.972