An Argument-Based Validation Framework for Automated Scoring of Higher-Order Thinking in Arabic Essays
Penulis
Automated essay scoring (AES) for Arabic has advanced quickly, but most studies report agreement statistics without stating what an automated score is meant to show or which uses it can support. The gap is sharper for higher-order thinking skills (HOTS), where good answers legitimately diverge. This article develops an argument-based validation framework for Taḥkīm, a semantic-similarity engine designed to score Arabic essays written by Indonesian undergraduates at the analysing, evaluating, and creating levels of the revised Bloom taxonomy. Using a conceptual design approach, the study specifies the intended interpretation and uses of Taḥkīm scores, derives an interpretation-use argument from Kane's framework and the Williamson-Xi-Breyer criteria, and synthesizes Arabic AES research to identify threats that arise in Arabic. The framework comprises four inferences (scoring, generalization, extrapolation, and decision), each with a warrant, testable assumptions, required evidence, and a decision rule fixed before data collection. It identifies eight threats, among them orthographic normalization, colloquial vocabulary, incomplete anchor coverage for divergent creating-level responses, and the confounding of reasoning with language proficiency, and pairs each with a design response and an empirical check. A staged evaluation plan sets the minimum evidence for each permitted use, from formative feedback to first-pass scoring under lecturer review, and excludes unsupervised grading until all four inferences hold. The framework gives Arabic departments in Islamic higher education a procedure for stating and testing claims about automated scores before adopting them, and it is the protocol for the empirical validation of Taḥkīm.
Abdul-Mageed, M., Elmadany, A., & Nagoudi, E. M. B. (2021). ARBERT & MARBERT: Deep bidirectional transformers for Arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (pp. 7088–7105). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.551.
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman.
Antoun, W., Baly, F., & Hajj, H. (2020). AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools (pp. 9–15). European Language Resources Association.
Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater V.2. The Journal of Technology, Learning and Assessment, 4(3), 1–30.
Batalipu, A., Miolo, M. I., & Sarif, S. (2026). Learning Arabic in heterogeneous classrooms: Experiences of non-pesantren students in Indonesian Islamic higher education. Al-Ta'rib: Jurnal Ilmiah Program Studi Pendidikan Bahasa Arab UIN Palangka Raya, 14(1), 81–96. https://doi.org/10.23971/altarib.v14i1.11341.
Beseiso, M., & Alzahrani, S. (2020). An empirical analysis of BERT embedding for automated essay scoring. International Journal of Advanced Computer Science and Applications, 11(10), 204–210. https://doi.org/10.14569/IJACSA.2020.0111027.
Beseiso, M., Alzubi, O. A., & Rashaideh, H. (2021). A novel automated essay scoring approach for reliable higher educational assessments. Journal of Computing in Higher Education, 33(3), 727–746. https://doi.org/10.1007/s12528-021-09283-1.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220. https://doi.org/10.1037/h0026256.
Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: A practical guide to Kane's framework. Medical Education, 49(6), 560–575. https://doi.org/10.1111/medu.12678.
Creswell, J. W., & Plano Clark, V. L. (2018). Designing and conducting mixed methods research (3rd ed.). SAGE.
Cumming, A. (1989). Writing expertise and second-language proficiency. Language Learning, 39(1), 81–135. https://doi.org/10.1111/j.1467-1770.1989.tb00592.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423.
Direktorat Jenderal Kependudukan dan Pencatatan Sipil. (2026). Data kependudukan bersih semester II tahun 2025 [Data set]. Kementerian Dalam Negeri Republik Indonesia.
Dong, F., Zhang, Y., & Yang, J. (2017). Attention-based recurrent convolutional neural network for automatic essay scoring. In Proceedings of the 21st Conference on Computational Natural Language Learning (pp. 153–162). Association for Computational Linguistics. https://doi.org/10.18653/v1/K17-1017.
Elhaddadi, C., Zeroual, I., & El Kah, A. (2025). Automatic Arabic essays scoring: A scoping review. In B. Hdioud & S. L. Aouragh (Eds.), Arabic language processing: From theory to practice (Communications in Computer and Information Science, Vol. 2339, pp. 45–60). Springer. https://doi.org/10.1007/978-3-031-79164-2_4.
Gaheen, M. M., ElEraky, R. M., & Ewees, A. A. (2021). Automated students Arabic essay scoring using trained neural network by e-Jaya optimization to support personalized system of instruction. Education and Information Technologies, 26(1), 1165–1181. https://doi.org/10.1007/s10639-020-10300-6.
Ghazawi, R., & Simpson, E. (2024). Automated essay scoring in Arabic: A dataset and analysis of a BERT-based system (arXiv:2407.11212). arXiv. https://doi.org/10.48550/arXiv.2407.11212.
Ghazawi, R., & Simpson, E. (2025). How well can LLMs grade essays in Arabic? Computers and Education: Artificial Intelligence, 9, Article 100449. https://doi.org/10.1016/j.caeai.2025.100449.
Kane, M. T. (1992). An argument-based approach to validity. Psychological Bulletin, 112(3), 527–535. https://doi.org/10.1037/0033-2909.112.3.527.
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000.
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103, Article 102274. https://doi.org/10.1016/j.lindif.2023.102274.
Ke, Z., & Ng, V. (2019). Automated essay scoring: A survey of the state of the art. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (pp. 6300–6308). IJCAI. https://doi.org/10.24963/ijcai.2019/879.
Kementerian Agama Republik Indonesia. (2019). Keputusan Menteri Agama Nomor 183 Tahun 2019 tentang Kurikulum Pendidikan Agama Islam dan Bahasa Arab pada Madrasah. Kementerian Agama Republik Indonesia.
Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory Into Practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. https://doi.org/10.2307/2529310.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741.
Mizumoto, A., & Eguchi, M. (2023). Exploring the potential of using an AI language model for automated essay scoring. Research Methods in Applied Linguistics, 2(2), Article 100050. https://doi.org/10.1016/j.rmal.2023.100050.
Nazaretsky, T., Mejia-Domenzain, P., Swamy, V., Frej, J., & Käser, T. (2026). Who gives feedback matters: Student biases towards human and AI-generated formative feedback. Journal of Computer Assisted Learning, 42(1), Article e70153. https://doi.org/10.1111/jcal.70153.
Pack, A., Barrett, A., & Escalante, J. (2024). Large language models and automated essay scoring of English language learner writing: Insights into validity and reliability. Computers and Education: Artificial Intelligence, 6, Article 100234. https://doi.org/10.1016/j.caeai.2024.100234.
Page, E. B. (1966). The imminence of grading essays by computer. The Phi Delta Kappan, 47(5), 238–243.
Panadero, E. (2017). A review of self-regulated learning: Six models and four directions for research. Frontiers in Psychology, 8, Article 422. https://doi.org/10.3389/fpsyg.2017.00422.
Pew Research Center. (2025, June 9). How the global religious landscape changed from 2010 to 2020. https://www.pewresearch.org/religion/2025/06/09/how-the-global-religious-landscape-changed-from-2010-to-2020/.
Putri, R. K. A., Kusaeri, K., & Suparto, S. (2025). Deep learning in automated essay scoring for Islamic education: A systematic review. Online Learning in Educational Research, 5(2), 319–338. https://doi.org/10.58524/oler.v5i2.753.
Ramesh, D., & Sanampudi, S. K. (2022). An automated essay scoring systems: A systematic literature review. Artificial Intelligence Review, 55(3), 2495–2527. https://doi.org/10.1007/s10462-021-10068-2.
Ramineni, C., & Williamson, D. M. (2013). Automated essay scoring: Psychometric guidelines and practices. Assessing Writing, 18(1), 25–39. https://doi.org/10.1016/j.asw.2012.10.004.
Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (pp. 3982–3992). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1410.
Shehab, A., Faroun, M., & Rashad, M. (2018). An automatic Arabic essay grading system based on text similarity algorithms. International Journal of Advanced Computer Science and Applications, 9(3), 263–268. https://doi.org/10.14569/IJACSA.2018.090337.
Susilawati, E., Siregar, E. S., Ekalestari, S., & Harahap, E. S. (2025). Development of a digital assessment model for automated short essay scoring in Islamic education study program. Fitrah: Journal of Islamic Education, 6(1), 208–223. https://doi.org/10.53802/fitrah.v6i1.1195.
Taghipour, K., & Ng, H. T. (2016). A neural approach to automated essay scoring. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 1882–1891). Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1193.
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540.
Venkatesh, V., Thong, J. Y. L., & Xu, X. (2012). Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology. MIS Quarterly, 36(1), 157–178. https://doi.org/10.2307/41410412.
Wang, Y., Wang, C., Li, R., & Lin, H. (2022). On the use of BERT for automated essay scoring: Joint learning of multi-scale essay representation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (pp. 3416–3425). Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.naacl-main.249.
Weigle, S. C. (2002). Assessing writing. Cambridge University Press. https://doi.org/10.1017/CBO9780511732997.
Williamson, D. M., Xi, X., & Breyer, F. J. (2012). A framework for evaluation and use of automated scoring. Educational Measurement: Issues and Practice, 31(1), 2–13. https://doi.org/10.1111/j.1745-3992.2011.00223.x.
Xi, X. (2010). Automated scoring and feedback systems: Where are we and where are we heading? Language Testing, 27(3), 291–300. https://doi.org/10.1177/0265532210364643.
Xue, L., Rashid, A. M., & Ouyang, S. (2024). The unified theory of acceptance and use of technology (UTAUT) in higher education: A systematic review. SAGE Open, 14(1). https://doi.org/10.1177/21582440241229570.
Yancey, K. P., Laflair, G., Verardi, A., & Burstein, J. (2023). Rating short L2 essays on the CEFR scale with GPT-4. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (pp. 576–584). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.bea-1.49.
Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2), 64–70. https://doi.org/10.1207/s15430421tip4102_2.
Bilah sisi Artikel
Hak Cipta (c) 2026 Rozaanah Rozaanah, Slamet Riyadi
Artikel ini berlisensi Creative Commons Attribution 4.0 International License.