Search for collections on FTS Digilib

When Similar Models Disagree: Model Multiplicity and Geographic Robustness in School Meal Utilization Prediction

Zenklinov, Amanatullah Pandu and Andriano, Andriano and Saputra, Wisnowan Hendy and Ariani, Marcella and Nisa, Nabilla Rida Tri and Arisanty, Sabrina Julietta (2026) When Similar Models Disagree: Model Multiplicity and Geographic Robustness in School Meal Utilization Prediction. Journal of Computing Theories and Applications, 4 (2). pp. 769-789. ISSN 3024-9104

[thumbnail of 18209-Article Text-64721-3-10-20261110.pdf] Text
18209-Article Text-64721-3-10-20261110.pdf - Published Version
Available under License Creative Commons Attribution.

Download (825kB)

Abstract

Competing predictive models can achieve similar aggregate performance while producing different classifications and downstream conclusions for the same individuals. This model disagreement raises an important question about whether conventional performance metrics adequately capture the reliability of model-based analyses. This study investigates model-choice disagreement across multiple analytical layers using voluntary-response survey data from 8,802 respondents who reported receiving meals through Indonesia’s Free Nutritious Meals program. Logistic Regression (LR), Random Forest, and XGBoost were evaluated using a nested school-disjoint validation framework. All three models were compared in terms of discrimination and probability calibration, while the competitive models were further examined through high-risk classification, predictive attribution, model-based scenario analysis, and province-holdout robustness. LR and XGBoost achieved nearly identical PR-AUC values (0.7270 and 0.7277, respectively), with similarly close pooled PR-AUC under province holdout (0.7288 and 0.7285). Despite this aggregate similarity, the models assigned different high-risk classifications to 5.41% of respondents. Their scenario analyses also produced different threshold-crossing rates: 8.87% of eligible respondents under LR (382 of 4,307) and 6.66% under XGBoost. Random Forest exhibited weaker probability calibration in the fitted configuration and was not retained for downstream comparison. Overall, the findings reveal that model agreement at the population level does not necessarily extend to individual classifications and scenario outputs. LR was selected as the reference model through documented judgment considering predictive performance, calibration, robustness, parsimony, and interpretability. These results demonstrate the importance of evaluating model-choice robustness beyond conventional predictive performance metrics, particularly when model outputs inform downstream interpretation or classification.

Item Type: Article
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Depositing User: dl fts
Date Deposited: 11 Nov 2026 06:31
Last Modified: 11 Nov 2026 06:31
URI: https://dl.futuretechsci.org/id/eprint/234

Actions (login required)

View Item
View Item