A REPRODUCIBLE FRAMEWORK FOR EVALUATING ROMANIZED SINDHI OUTPUT CONSISTENCY IN CONVERSATIONAL LARGE LANGUAGE MODELS: PILOT EVIDENCE FROM REPEATED PROMPTING

Abstract

The informal digital communication is in the form of Romanized Sindhi which is not well represented in the benchmarks of natural language processing. This study suggests an evaluation method which is clear and concise to check whether a conversational large language model retains Romanized Sindhi in repeated prompting or not and tests it on the only source record recovered, one instruction-free sentence, repeated 10 times. Coding was done with the use of non-exclusive indicators for Sindhi script, Romanized Sindhi, Urdu script, Romanized Urdu and English. The main outcome measure was retention of Romanized-Sindhi, defined as having any Romanized-Sindhi span in a response. Romanized Sindhi was used in 6 out of 10 responses (60%; Wilson 95% confidence interval 31.3%-83.2%) while Sindhi script was used in 7 responses. Urdu, Urdu Romanized and English were used once each. There were 5 different language/script combinations observed. The results indicate that there is variability in output format for the recorded interface condition, but not for semantic accuracy, learning or overall model performa nce. Limited interpretation due to single prompt, no model and decoding metadata, uncertain session independence and duplicated screenshot assets and no independent native-speaker annotation. The main difference is the inclusion of a reproducible evaluation specification that differentiates between script retention and semantic adequacy and retains the original of the evidence in an attached annexure. A multi-sentence benchmark, controlled model snapshots, machine-readable outputs and blinded annotation by qualified Sindhi speakers are required for a full-scale confirmatory study.