Collecting automatically Sino-Nom variants to improve transliteration from Nom scripts into Vietnamese national scripts

Abstract

Throughout the millennium of formation and development, the Sino-Nom scripts encompass a multitude of variants for the same character. This phenomenon has posed many challenges for automatic transliteration systems from Sino-Nom scripts to Vietnamese National scripts as the training corpus lacks comprehensive coverage of these variant characters. To resolve this issue, our research proposes the use of computer software to automatically collect Sino-Nom variants from reputable websites. The implementation method involves the following steps  collecting automatically Sino-Nom variants from the internet, building a tool for automated data extraction and applying statistical methods to verify the quality of the collected data repository. The results of our proposed method demonstrate the substantial expansion of the Sino-Nom dictionary with a multitude of new variants, thereby enhancing the accuracy of automatic transliteration from Sino-Nom scripts to national scripts.