FUSE: A New Metric for Evaluating Machine Translation in Indigenous Languages

Imagine you are trying to translate a sentence from English into a language that has very different rules for words and sounds. Traditional methods like BLEU or ChrF might fail because they rely on exact word matches or simple overlaps. But what if the language has words that change form based on context, or sounds that are hard to represent with standard text? This is the challenge FUSE tackles, especially for indigenous languages like Bribri, Guarani, and Nahuatl. These languages often have rich morphological structures and phonetic variations that make them tricky for standard evaluation.

FUSE works by combining multiple layers of analysis. It does not just look at words. It also listens to how they sound using phonetic algorithms like Metaphone and Soundex, checks how similar they are in meaning using semantic embeddings from models like LaBSE, and even measures how closely they match on a character level with fuzzy token matching. This multi-angle approach helps it understand translations in a way that is closer to how humans judge language by looking at meaning, sound, and structure together.

The results from testing FUSE show it is highly effective. It achieved correlation scores of 0.85 Pearson and 0.80 Spearman with human judgments, significantly outperforming older metrics like BLEU, ChrF, and TER across all tested language pairs. This means that when people rated translations as good or bad, FUSE’s scores aligned closely with those human ratings, showing it understands quality in a more human-like way.

For anyone working with machine translation, especially for less-resourced languages, FUSE represents a promising step forward. It highlights that evaluation metrics need to be as sophisticated as the languages they are assessing. By integrating multiple linguistic dimensions—lexical, phonetic, semantic, and structural—FUSE ensures that translations are not just correct word-for-word, but also sound natural and convey the intended meaning. This is crucial for applications in education, cultural preservation, and technology development for diverse linguistic communities.