To test the system, Google had human raters evaluate translations on a scale from 0 to 6. Translating from English to Spanish, the new Google tool’s translation was rated an average of 5.43; human translators earned an average of 5.5. For Chinese to English, the only public-facing option that currently utilizes the new system, Google Translate was rated an average of 4.3 while human translators got 4.6.
Overall, across all three languages, Google said its new tool is 60 percent more accurate than the old Google Translate tool, which used phrase-based machine translation, or PBMT. “With the previous PBMT model, when we translate a sentence from one language to another, we would translate one word or a phrase in the source sentence at a time, then re-order the words in the correct grammar of the target language,” said Quoc Le, a Google researcher who worked on the project. “The complication is language has a lot of ambiguity. In our new GNMT system, we treat a whole sentence as a unit, and translate [the words] in a group.”
The complexity of a translation machine that can digest entire phrases rather than rely on word-by-word translation are somewhat lost even on the researchers themselves. In an interview with MIT Technology review, Le called the new translator “unsettling. But we’ve tested it in a lot of places and it just works.”
Despite its improved accuracy, the GNMT model still mistranslates rare terms and occasionally drops words. And it hasn’t acquired any common sense. Given the sentence “The trophy cannot fit in the cabinet because it’s too big,” the model could mistranslate because it doesn’t know which “it” is the one that’s too big. “GNMT doesn’t have a model of how the world actually works yet,” said Le.
While the tool is only being offered for one language pair now, Google said it plans to roll out the new system to more of the 10,000 language pairs that Google Translate currently supports. “We hope this research increases the ability for people around the world to communicate with others regardless of what languages they speak,” said Le.