Testing AI chatbots' expertise in translating Japanese texts

The Japan Times tested AI-assisted tools for Japanese-to-English translation, pitting ChatGPT with GPT-4 against Bing, Bard, and DeepL to determine which one best-understood context and translated accurately into natural English. A bilingual senior editor from TJT evaluated the results.

The test involved translating literature, song lyrics, and a speech from Japanese to English. GPT-4 scored highest overall, closely followed by Bard and Bing, although it struggled to translate the subject matter of the literature sample. DeepL consistently scored the lowest. AI chatbots can produce varying results based on factors such as device, timing, and phrasing, but ChatGPT-4 consistently offers the highest quality, according to language expert Tom Gally of the University of Tokyo.