Local PDF.pdf

Researchers detected that the WMT 2020 Czech reference translation was created by post-editing a machine translation (MT) system, despite instructions to translate from scratch. Two methods based on automatic metrics, BLEU difference between suspected MT and another MT, and BLEU difference using additional references, revealed suspicion of post-editing. Manual analysis confirmed this, finding concrete proofs of post-editing in particular sentences. A typology of post-editing changes was presented, classifying typical errors or changes made by the post-editor or adopted from the MT. The study used the English-Czech part of WMT2020 news test set, with 130 documents and 1418 segments, and hired professional translators to create additional references. The results showed that the official reference translation, REF1, was actually post-edited versions of ONLINE-B, an MT system, and that BLEU evaluated one of the MT systems as better than REF1. The study highlights the importance of detecting post-editing in reference translations to ensure reliable evaluation of MT systems.