Human Evaluation of Machine Translation Output

This is another post from Juan Rowda, this time co-written with his colleague Olga Pospelova, that provides useful information on this subject. Human evaluation of MT output is key to understanding several fundamental questions about MT technology including:

1. Is the MT engine improving?
   
2. Do the automated metrics (which MT systems developers use to guide development strategy) correlate with independent human judgments of the same MT output?
   
3. How difficult/easy is the PEMT output task?
   
4. What is the fair and reasonable PEMT rate?
   

The difficulty with human evaluation of MT output is closely related to the difficulty associated with any discussion of translation quality in general in the industry. It is challenging to come up with an approach that is consistent over time and across different people. Being scientifically objective is particularly a challenge. So, like much else in MT, estimates have to be made, and some rigor needs to be applied to ensure consistency over time and across evaluators. I will add some additional thoughts and references after Juan's post below, to provide a view on some new ways being explored to approach this issue of how humans could evaluate MT output in an objective way. 
 

-------------- 

 
Machine translation (MT) output evaluation is essential in machine translation development. This is key to determining the effectiveness of the existing MT system, estimating the level of required postediting, negotiating the price, and setting reasonable expectations. As we discussed in our article on quality estimation, machine translation output can be evaluated automatically, using methods like BLEU and NIST, or by human judges. The automatic metrics use one or more human reference translations, which are considered the gold standard of translation quality. The difficulty lies in the fact that there may be many alternative correct translations for a single source segment.

Human evaluation, however, also has a number of disadvantages. Primarily because it is a costly and time-consuming process. Human judgment is also subjective in nature, so it is difficult to achieve a high level of intra-rater (consistency of the same human judge) and inter-rater (consistency across multiple judges) agreement. In addition, there are no standardized metrics and approaches to human evaluation. 

Let us explore the most commonly used types of human evaluation:

 

Rating :-

Judges rate translations based on a predetermined scale. For example, a scale from 1 to 5 can be used, where 1 is the lowest and 5 is the highest score. One of the challenges of this approach is establishing a clear description of each value in the scale and the exact differences between the levels of quality. Even if human judges have explicit evaluation guidelines, they still find it difficult to assign numerical values to the quality of the translation (Koehn & Monz, 2006).

The two main dimensions or metrics used in this type of evaluation are adequacy and fluency.

Adequacy , according to the Linguistic Data Consortium, is defined as "how much of the meaning expressed in the gold-standard translation or source is also expressed in the target translation." 

The annotators must be bilingual in both the source and target language in order to judge whether the information is preserved across translation.

A typical scale used to measure accuracy is based on the question "How much meaning is preserved?"
5: all meaning
4: most meaning
3: some meaning
2: little meaning
1: none 

Fluency refers to the target only, without taking the source into account; the main evaluation criteria are grammar, spelling, choice of words, and style.

A typical scale used to measure fluency is based on the question "Is the language in the output fluent?"
5: flawless
4: good
3: non-native
2: disfluent
1: incomprehensible 

Ranking:-

Judges are presented with two or more translations (usually from different MT systems) and are required to choose the best option. This task can be confusing when the ranked segments are nearly identical or contain difficult-to-compare errors. The judges must decide which errors have a greater impact on the quality of the translation (Denkowski & Lavie, 2010). On the other hand, it is often easier for human judges to rank systems than to assign absolute scores (Vilar et al., 2007). This is because it is difficult to quantify the quality of the translation. 

Error Analysis :-

Human judges identify and classify errors in MT output. Classification of errors might depend on the specific language and content type. Some examples of error classes are "missing words", "incorrect word order", "added words", "wrong agreement", "wrong part of speech", and so on. It is useful to have reference translations in order to classify errors; however, as mentioned above, there may be several correct ways to translate the same source segment. Accordingly, reference translations should be used with care.

When evaluating the quality of eBay MT systems, we use all the aforementioned methods. However, our metrics can vary in the provision of micro-level details about some areas specific to eBay content. As a result, one of the evaluation criteria is to identify whether brand names and product names (the main noun or noun phrase identifying an item) were translated correctly. This information can help in identifying the problem areas of MT and focusing on the enhancement of that particular area.

Some types of human evaluation, such as error analysis, can only be conducted by professional linguists, while other types of judgment can be performed by annotators who are not linguistically trained.

Is there a way to cut the cost of human evaluation? Yes, but unfortunately, low-budget crowd-sourcing evaluations tend to produce unreliable results. How then, can we save money without compromising the validity of our findings?

• Start with a pilot test - a process of trying out your evaluation on a small data set. This can reveal critical flaws in your metrics, such as ambiguous questions or instructions.
• Monitor response patterns to remove judges whose answers are outside the expected range.
• Use dynamic judgments - a feature that allows fewer judgments on the segments where annotators agree, and more judgments on segments with a high inter-rater disagreement.
• Use professional judgments that are randomly inserted throughout your evaluation job. Pre-labeled professional judgments will allow for the removal of judges with poor performance.

Human evaluation of machine translation quality is still very important, even though there is no clear consensus on the best method. It is a key element in the development of machine translation systems, as automatic metrics are validated through correlation with human judgment. 

Juan Rowda
Staff MT Language Specialist, eBay
https://www.linkedin.com/in/juan-martín-fernández-rowda-b238915 
 
Juan is a certified localization professional working in the localization industry since 2003. He joined eBay in 2014. Before that, he worked as translator/editor for several years, managed and trained a team of +10 translators specialized in IT, and also worked as a localization engineer for some time. He first started working with MT in 2006. Juan helped to localize quite a few major video games, as well. 
 
He was also a professional CAT tool trainer and taught courses on localization.
Juan holds a BA in technical, scientific, legal, and literary translation. 
 

Olga Pospelova
Localization Program Manager, eBay
 

Olga Pospelova is a Localization Program Manager at eBay where she has been leading linguistic support to Human Language Technology Initiative. The broad spectrum of services includes human evaluation of MT output, creating testing and training data for MT systems, semantic annotation, etc. Prior to joining eBay, Olga worked as an Assistant Professor of Russian at the Defense Language Institute Foreign Language Center. Olga holds an MA in Linguistics from San Jose State University. 

------------------------------------------

So in this quest for evaluation objectivity, there are a number of recent initiatives that are useful. For those who are considering the use of Error Analysis, TAUS has attempted to provide a comprehensive error typology scheme together with some tools. These errors can then be used to develop a scoring system that will weight errors differently and does help to develop a more objective approach to a discussion on MT quality. 

 
This presentation by Alon Lavie provides more details on the various approaches that are being tried and includes some information on the Edit Distance measurements which can only be done after a segment is post-edited. 
 
This study is interesting since it provides evidence that monolingual editors can perform quite efficiently and consistently provided they are only shown Target data i.e. without the source. In fact, they do this more efficiently and consistently than bilinguals who are shown both source and target. I also assume that an MT system has to reach a certain level of quality before this is true.


Eye-tracking studies are another way to get objective data and initial studies provide some useful information for MT practitioners. The premise is very simple: Lower quality MT output slows down an editor who has to look much more carefully at the output to correct it. It also shows that some errors cause more of a drag than others. In both charts shown below the shorter the bar the better. Three MT systems are compared here and it is clear that the Comp MT system has a lower frequency of most error types.

The results from the eye fixation times are consistent and correlated with automatic metrics like BLEU. In the chart below we see that Word Order errors cause a longer gaze time than other errors. This is useful information for productivity analysts who can use it to determine problem areas or determine the suitability of MT output for comprehension or post-editing.

The basic questions being addressed by all these efforts are all inching us forward slowly and perhaps some readers can shed light on new approaches to answer them. Adaptive MT and the Fairtrade initiative are also different ways to address the same fundamental issues. My vote so far is for Adaptive MT as the most promising approach for professional business translation, even though it is not strictly focused on static quality measurement.

De Domo Sua – The Rationale for Hiring a Consultant when Working with MT

I have always found Luigi Muzii's aka Il barbaro (The Barbarian) opinions on the "translation industry" interesting and have often felt that his blog never got the attention it deserved. Possibly because he often wrote with frequent references to Roman historical precedence, in what I have been told is an Italian scholarly style, and perhaps because English is not his preferred language to communicate complex thoughts. 

 

His willingness to state the obvious (to common sense) sometimes makes him unpopular, especially with the MT naysayers e.g. "Moreover, translation data - i.e. project data - has a limited lifespan and, at some point in time, it becomes outdated, possibly inaccurate, and definitely irrelevant." I think that many in the "translation industry" still fail to realize that the bulk of the material that they translate;  i.e. the manuals, documentation and generic business content on websites focused on the wonder and excellence of corporate goods and services, become less relevant, and less valuable with each passing day no matter how well it has been translated. They clearly don't like to hear someone suggest that this might be so. And it is only natural that the buyers of business translation services might ask: "Is there a cheaper way to do this?" since they are acutely aware of the rapidly declining relevance and low value of a lot of the corporate content they produce.

His willingness to raise fundamental questions I think makes his voice worthy of attention, for me at least. Anyway, I am glad that he sent me this brief overview which I am told will be expanded in future, and I hope that he will be back in future on this blog with other observations on the business of translation related to MT or not. 
 

Judge a man by his questions, not his answers … Voltaire

----------------

On his house [*]

[*] De domo sua is a famous speech that Marcus Tullius Cicero gave in theRoman Senate in 57 B.C. against tribune Publius Clodius Pulcher, to have his house rebuilt at public cost, after it was destroyed by Clodius' supporters. 

"On his house" is the equivalent of "De domo sua" in English, which we largely use in Italy to label a cause one pleads for his own benefit. In this case, "domus mea" (my home) is the rationale for hiring a consultant. 

Implementing machine translation (MT) effectively is not an easy task for a translation agency, or even for a large translation buyer whose core business is not translation. Still, with 99,9% of translations performed daily by machines, it is more and more perfectly reasonable that businesses are eager to have more and more of their content translated, quickly and cost-effectively using the technology that is available. 

Speed and cost-effectiveness are crucial to attain competitive advantage in global market competition, and we should understand that not all content is equal in terms of how it should and can be properly translated. And not all content deserves the same TEP (Translate>Edit>Proof) consideration and production process. 

Most enterprises, especially SMEs seeking a foothold in international markets, see machine translation as a viable solution, but they do not have the necessary knowledge and skills needed to deal with the challenging effort of implementing MT properly. Most logically, then, a SME would seek help from translation agency vendors assuming they would know how to use and implement MT technology. 

Unfortunately, most translation agencies often do not have these skills either. Considering that the translation technology market - which includes MT - is noticeably smaller than the broader linguistic services market, and is somewhat under-served in terms of availability of competent service providers, more and more language service vendors unfortunately have been adding MT consulting to their service offerings as a means to generate new forms of revenue. This can only result in a situation somewhat like that shown below. 

The following very brief tips are intended as basic practical guidelines for enterprises interested in taking advantage of MT. 

Tip #1

Hire an independent consultant. If you are a language service vendor, being supported by an independent advisor will spare you the cost and the hassle of an internal department while offering your customers the peace of mind of an unbiased opinion and professional help. 

Tip #2

Either ask for or run a preliminary analysis. Through interviews with staff and examination of the current modus operandi (MO), an independent consultant can help you objectively assess your processes and facilities in view of the potential implementation and integration of a machine translation platform, as well as provide guidance on the possibility of the provision of any ancillary services. 

Tip #3

Either ask for or draft a report that provides the following:

• Review of your goals,
• The outcomes of the preliminary analysis
• Appraisal of the suitability of current modus operandi (MO) for machine translation, and
• An outline of an exploratory program.

Use flowcharts and/or sequence diagrams to draw your modus operandi (MO), starting from future MO (FMO), then present MO (PMO) and finally transitional MO (TMO.) By overlapping the three diagrams you should be able to track the differences and be able to make adjustments to streamline the transitional process and ensure a smooth update of the existing processes. 

Tip #4

Write down the requirements that have been collected during the interviews for the preliminary analysis. Prepare a grid with the technical specifications (TS) and the statement of work (SoW) detailing what is to be done. 

Tip #5

Write a clear template for a request for proposals (RFP) including the TS and the SoW and send it out to a group of selected MT vendor candidates, asking them to fill in the grid. 

Tip #6

When hiring the advisor, consider that the advisor should have comprehensive insight and understanding of the general MT market and its players to help identify the best candidates. 

Do not forget 

 

Discuss the report and the proposals with the advisor and the customer, if you are a translation agency. Provide a program, a configuration plan and a training/recruiting program including all tasks, from benchmarking to data qualification and preparation, from vendor selections to proposal evaluation, from negotiations to implementation, from testing to training. 

Luigi Muzii has been in the "translation business" since 1982 and has been a business consultant since 2002, in the translation and localization industry through his firm . He focuses on helping customers choose and implement best-suited technologies and redesign their business processes for the greatest effectiveness of translation and localization related work.

This link provides access to his other blog posts.

Comparing MT Based Translation Errors with Human Translation Errors

This is an important subject and one that needs ongoing examination and continuing study to get to real insight and find greater process efficiency. I hope that this post by Silvio Picinini will trigger further discussion and I also invite others who have information, and opinions to share on this subject, to step forward. In my observation of MT output over time, I have seen that MT produces a greater number of actual errors, but the types of errors most often generated by MT are easy to spot and clean up. Unlike the incorrect or inconsistent terminology and sometimes misunderstood source errors, that may often be hidden in the clear grammar and flow of a human translation.  These human errors are much harder to find, and not as easy to correct without multiple stages of review by independent reviewers. 

-----

It is common to see a focus on the errors that machine translation makes. But maybe there are also strengths in MT that can benefit translators and customers. So I wrote an article in Multilingual magazine comparing errors made by MT and human translators. The article is below, reproduced here with permission. 

Please find the original article published on Multilingual magazine, July-August 2016, “Errors in MT and human translation”, pg. 46-50.

Post-editing could be defined as the correction of a translation suggested by a machine. When a translator receives a suggested translation from Google, Bing or another machine translation (MT) engine, the translator is working as a post-editor of that suggestion. 

If instead the translator receives a suggestion from a translation memory (TM), the suggestion for that segment was created by a human translator in the past. If there is no suggestion from a TM, human translation is the creation of a translation by a human. If we now assume that a human translator and an MT engine “think” differently about the translation of a sentence, we can explore how a human translator makes different errors compared to a statistical machine translation process. If we make translators and post-editors more aware of the types of errors that they will find, they will be able to improve their translation quality. 

Inconsistency 

A human translator will likely go through a text and consistently translate the word “employee” with the same words in the target language, keeping his or her favorite word in mind and using it all the time in the translation. The MT engine, on the other hand, has no commitment to consistency. Statistical machine translation (SMT) is a popularity contest. So, in one sentence, the translation for “employee” may be the usual one. But the corpus used to train the engine may have come from governments, or from the European Parliament, for example. And governments may like to call employees “public servants.” If that is popular enough, the MT may choose this translation for some sentences. The translation will be inconsistent and could look something like “Company employees may only park on the east parking lot. If that is full, the public servant may park on the west parking lot.” 

Thus, MT is more likely than humans to translate inconsistently. 

However, here are two more thoughts on this. First, humans create inconsistencies also. If the TM contains translations made by several translators with different preferences for words, or contains translations created over a long period of time, chances are that there are inconsistencies in the TMs. Those inconsistencies will be accepted by the human translator. 

Second, we are not weighing glossaries to one side or the other. The human translators could follow the glossary, or run glossary consistency checks and their translations would be consistent. But the same applies to post-editing. 

Mistranslations 

Many words have more than one meaning, and are defined as polysemous. The word “holder,” for example, may mean a person who owns a credit card (as in credit card holder), but it may also be a stand designed to hold an object, such as a plate holder. If you are thinking of items on eBay, you are most likely expecting the translation that means plate holder. A human translator will easily know the correct translation. The machine, however, may have a lot of training data related to finance or laws, and the most popular translation there could be the person. The MT could choose this most popular meaning and the result could be “fashion sponge holder with suction cup” translated as “fashion sponge card holder with suction cup.” 

Thus, MT is more likely than humans to make mistranslation substitutions. 

Words that are not to be translated 

For a human translator, it is easy to know that Guess, Coach and Old Navy are brands, and therefore should not be translated into most target languages. As you know, SMT is a popularity contest, and the very common word “guess” likely appears quite frequently in the corpus that trained the SMT engine. The consequence — you guessed it — is that the MT is likely to translate the word instead of leaving it untouched because it is a brand.
This may happen with product names as well. Air Jordan sneakers could have the word Air translated. It could happen with the brand Apple versus the fruit apple, but since it is a popularity contest, the fruit may now be left untranslated instead of the iPhone having a fruit to go with it. 

Thus, MT is more likely than humans to translate words that are not supposed to be translated.

Untranslated Words 

The MT leaves out of vocabulary (oov) words untranslated, and humans will translate them. This will favor humans depending on how many oov words are present in the content to be translated. If the content is specific and different from the corpus used to train the engine, it is more likely that some words will not be known by the MT engine. But if the MT engine is well trained with the same kind of subject that is being translated, then the MT engine will minimize the number of untranslated words. On the other hand, MT takes the collective opinion into account. It may not translate words that are now commonly used untranslated, while a translator could be more traditional or old-fashioned and would translate the word. Would you translate “player” in “CD player” in your language? The word “player” used to be translated a few decades ago, but the usage changed and the English “CD player” is common now in many languages. The MT will learn from the corpus the most current and frequent usage, and may do better than a human translator. Overall, this issue still slightly favors the human side.
Thus, MT will leave more wrongly untranslated words than humans. 

Gender and Number Agreement 

The MT engine may learn from the corpus the correct translation for “beautiful skirt,” and skirt is a feminine word in many languages. However, the first time the source contains the combination “beautiful barometer,” it will pick from what it knows and it may translate beautiful as a feminine word. If barometer is masculine in the target language, this creates an error of gender agreement. The MT is more likely to make this error than a human translator, who intuitively knows the gender of objects. The same applies to singular and plural. English uses invariant adjectives for both, as in “beautiful skirt” and “beautiful skirts.” Thus, the MT engine may pick the singular translation for the adjective next to a plural noun. The MT is more likely to make a number agreement error than a human translator, who knows when singular or plural is needed. 

Thus, MT will make more grammar errors than humans. 

So far we have seen several examples of situations where humans translate better than MT engines. Now we will look at how a “self-correcting” property of MT, created from the popularity of a translation, can often do a better job than humans. A statistical MT engine can be seen as a “popularity contest” where the translation that is suggested is the most popular translation for a word or group of words present in the “knowledge” (corpus) that trained the MT engine.

Spelling

There are two types of spelling errors: the ones that create words that don’t exist (and can be caught by a spellchecker) and the errors that turn a word into another existing word. You may have turned “from” into “form” and “quite” into “quiet.” The first type, a pure spelling error made by MT would require that you have in the corpus more instances of the error than of the correction. Can you imagine a corpus that contains “porduct” 33 times and “product” only 32 times? So MT almost never makes a spelling error of this kind.

For the second type, humans turn words into other words, and the spellchecker will miss it. The MT engine will not make this error because it is not likely that the corpus will contain the misspelled word more frequently than the correct word for that context. This would require having “I am traveling form San Francisco to Los Angeles” more frequently in your corpus than you would have “I am traveling from San Francisco to Los Angeles” and which one is more likely to be popular in a corpus? The correct one. This is why MT will almost never make this kind of spelling error, while it is easy for a human translator to do so.

Thus, humans are more likely than MT to make spelling errors of any kind. 

False Friends 

False friends are words that look similar to a word in a different language, but mean something different. One example is the word “actual,” which means “real” in English. In Portuguese, the word atual means current or as of this moment. So a presentation mentioning “actual numbers” could be translated as “current numbers,” seriously changing the meaning and causing confusion. A human translator may make this error, but the MT would require the wrong translation for “actual” to be more popular in the corpus than the correct translation. You would need “actual numbers” to be translated more frequently as “current numbers” than as “real numbers.” Do you think this would happen? No, and that is why MT almost never falls for a false friend, while a human translator falls for it occasionally.

Thus, humans are more likely to make false friend errors than MT. 

Fuzzy Match

There are several errors that result from the use of TM. These memories offer the human translator suggestions of translation that are similar to the segment they are translating. Similar does not mean equal, so if the suggested translation is a fuzzy match, the human translator must make changes. If they don’t make any change and accept the fuzzy match as it is, they risk making errors. There are three sub-types of errors to mention here:

Different terms. Think of a medical procedure where the next step is “Administer the saline solution to the patient.” If a fuzzy match shows “Administer the plasma to the patient,” this might risk a person’s life.

Opposite meaning. Think of “The temperature of the solution administered to the patient must stay below XX degrees.” If a fuzzy match shows “must stay above XX degrees,” this might risk a person’s life. For an eCommerce environment, this type of error could be a major issue: “This item will not be shipped to Brazil” versus “This item will be shipped to Brazil.” 

Numbers that don’t match. Fuzzy matches from a year before may offer the translator a suggested translation of “iPhone 5” because that was the model from a year ago. The new model is the iPhone 6. If a fuzzy match is accepted with the wrong number, the translator is introducing an old model.

Thus, humans are much more likely to make errors for accepting fuzzy matches than MT.

Acronyms 

MT may leave acronyms as they are, because they may not be present in the corpus. The MT engine has the advantage of having the corpus to clarify if an acronym should be translated as the same acronym as in the original, if it should be translated as a translated acronym or if it should be translated using the expanded words from the meaning of the acronym. Human translators may make errors here. If they do research, and the research does not clarify the meaning, the original acronym may be left in the translation. So this is an issue that favors the MT over humans, although not heavily.
The best solution for both humans and MT is to try to find the expanded form of the acronyms. This will help MT and humans produce a great and clear translation. 

Thus, humans are slightly more likely to make errors translating acronyms than MT. 

Terminology 

MT may handle terminology remarkably better than a human translator. If an engine is trained with content that is specific to the subject being translated, and that has been validated by subject matter experts and by feedback from the target audience that reads that content, the specific terminology for that subject will be very accurate and in line with the usage. Add to this the fact that multiple translators may have created those translations that are in the corpus, and it becomes easy to see how an MT engine can do a better job with terminology than a single human translator, who often translates different subjects all the time and cannot be a subject matter expert on every subject. 

Consider the following example: 

English: “In photography and cinematography, a wide-angle lens refers to a lens whose focal length is substantially smaller than the focal length of a normal lens for a given film plane. This type of lens allows more of the scene to be included in the photograph.”

Portuguese machine translation: “Em fotografia e cinematografia, uma lente grande angular refere-se a uma lente cuja distância focal é substancialmente menor do que a distância focal de uma lente normal para um determinado plano do filme. Este tipo de lente permite que mais da cena a ser incluída na fotografia.” 

In this example about photography, the MT already proposed the translation of wide-angle as grande angular, which is the term commonly used to refer to this type of lens. This translation means approximately “large angular.” A human translator knows the translations for the words wide and angle. The translator could then be tempted to translate the expression wide-angle literally as “wide angle” lenses (lente de ângulo amplo), missing the specific terminology used for the term. The same could happen for focal length. Portuguese usually uses distância focal, which means “focal distance." A human translator, knowing the translation for focal and length, would be tempted to translate this as comprimento focal and would potentially miss the specific terminology distância focal. 

The quality of the terminology is, of course, based on the breadth and depth of the corpus for the specific subject. A generic engine such as Google or Bing may not do as well as an MT engine custom-trained for a subject. But overall, this is an issue that could favor the MT over humans. 

Thus, humans are more likely to make errors for inappropriate terminology than MT.

 Figure 1: MT and human translation errors

Emerging technologies for post-editing 

Now that we are aware of the issues, which are summarized in Figure 1, we are in a better position to look at emerging technologies related to post-editing. Post-editing work has one basic requirement: that the translator is able to receive MT suggestions to correct, if need be. This technology is now available integrated on several computer-aided translation (CAT) tools. 

Considering the above, the next application of technology is the use of quality assurance (QA) tools to find MT errors. The technology itself is not new and has been available in CAT tools and in QA tools such as Xbench or Okapi CheckMate. What is new is the nature of checks that must be done with these tools. One example: in human translation you use a glossary to ensure the consistent translation of a term. In MT, you could create a check to find a certain polysemous word and the most likely wrong translation for it. Case frequently has the meaning of an iPhone case, but it is often wrongly translated with the meaning of a legal case. Your glossary entry for MT may say something like “find case in the source and legal case in the target.” This check is very different from a traditional check for human translation that looks for the use of the correct translation instead of the use of the “probably wrong” one.

After doing post-editing and finding errors, the last area of application of this technology is in the measurement of the post-editing, since what makes post-editing most attractive is the promise of increasing the efficiency of the translation process. We will briefly mention some of the main technologies being used or researched: 

Post-editing speed tracking. The time spent post-editing can be tracked at a segment level. These numbers can then be compiled for a file. Some examples of use of this technology include the MateCat tool, the iOmegaT tool and the TAUS DQF platform.

MT confidence. Another technology worth mentioning is MT confidence scores. Based on several factors, an MT engine can express how confident it is on the translation of a certain word. If this confidence can be expressed in terms of coloring the words in a segment, this feature will help the post-editor focus on words with less confidence that are therefore more likely to require a change. This feature appeared in the CASMACAT project, as illustrated in Figure 2. 

 Figure 2: The CASMACAT project shows MT color-coded according to what is most likely to need post-editing

Edit distance. A concept that is not new but could be more used more often is the concept of edit distance. It is defined as the number of changes — additions, deletions or substitutions — made to a segment of text to create its final translated form. Comparing the final form of a post-edited segment to the original segment that came out of the MT engine provides a significant indication of the amount of effort that went into the post-editing task. The TAUS DQF platform uses edit distance scores.

We use the concept of edit distance in a broader sense here, indicating the amount of changes. This includes the “raw” number of changes made, but also includes normalized scores that divide the number of changes by the length of the text being changed, either in words or characters. The TER (Translation Edit Rate) score is used to measure the quality of MT output, and is an example of a normalized score. 

The final quality that needs to be achieved through post-editing defines levels of “light post-editing” and “full post-editing.” There are discussions to define and measure these levels. The scores based on edit distance may provide a metric that helps in this definition. It is expected that the light post-editing should require fewer changes than a full post-editing, therefore the scores based on edit distance for a light post-editing should always be lower than the score for a full post-editing. Figure 3 below shows a hypothetical example with numbers.

Figure 3: Edit distance shows how much editing is required for a given machine-translated file.

Scores based on edit distance can be an important number in the overall scenario of measuring post-editing, combined with the measurements of speed. The TAUS DQF efficiency score proposed a combination of these measurements.

Silvio Picinini is a Machine Translation Language Specialist at eBay since 2013. With over 20 years in Localization, he worked on quality-focused positions for an LSP, and as an in-house English into Brazilian Portuguese translator for Oracle. A former quality engineer, Silvio holds degrees in electrical and electronic/software engineering.

LinkedIn: https://www.linkedin.com/in/silviopicinini