Understanding Machine Translation Quality: A Review

This is a reprint of a post I wrote and already published here with some minor formatting changes made for emphasis. It is the first of a series of ongoing posts that will be published at that site and also shared here if it seems appropriate. 

For those who may seek or insist that I maintain a truly objective viewpoint, I should warn you that these posts will reflect my current understanding that ModernMT is truly a superior MT implementation for enterprise MT use. I will stress this often in future posts as I have not seen a better deployment of MT technology for professional business translation in the 15 years I have been involved with Enterprise MT.

==============

Today we live in a world where machine translation (MT) is pervasive, and increasingly a necessary tool for any global enterprise that seeks to understand, communicate and share information with a global customer base.

It is estimated by experts that trillions of words are being translated daily with the aid of the many “free” generic public MT portals worldwide.

This is the first in a series of posts that will explore the issue of MT Quality in some depth, with several goals:

• Explain why MT quality measurement is necessary,
• Share best practices,
• Expose common misconceptions,
• Understand what matters for enterprise and professional use.

While much has been written on this subject already, it has not seemed to have reduced the amount of misunderstanding and confusion around this subject. Thus, there is value in continued elucidation to ensure that greater clarity and understanding are achieved.

So let’s begin.

MT Quality and Why Does It Matter?

Machine Translation (MT) or Automated Translation is a process when computer software “translates” text from one language to another without human involvement.

There are ten or more public MT portals available to do this in the modern era, and additionally, many private MT offerings are available to the modern enterprise to address their large-scale language translation needs. For this reason, the modern global enterprise needs an understanding of the relative strengths and weaknesses of the many offerings available in the marketplace.

Ideally, the “best” MT system would be identified by a team of competent translators who would run a diverse range of relevant content through the MT system after establishing a structured and repeatable evaluation process. 

This is slow, expensive, and difficult, even if only a small sample of 250 sentences is evaluated.

Thus, automated measurements that attempt to score translation adequacy, fluency, precision, and recall have to be used. They attempt to do what is best done by competent humans. This is done by comparing MT output to a human translation in what is called a Reference Test set. These reference sets cannot provide all the possible ways a source sentence could be correctly translated. Thus, these scoring methodologies are always an approximation of what a competent human assessment would determine, and can sometimes be wrong or misleading.

Thus, identifying the “best MT” solution is not easily done. Consider the cost of evaluating ten different systems on twenty different language combinations with a human team versus automated scores. Even though it is possible to rank MT systems based on scores like BLEU and hLepor, they do not represent production performance. The scores are a snapshot of an ever-changing scene. If you change the angle or the focus the results would change.

A score is not a stable and permanent rating for an MT system. There is no single, magic MT solution that does a perfect job on every document or piece of content or language combination. Thus, the selection of MT systems for production use based on these scores can often be sub-optimal or simply wrong.

Additionally, MT technology is not static: the models are constantly being improved and evolving, and what was true yesterday in quality comparisons may not be true tomorrow.

For these reasons, understanding how the data, algorithms, and human processes around the technology interact is usually more important than any comparison snapshot.  

In fact, building expertise and close collaboration with a few MT providers is likely to yield better ROI and business outcomes than jumping from system to system based on transient and outdated quality score-based comparisons.

Two primary groups have an ongoing and continuing interest in measuring MT quality. They are:

1. MT developers
2. Enterprise buyers and LSPs

They have very different needs and objectives and it is useful to understand why this is so.

Measurements that may make sense for developers can often be of little or no value to enterprise buyers and vice versa. 

 

MT Developers

MT developers typically work on one model at a time, e.g.: English-to-French. They will repeatedly add and remove data from a training set, then measure the impact to eventually determine the optimal data needed.

They may also modify parameters on the training algorithms used, or change algorithms altogether, and then experiment further to find the best data/algorithm combinations using instant scoring metrics like BLEU, TER, hLepor, ChrF, Edit Distance, and Comet.

While such metrics are useful to developers, they should not be used to cross-compare systems, and have to be used with great care.  The quality scores from several (data/algorithm) combinations are calculated by comparing MT output from each of these systems (models) to a Human Reference translation of the same evaluation test data. The highest scoring system is usually considered the best one.

In summary, MT developers use automatically calculated scores that attempt to mathematically summarize overall precision, recall, adequacy, and fluency characteristics of an MT system into a numeric score, This is done to identify the best English-to-French system, as stated in our example, that they can build with available data and computing resources.

However, a professional human assessment may often differ from what these scores say.

In recent years, Neural MT (NMT) models have exposed that using these automated scoring metrics in isolation can lead to sub-optimal choices. Increasingly, human evaluators are also engaged to ensure that there is a correlation between automatically calculated scores and human assessments.

This is because the scores are not always reliable, and human rankings can differ considerably from score-based rankings. Thus, the quality measurement process is expensive, slow, and prone to many procedural errors, and sometimes even deceptive tactics.

Some MT developers test on training data which can result in misleadingly high scores. (I know of a few who do this!) The optimization process described above is essentially how the large public MT portals develop their generic systems, where the primary focus is on acquiring the right data, using the best algorithms, and getting the highest (BLEU) or lowest (TER) scores.

Enterprise Buyers and LSPs

Enterprise Buyers and LSPs usually have different needs and objectives. They are more likely to be interested in understanding which English-to-French system is the “best” among five or more commercially available MT systems under consideration.

Using automated scores like BLEU, hLepor and TER do not make as much sense in this context. The typical enterprise/LSP is also additionally interested in understanding which system can be “best” modified to learn enterprise terminology and language style.

Optimization around enterprise content and subject domain matters much more, and a comparison of generic (stock) systems can often be useless in the considered professional use context.

Many forget that many business problems require a combination of both MT and human translation to achieve the required level of output quality. Thus, a tightly linked human-in-the-loop (HITL) process to drive MT performance improvements has increasingly become a key requirement for most enterprise MT use cases.

Third-party consultants have compared generic (stock or uncustomized) engines and ranked MT solutions using a variety of test sets that may or may not be relevant to a buyer. These rankings are then often being used to dynamically select different MT systems for different languages, but it is possible and even likely, that they are making sub-optimal choices.  

The ease, speed, and cost of tuning and adapting a generic (stock) MT system to enterprise content, terminology, and language style matter much more in this context, and comparisons should only be made after determining this aspect.

However, as generic system comparisons are much easier and less costly to do, TMS systems and middleware that allow MT system selection using these generic evaluation test data scores, often make choices based on irrelevant and outdated data and can thus be sub-optimal. This is a primary reason that so many LSP systems perform so poorly and why MT is so underutilized in this sector.

While NMT continues to gain momentum as the average water level keeps rising, there is still a great deal of naivete and ignorance in the professional translation community about MT quality assessment and MT best practices in general. The enterprise/LSP use of MT is much more demanding in terms of focused accuracy and sophistication in techniques, practices, and deployment variability, and few LSPs are capable or willing to make the investments needed to achieve ongoing competence as the state-of-the-art (SOTA) continues to evolve.

Dispelling MT Quality Misconceptions

1) Google has the “best” MT systems

This is one of the most widely held misconceptions. While Google does have excellent generic systems and broad language coverage, it is not accurate to say that they are always the best.

Google MT is complicated and expensive to customize for enterprise use cases, and there are significant data privacy and data control issues to be navigated.  Also, because Google has so much data underlying their MT systems, they are not easily customized by the relatively meager data volumes that most enterprises or LSPs have available. DeepL is often a favorite of translators, but also has limited customization and adaptation options.

ModernMT is a dynamically adaptive, and continuously learning breakthrough neural MT system. As it is possibly the only MT system that learns and improves with every instance of corrective feedback in real-time, a comparative snapshot based on a static system is even less useful.

A properly implemented ModernMT system will improve rapidly with corrective feedback, and easily outperform generic systems on the enterprise-specific content that matters most. Enterprise needs are more varied, and rapid adaptability, data security, and easy integration into enterprise IT infrastructure typically matter most.

2) MT Quality ratings are static & permanent

MT systems managed and maintained by experts are updated frequently and thus snapshot comparisons are only true for a single test set at a point in time. These scores are a very rough historical proxy for overall system quality and capability, and deeper engagement is needed to better understand system capabilities.

For example, to make proper assessments with ModernMT, it is necessary to actively provide corrective feedback to see the system improve exactly on the content that you are most actively translating now. If multiple editors concurrently provide feedback, ModernMT will improve even faster. These score-based rankings do not tell you how responsive and adaptive an MT system is to your unique data.

TMS systems that switch to different MT systems via API for each language are of dubious value since selections are often based on static and outdated scores. Best practices recommend that efforts to improve an MT systems adaptation to enterprise content, domain, and language style yield higher value than using MT system selection based on embedded scores built into TMS systems and middleware.

3) MT quality ratings for all use cases are the same.

The MT quality discussion needs to evolve beyond targeting linguistic perfection as the final goal, or comparison of BLEU, TER, or hLepor scores, and proximity to human translation.

It is more important to measure the business impact and make more customer-relevant content multilingual across global digital interactions at scale. While it is always good to get as close to human translation quality as possible, this is simply not possible with the huge volumes of content that are being translated today.

There is evidence now that shows that for many eCommerce use scenarios, even gist translations that contain egregious linguistic errors can produce a positive business impact. In information triage scenarios typical in eDiscovery (litigation, pharmacovigilance, national security surveillance) the translation needs to be accurate on key search parameters but not on all the text.  Translation of user-generated content (UGC) is invaluable to improving and understanding the customer experience and is also a primary influence on new purchase activity. None of these scenarios require perfect linguistic quality MT output, to have a positive business impact and drive successful customer engagement.

4) The linguistic quality of MT output is the only way to assess the “best” MT system.

The linguistic quality of MT output is only one of several critical criteria needed for robust evaluation for an enterprise/LSP buyer. Enterprise requirements like the ease and speed of customization to enterprise domain, data security and privacy, production MT system deployment options, integration into enterprise IT infrastructure,  overall MT system manageability, and control also need to be considered.

Given that MT is rapidly becoming an essential tool for a globally agile enterprise, we need new ways to measure the quality and value of MT in global CX scenarios. In the scenarios where MT enables better communication, information sharing, and understanding of customer concerns on a global scale, we need new ways to measure success.  A closer examination of business impact reveals that the metrics that matter the most would be:

• Increased global digital presence and footprint
• Enhanced global communication and collaboration
• Rapid response in all global customer service/support scenarios
• Productivity improvement in localization use cases to enable more content to be delivered at higher quality
• Improved conversion rates in eCommerce

And ultimately the measure that matters at the executive level is the measurably improved customer experience of every customer in the world. 

This is often more a result of process and deployment excellence than the reported semantic similarity scores of any individual MT system.

The reality today is that increasingly larger volumes of content are being translated and used with minimal or no post-editing.  The highest impact MT use cases may only post-edit a tiny fraction of the content they translate and distribute.

However, much of the discussion in the industry today still focuses on post-editing efficiency and quality estimation processes that assume all the content will be post-edited.

It is time for a new approach that easily enables tens of millions of words to be translated daily, in continuously learning MT systems that improve by the day and enable new communication, understanding, and collaboration with globally distributed stakeholders.

In the second post in this series, we will dig deeper into BLEU and other automated scoring methodologies and show why competent human assessments are still the most valuable feedback that can be provided to drive ongoing and continuous improvements in MT output quality.

Evaluating Machine Translation Systems

This post is the first in a series of upcoming posts focusing on the issue of quality evaluation of multiple MT systems. MT system selection has become a more important issue in recent times as users and buyers realize that potentially multiple MT systems can be viable for their needs, but would like to develop better, more informed selection procedures.

I have also just ended my tenure at SDL, and this departure will also allow my commentary and opinion in this blog to be more independent and objective, from this point onwards. I look forward to looking more closely at all the most innovative MT solutions in the market today and providing more coverage on them.  

As NMT technology matures it has become increasingly apparent to many buyers that traditional metrics like BLEU that are used to compare/rank different MT systems and vendors are now often inadequate for this purpose, even though these metrics are still useful to engineers who are focused on building a single MT system.  It is now much more widely understood that best practice involves human evaluations used together with automated metrics. This combined scoring approach is a more useful input in conducting comparative evaluations of MT systems.  To the best of my knowledge, there are very few in the professional translation world who do this well, and it is very much an evolving practice and learning that is happening now. Thus, I invite any readers who might be willing to share their insights into conducting consistent and accurate human evaluations to contact me about doing this here.

Most of the focus in the localization world's use of MT remains on MTPE efficiencies (edit distance, translator productivity), often without consideration of how the volume and useable quality might change and impact the overall process and strategy. While this focus has value, it misses the broader potential of MT and "leaves money on the table" as they say.

We should understand the questions that we are most frequently asking is: 

• What MT system would work best for our business purposes?
• Is there really enough of a difference between systems to use anything but the lowest cost vendor?
• Is there a better way to select MT systems than just looking at generic BLEU scores?

I have covered these questions to some extent in prior posts and I would recommend this post and this post to get some background on the challenges in understanding the MT quality big picture.

The COVID-19 pandemic is encouraging MT-use in a positive way. Many more brands now realize that speed, digital agility, and a greater digital presence matter in keeping customers and brands engaged. As NMT continues to improve, much of the "bulk translation market" will move to a production model where most of the work will be done by MT.  Translators who are specialists and true subject matter experts are unlikely to be affected by the technology in a negative way, but NMT is poised to penetrate standard/bulk localization work much more deeply, driving costs down as it does so.

This is a guest post and an unedited independent opinion from an LSP (Language Service Provider) and it is useful in providing us an example of the most common translation industry perspective on the subject of multiple MT system evaluations. It is interesting to note that the NMT advances over SMT are still not quite understood by some, even though the bulk of the research efforts and most new deployments have shifted to NMT. 

Most LSPs continue to stress that human translation is "better" than MT which most of us on the technology side would not argue against, but this view loses something when we see that the real need today is to "translate" millions of words a day. This view also glosses over the fact that all translation tasks are not the same. Even in 2020 most LSPs continue to overlook that MT solves new kinds of translation problems that involve speed and volume and that new skills are needed to really leverage MT in these new directions. There is also a tendency to position the choice as binary MT vs Human Translation, even though much of the evidence is pointing to new man + machine models that provide an improved production approach. The translation needs of the future are quite different from the past and I hope that more service providers in the industry start to recognize this. 

I also think it is unwise for LSPs to start building their own MT systems, especially with NMT. The complexity, cost and expertise required are prohibitive for most. MT systems development should be left to real experts who do this on a regular and continuing basis. The potential for LSPs adding value is in other areas, and I hope to cover this in the coming posts.

Source: MasterWord

                                                                                                                                                               * =======*

It’s not a secret that machine translation (MT) has taken the world by storm. Almost everyone now has had some experience with MT, mostly in the form of a translation app such as Google Translate being popular. But MT comes in a variety of formats and is heavily utilized by businesses and institutions all over the world.

With that in mind, which MT system is best? Since MT comes in many colors, figuratively speaking, which one should you ought to rely on if you decide to build your own MT system? We’ll also talk more about translation quality and whether or not MT is suitable for specialized translations such as medical translation; a critical field now for any active translation company in light of the current coronavirus pandemic that has the whole world at a standstill.

What is Machine Translation?

Machine Translation, or MT, is software that is capable of translating text from a source language to a translated text of the target language. Over the years, there have been multiple variations of MT, but there are three definitive types; Rules-based Machine Translation (RBMT), Statistical Machine Translation (SMT), and Neural Machine Translation (NMT). Here’s a quick rundown of their characteristics, including their pros and cons between each other;

1. RBMT

Rules-Based Machine Translation is one of the earliest forms of MT. Its algorithm is language-based, meaning for it to know how to translate one source language to the other, it must rely on input data in the form of a lexicon, grammar rules, and other linguistic fundamentals. The problem with RBMT systems is scaling it efficiently as it becomes more complicated as more language rules are added. Also, RBMT is never ideal for obscure languages with minuscule data. However, with the development of advanced MT systems over the years, RMBT has largely been superseded, in which you'll know more about its successor next.

2.  SMT

Statistical Machine Translation, compared to RBMT, is designed to translate languages from statistical algorithms. SMT works by being fed with data in the form of bilingual text corpora, SMT is programmed to identify patterns in the data and form its translations from it. Patterns in this context mean how many times a certain word/phrase appears consistently in a certain context. This probability learning model allows SMT systems to render relatively appropriate translations compared. It’s pretty much like ‘If this is how was it done, then this is how it should be done’. 

SMT also must be fed with plenty of data just like RBMT, but MT developers of which includes translation app developers prefer SMT due to its ease of setting up due to numerous open-source SMT systems available, cost-effectiveness due to free quality parallel text corpora that are available online, higher translation accuracy than RMBT, and its ease of scalability as the system grows bigger.

But just like RBMT, SMT can’t function well if it’s fed with insufficient and poorly structured parallel text corpora. That being said, it’s not that ideal to translate obscure languages.

3. NMT

Neural Machine Translation is the latest development in MT. Think of it as an upgraded version of SMT in which its abilities are now supplemented with artificial intelligence (AI), specifically deep learning. Not only is it capable of coming through data faster, but it can also produce better outputs through constant trial and error. SMT does it the same way as well but the only difference, albeit a definitive one, is that it’s able to do it much faster and more accurately. Google Translate recently made the switch in 2016 to NMT from its old SMT system.

Its deep learning capability is such a real game-changer that it’s able to accomplish what RBMT and SMT; translating obscure regional languages. That’s why Google Translate can cover over 100 languages such as Somalian and Gaelic. But its outputs are questionable, to say the least as it needs some time to learn a language that has little reliable data lying around for it to use. However, the development of NMT just goes to show how far MT overall has evolved over the years.

What Makes A Good Machine Translation (MT) System?

There have been many MT systems over the years and many still in development. The ones that happened to survive the test of time are select variants of RBMT and most variants of SMT. NMT has quickly gained popularity and will slowly replace SMT as the years go by. What’s generally expected out of a good custom-built MT system is reliability and quality of outputs, pretty much like any other product or service out there.

If you’re looking for a reliable metric, then BLEU (Bilingual Evaluation Understudy) is one of the most widely used MT evaluation metrics. BLEU ranks MT systems between 0 being the worst and 1 being the best. It rates how close the translated text is to a human. The more human-like and natural-sounding the translation is, the better the score.

That being said, every MT developer creates their system according to not only the developer’s but also a client’s specifications and linguistic needs. So not one of them is alike. But there are MT platforms that are widely used by multiple clients due to their flexibility of being adapted to the client’s needs and ease of use. But even with a variety of MT systems being developed over the years, one thing remains the same; MT systems have to learn from a lot of quality data and must be given the time to learn.

They say that machines are inherently dumb and that they’re only as good as job or data are given to them. For MT, that notion still rings true up to this day and will most likely keep ringing for decades to come. However, quality data isn’t only what makes a good MT system.

There are platforms in which MT is integrated with other processes for it to render quality or at the very least, passable translations. Indeed, MT itself is a process onto its own, but its outputs, even with deep learning capabilities, is still not up to par with that of a professional translator. MT has to be integrated with other processes, namely computer-assisted translation (CAT) tools.

There are many CAT tools but two of the most essential are a glossary tool and translation memory. A glossary is simply a database of terminologies and approved translations. It’s a very simple feature but very important as it saves up a lot of time for the translator as they don’t need to constantly look back and forth which translation is the perfect choice for the source text at hand.

A translation memory is also like a glossary, but stores phrases and sentences. It also saves the translator valuable time as many translations recycle the same language such as user manuals, marketing collateral, and etc. A translation memory also helps by providing consistent language at a given domain and language pair.

I Now Pronounce You Man and Machine

However, even with all the bells and whistles, developers can equip an MT system with, is MT alone enough? Can MT alone produce accurate and quality translations that are demanded by the clients of language services today? MT is part of the solution but doesn’t comprise the complete picture. It sounds counterintuitive, but MT is best paired with a professional translator as a means of optimizing the translation process.

This unlikely union broke the predictions of many that saw MT giving professional translators a run for their money and driving translation companies out of business. Professional translators work with CAT tools as it helps them be more churn out more words than ever before and helps them be more consistent. Why the need for speed? Domo’s latest report states that “2.5 quintillion bytes of data are created every single day”—that’s a lot of data and most of it is not in English which creates the rising demand for translation services.

Also, by having a translator work together with an MT system, the translator is doing the MT system a favor as well by constantly feeding back revisions for the MT to learn from and render better outputs and suggestions. All in all, it’s a highly productive and beneficial two-way street between a translator and an MT system.

Of course, this ‘relationship’ will be all for moot if the MT system wasn’t developed to a satisfactory standard. That being said, developers have to take into account both translation clients and translators themselves.

They have to ensure that not only will the MT system procure quality translations for clients but can also adapt to the needs of the translators using them. Being convenient to use and having a friendly UX design is one thing, but being able to incorporate the inputs of a translator and accurately replicating it in similar contexts is also another thing.

What Do Professional Translation Services Have Over MT?

Specifically, what can a translation company that hires professional translators to do better than artificial intelligence (AI)? Apart from translation quality and consistency, a professional translator has one advantage; they’re human. It may sound cliche but a human can understand nuances and no MT or AI are light years away from replicating.

Unable to Understand Emotional, Cultural, and Social Nuances

As of now, there is no MT yet that is capable of accurately understanding jokes, slang, creative expressions, and so on. The abilities of MT shine brightly with formulaic sentences and predictable language conventions. But if confronted with linguistic habits that are natural in everyday conversations, MT falls apart. This problem is made more pronounced at a global scale since every culture and society has its own way of speaking all the way down to highly distinct street lingo.

Unable to Process Linguistic Nuances

Parent languages are divided by their regional vernaculars and dialects. When someone’s trying to translate English to Spanish, it’s actually just generic Spanish with no local ‘flavoring’. But if you’re aiming for translations that resonate true to how Spanish people or how Mexican people speak, then a professional translator with native-speaking ability is who you need. No MT system now is able to comprehend, let alone translate linguistic nuances reliably.

Unable to Keep Up With Linguistic Trends

Languages change every day with new words being constantly added and removed to the lexicon of world languages. Humor, slang, and creative expressions are a testament to that notion. Even social media has given rise to new creative expressions in ways human society has never experienced before with meme culture as one of the most notable examples. Even if NMT was somehow capable of keeping up, it would still need time for the data to accumulate for it to start translating. By that time, new slang would have already popped out.

Unable to Render Specialized and Highly Contextual Translations

What we mean by specialized here is text with highly nuanced terminology such as the literary field and also texts belonging to critical fields such as the legal, scientific, medical sector. Authors inherently embed their works with highly nuanced expressions and linguistic ‘anomalies’, so much so that there is no identifiable pattern for any MT that can work with since each author has their own voice.

For the legal, and the medical sector, have their own language conventions that although seem formulaic on the surface, the inherently specialized terminologies and the risk factor involved in these fields means no margin of error can be given to MT. There are MT systems used in these sectors but are always paired with a professional legal translator and professional medical translator.

Developing Your Own MT System

Even with the quality issues and other imperfections associated with MT, the demand for machine translation services. According to a report published in Market Watch, “The Global Machine Translation Market was valued at USD 550.46 million in 2019 and is expected to reach USD 1042.46 million by 2025, at a CAGR of 11.23% over the forecast period 2020 - 2025.”.

However, many are looking to develop their own company MT instead of ‘borrowing’ one from an external provider and for good reason. If a translation company is rendering plenty of niche translations in a given year, then configuring their own MT system is the most cost-effective investment as there will be no need to pay for licensing fees to external MT providers.

Many industries have their language conventions and jargon, in regards to internal communication mostly. For example, legalese is perfectly comprehensible to lawyers but downright alien-sounding to those with little legal knowledge. That being said, even businesses and organizations have their own language conventions that veer off from the industry norm. In that case, they would then have to build their very own MT systems, especially if they’re focusing on specific target foreign markets and audiences. 

So out of the 3 listed earlier, which one should you choose? It’s most likely SMT due to its popularity and how much support it gets. There are who have gone for a Hybrid MT by combining SMT and RBMT but that’s probably too intimidating for first-timers. If you want to make the big leap right from the start, then, by all means, go NMT if it meets your company’s objectives. 

Mind you that investing and training any MT system does come at a price and will take time. It’ll take time for glossaries and translation memories to develop, provided that the data used to feed the system is of standard. For a translation company, that usually isn’t a problem as in tandem with open-source parallel text corpora are the translation company’s own document archives.

Can You Choose MT Over a Translation Company?

Back then, instant language translation belonged to the category of futuristic science fiction gadgets. In fact, it still is today albeit we’ve heightened our standards. What we dream of now is instant voice interpretation. Specifically, being able to conduct a seamless multilingual conversation with anyone without the awkward pauses. But let’s get back to reality now. It’s hard not to be impressed with the abilities of MT today since we can easily witness it from our smartphones.

Even so, there are plenty of flaws associated with MT as discussed earlier that’s actually hindering it from developing serious widespread adoption. Be that as it may, MT as it now nevertheless has its own perks. Although one shouldn’t rely too much on MT at certain thresholds, doesn’t mean that you shouldn’t use it at all at specific situations. Here are some reasons why.

Cost

There are plenty of translation apps out there such as Google Translate as you might know already. All of them are free with the exception of premium access subscription payments to unlock more features. There are plenty of free translation plugins as well for website developers. Keep in mind that we’re talking about generic translators here and not the specialized MT systems from external providers that have licensing fees.

Speed and Convenience

At specific situations, some are just looking to have translation at the very moment they want it. Whether you’re a language student or a traveling businessperson, MT is your answer. It’s free and they can get results the moment they click the translate button. Even if it’s not 100% accurate, it at least gives them an implied meaning behind the translation.

For Generic, Repetitive, and Well-Resourced Languages

*Consider this pointer at your own risk*. One can certainly find MT if they have non-contextual and predictable text at hand such as simple and formulaic phrases. What you decide to do with it is all on you whether you use it only as a reference or actually employ it in a professional setting. That being said, the most quality translations you can get are from well-resourced such as Spanish, German, French, etc. If you tried translating, even a simple phrase from English to Chinese, you’ll unlikely get a similarly accurate translation since English and Chinese have vastly different language rules and an unrelated linguistic history.

A Note on Translation Quality in the Context of the Coronavirus Pandemic

Despite the vast improvements to MT, quality is still a significant issue and as you’re aware, human translators are there to guarantee that. However, in no situation is quality ever more necessary than in global communication in crisis as made evident by the current coronavirus pandemic, specifically in the form of medical translation. Medical translation is a highly specialized niche in translation and critical one too wherein the slightest mistranslation would lead to potentially unfortunate and even fatal consequences.

Medical translation must be provided by specialized medical translators who have complete mastery over their language pair (Ex. English to Spanish, Spanish to English) and extensive familiarity with medical terminology, medical practices, and code of ethics. They must undergo additional lengthy training before they can be classified as certified medical translators. That being said, are MT systems out of the picture?

There are MT systems that translate medical documents and medical research, but it must be under constant supervision from a certified medical translator. Connecting it to today’s crisis, there hasn’t been a recent time in history where a speedy translation of medical research has been more important than ever. Medical scientists all over the world are working together to understand the COVID-19 virus for them to come up with viable treatments and eventually, a vaccine. With that in mind, medical translation is the only bridge that’s making this level of coordination between medical scientists around the world possible.

Final Takeaway

Will there be a future where MT would be so advanced and almost human-like that professional translators would be an endangered species? If you were to judge by the pace of development of MT in such a short period, it would not be that unreasonable to believe in a future like that. However, let’s not put too much thought into it as it doesn’t pay attention enough to what is demanded from translation in the first place.

It’s apparent now that MT is good at servicing the translation speed and optimization needs, but as for quality, much of it belongs to the hands, or should I say the mind of a professional translator. That union would likely last for the next few decades. But let’s not hold ourselves to that prediction. Perhaps a game-changing MT feature is just a few years away or if our prediction holds true decades. But still, that’s considering our standards on translations, particularly on quality and human-ness, haven’t changed.

Author Bio:

Laurence Ian Sumando is a freelance writer penning pieces on business, marketing, languages, and culture.

In a Funk about BLEU

MT quality measurement, like human translation quality measurement, has been a difficult and challenging subject for both the translation industry and for many MT researchers and systems developers as the most commonly used metric BLEU, is now quite widely understood to be of especially limited value with NMT systems. 

Most of the other text-matching NLP scoring measures are just as suspect, and practitioners are reluctant to adopt them as they are either difficult to implement, or the interpretation pitfalls and nuances of these other measures are not well understood. They all can generate a numeric score based on various calculations of Precision and Recall that need to be interpreted with great care. Most experts will say that the only reliable measures are those done by competent humans and increasingly best practices suggest that a trust-but-verify approach is better. There are many variations of superficially accurate measures available today, but on closer examination, they are all lacking critical elements to make them entirely reliable and foolproof.

So, as much as BLEU scores suck, we continue to use them since some, or perhaps even many of us understand them. Unfortunately, many still don't have a real clue, especially in the translation industry. 

I wonder sometimes if all this angst about MT quality measurement is much ado about nothing. We do in fact, need very rough indicators of MT quality to make judgments of suitability in business use cases, but taking these scores as final indicators of true quality is problematic. It is likely that the top 5, or even top 10 systems are essentially equivalent in terms of the MT quality impact on the business purpose. The real difference in business impact comes from other drivers: competence, experience, process efficiency and quality of implementation.

I would argue that even for localization use cases, the overall process design and other factors matter more than the MT output quality.

 As we have said before, technology has value when it produces favorable business outcomes, even if these outcomes can be somewhat challenging to measure with a precise and meaningful grade. MT is a technology that is seldom perfect, but even in its imperfection can provide great value to an enterprise with a global presence. MT systems with better BLEU or Lepor scores do not necessarily produce better business outcomes. I would argue that an enterprise could use pretty much any "serious" MT system without any impact on the final business outcome. 

This is most clear with eCommerce and global customer service and support use cases, where the use of MT can very rapidly yield a significant ROI. 

"eBay’s push for machine translation has helped the company increase Latin American exports by nearly 20%, according to researchers from the Massachusetts Institute of Technology, and illustrates the potential for increased commercial activity as translation technologies gain wider adoption in business."

MT deployment use case presentations shared by practitioners who have used MT to translate large volumes of knowledgebase support content show that what matters is whether the content helps customers across the globe get to answers that solve problems faster. Translation quality matters but only if it helps understandability. In the digital world, speed is crucial and often more important.

Some 100,000 buyers exchange a total of 2 billion translated text messages every week on the Alibaba.com global-trade platform. This velocity and volume of communication that is enabled by MT enable new levels of global commerce and trade. How many of these messages do you think are perfect translations? 

A monolingual Live Support Agent who can service thousands of global customers a week because he/she can quickly understand the question and send back relevant and useful support content back to a customer using MT  is another example. The ability to do this in volume matters more than perfect linguistic quality.

So then the selection of the right MT technology or solution will come down to much more enterprise relevant issues like:

• Data Security & Privacy 
• Adaptability to enterprise unique terminology and use cases
• Scalability - from billions of words to thousands per hour 
• Deployment Flexibility - On-premise, cloud or combinations of both
• Integration with key IT infrastructure and platforms
• Availability of expert consulting services for specialization 
• Vendor focus on SOTA
• MT system manageability
• Cost 
• Vendor reputation, profile and enterprise account management capabilities

Pete Smith will be presenting more details of his research study at SDL Connect next month.

===============

There is little debate: the machine translation research and practitioner communities are in a funk about BLEU. From recent webinars to professional interviews and scholarly publications, BLEU is being called on the carpet for its technical shortcomings in the face of a rapidly-developing field, as well as the lack of insight it provides to different consumers such as purchasers of MT services or systems.

BLEU itself is used widely, especially in the MT research community, as an outcome measure for evaluating MT. Yet even in that setting, there is considerable rethinking and re-evaluation of the metric, and BLEU has been an active topic of critical discussion and research for some years, including the challenges faced by evaluating automated translation across the language typology spectrum and especially in cases of morphologically rich languages. And the issue is not limited, of course, to machine translation—the metric is also a topic in NLP and natural language generation discussions generally.

BLEU’s strengths and shortcomings are well-known. At its core, BLEU is a string matching algorithm for use in evaluating MT output and is not per se a measure of translation quality. That said, here is no doubt that automated or calculated metrics are of great value, as total global MT output approaches levels of one trillion words per day.

And few would argue that, in producing and evaluating MT or translation in general, context matters. A general-purpose, public-facing MT engine designed for broad coverage among users and use cases is just that—general-purpose, and likely more challenged by perennial source language challenges such as specific domain style/terminology, informal language usage, regional language variations, and other issues.

It is no secret that many MT products are trained (at least initially) on publicly available research data and that there are, overall, real thematic biases in those datasets. News, current events, governmental and parliamentary data sets are available across a wide array of language pairs, as well as smaller amounts of data from domains such as legal, entertainment, and lecture source materials such as TED Talks. Increasingly, datasets are available in the IT and technical domains, but there are few public bilingual datasets available that are suitable for major business applications of MT technology such as e-commerce, communication, and collaboration, or customer service.

Researchers and applied practitioners have all benefited from these publicly-available resources. But the case for clarity is perhaps most evident in the MT practitioner community.

For example, enterprise customers hoping to purchase machine translation services face a dilemma: how might the enterprise evaluate an MT product or service for their particular domain, and with more nuance and depth than simply relying on marketing materials boasting scores or gains in BLEU or LEPOR? How might you evaluate major vendors of MT services specific to your use case and needs?

And as a complicating factor, we know an increasing amount about the “whys” and “hows” of fine-tuning general-purpose engines to better perform in enterprise cases such as e-commerce product listings, technical support knowledgebase content, social media analysis, and user feedback/reviews. In particular, raw “utterances” from customers and customer support personnel in these settings are authentic language, with all of its “messiness.”

The UTA research group has recently been exploring MT engine performance on customer support content, building a specialized test set compiled from source corpora including email and customer communications, communications via social media, and online customer support. In particular, we explored the utilization of automation and standard NLP-style pre-processing to rapidly construct a representative translation test set for the focused use case.

At the start, an initial set of approximately 3 million English sentence strings related to enterprise communication and collaboration were selected. Source corpora represented tasks such as email communication, customer communications, communications via social media, and online customer support.

Candidate sentence strings from these larger corpora were narrowed via a sentence clustering technique, training a FastText model on the input documents to capture both the semantic and non-semantic (linguistic) properties of the corpora. To give some sense of the linguistic features considered in string selection, corpora elements were parsed using the spaCy natural language processing library’s largest English model to consider features in a string such as the number of “stop words”; the number of tokens that were punctuation, numbers, e-mail addresses, URLs, alpha-only, and out-of-vocabulary; the number of unique lemmas and orthographic forms; number of named entities; the number of times each entity type, part-of-speech tag and dependency relation appeared in the text; and the total number of tokens. Dimensionality reduction and clustering were used in the end, to result in 1050 English-language strings for the basic bespoke test set.

The strings from the constructed set were translated into seven languages (French, German, Hindi, Korean, Portuguese, Russian, Spanish) by professional translators. Then the translated sentences from the test set were utilized as translation prompts in seven language pairs (English-French, English-German, English-Hindi, English-Korean, English-Portuguese, English-Russian, English-Spanish) by four major, publicly-available MT engines via API or web interface. At both the corpus as well as the individual string level, BLEU, METEOR, and TER scores were generated for each major engine and language pair (not all of the seven languages were represented in all engine products).

Our overall question was: does BLEU (or any of the other automated scores) support, say, the choice of engine A over engine B for enterprise purchase when the use case is centered on customer-facing and customer-generated communications? 

To be sure, the output scores presented a muddled picture. Composite scores of the general-purpose engines clustered within approximately 5-8 BLEU points of each other in most languages. And although we used a domain-specific test set, little in the results would have provided the enterprise-level customer with a clear path forward. As Kirti Vashee has pointed out recently, in responding effectively to the realities of the digital world, “5 BLEU points this way or that is negligible in most high-value business use cases.”

What are some of the challenges of authentic, customer language? Two known challenges to MT include the formality/informality of language utterances and emotive content. The double-punch of informality and emotion-laden customer utterances pose a particularly challenging case.

As we reviewed in a recent webinar, customer-generated strings in support conversations or online interactions present a translator with a variety of expressions of emotion, tone, humor, sarcasm, all embedded within a more informal and Internet-influenced style of language. Some examples included:

             Support…I f***ing hate you all. [Not redacted in the original.]
            Those late in the day deliveries go “missing” a lot.
            Nope didn’t turn up…just as expected…now what dude?
            I feel you man, have a good rest of your day!
           Seriously, this is not OK.
           A bunch of robots who repeat the same thing over & over.
           #howdoyoustayinbusiness

Here one can quickly see how an engine trained primarily with formal, governmental or newspaper source would be quickly challenged. But in early results, our attempts to unpack the issues of how MT may perform on emotive content (i.e., not news, legal, or technical content) have provided little insight to date. Early findings suggest surprisingly little interaction between standard ratings of sentiment and emotion run on the test set individual strings (VADER positive, negative, neutral, composite and IBM tone analysis) and variance in downstream BLEU scores.

Interestingly, as an aside in our early work, raw BLEU scores across languages for the entire test set did generally correlate comparatively highly with METEOR scores. Although this correlation is expected, the strength of the relationship was surprising in an NMT context, as high as r=.9 across 1000+ strings in a given language pair. If, as the argument goes, NMT brings strengths in fluency which includes elements METEOR scoring is, by design, more sensitive to (such as synonyms or paraphrasing), one might expect that correlation to be weaker. More broadly, these and other questions around automatic evaluation have a long history of consideration by the MT and WMT communities.

One clearly emerging practice in the field is to combine an automated metric such as BLEU along with human evaluation on a smaller data set, to confirm and assure that the automated metrics are useful and provide critical insight, especially if the evaluation is used to compare MT systems. Kirti Vashee, Alon Lavie, and Daniel Marcu have all written on this topic recently.

Thus, the developing, more nuanced understanding of the value of BLEU may be as automated scores seen as initially most useful during MT research and system development, where they are by far the most widely-cited standard. The recent Machine Translation Summit XVII in Dublin, for example, had almost 500 mentions or references to BLEU in the research proceedings alone.

But this measure may be potentially less accurate or insightful when broadly comparing different MT systems within the practitioner world, and perhaps more insightful again to both researcher and practitioner when paired with human or other ratings. As one early MT researcher has noted, “BLEU is easy to criticize, but hard to get away from!”

Discussions at the recent TAUS Global Content Conference 2019 further developed the ideas of MT engine specialization in the context of the modern enterprise content workflow. Presenters such as SDL and others offered views future visions of content development personalization and use in a multilingual world. These future workflows may contain hundreds or thousands of specialized, specially-trained and uniquely maintained automated translation engines and other linguistic algorithms, as content is created, managed, evaluated, and disseminated globally.

There is little doubt that the automated evaluation of translation will continue to play a key role in this emerging vision. However, a better understanding of the field’s de facto metrics and the broader MT evaluation process in this context is clearly imperative.

And what of use cases that continue to emerge, such as the possibility of intelligent or MT content in the educational space? The UTA research group is also exploring MT applications specific to education and higher education as well. For example, millions of users daily make use of learning materials such as MOOCs—educational content that attracts users across borders, languages, and cultures. A significant portion of international learners come to and potentially struggle with English-language content in edX or other MOOC courses—and thousands of MOOC offerings exist in the world’s languages, untranslated for English-speakers. What role might machine translation potentially play in this educational endeavor?

This is a more fleshed-out version of a blog post by Pete Smith and Henry Anderson of the University of Texas at Arlington already published on SDL.com. They describe initial results from a research project they are conducting on MT system quality measurement and related issues. 

Dr. Pete Smith, Chief Analytics Officer, and Professor

Mr. Henry Anderson, Data Scientist

Localization and Translation Program

Department of Modern Languages and Office of University Analytics

The University of Texas at Arlington

Understanding MT Quality - What Really Matters?

This is the second post in our posts series on machine translation quality. 

The reality of many of these comparisons today is that scores based on publicly available (i.e. not blind) news domain tests are being used by many companies and LSPs to select MT systems which translate IT, customer support, pharma, financial services domain related content. Clearly, this can only result in sub-optimal choices.

The use of machine translation (MT) in the translation industry has historically been heavily focused on localization use cases, with the primary intention to improve efficiency, that is, speed up turnaround and reduce unit word cost. Indeed, machine translation post-editing (MTPE) has been instrumental in helping localization workflows achieve higher levels of productivity.

Many users in the localization industry select their MT technology based on two primary criteria:

1. Lowest cost
2. “Best quality” assessments based on metrics like BLEU, Lepor or TER, usually done by a third party

The most common way to assess the quality of an MT system output is to use a string-matching algorithm score like BLEU. As we pointed out previously, equating a string-match score with the potential future translation quality of an MT system in a new domain is unwise, and quite likely to result in disappointing results. BLEU and other string-matching scores offer the most value to research teams building and testing MT systems. When we further consider that scores based on old news domain content are being used to select systems for customer support content in IT and software subject domains it seems doubly foolish.

One problem with using news domain content is that it tends to lack tone and emotion. News stories discuss terrorism and new commercial ventures in almost exactly the same tone.  As Pete Smith points out in the webinar link below, in business communication, and customer service and support scenarios the tone really matters. Enterprises that can identify dissatisfied customers and address the issues that cause dissatisfaction are likely to be more successful. CX is all about tone and emotion in addition to the basic literal translation. 

Many users consider only the results of comparative evaluations – often performed by means of questionable protocols and processes using test data that is invisible or not properly defined – to select which MT systems to adopt.  Most frequently, such analyses produce a score table like the one shown below, which might lead users to believe they are using the “best-of-breed” MT solution since they selected the “top” vendor (highlighted in green). 

English to French	English to Chinese	English to Dutch
Vendor A – 46.5	Vendor C – 36.9	Vendor B – 39.5
Vendor B – 45.2	Vendor A – 34.5	Vendor C – 37.7
Vendor C – 43.5	Vendor B – 32.7	Vendor A – 35.5

While this approach looks logical at one level, it often introduces errors and undermines efficiency because of the administrative inconsistency between different MT systems. Also, the suitability of the MT output for post editing may be a key requirement for localization use cases, but this may be much less important in other enterprise use cases.

Assessing business value and impact

The first post in this blog series exposes many of the fallacies of automated metrics that use string-matching algorithms (like BLEU and Lepor), which are not reliable quality assessment techniques as they only reflect the calculated precision and recall characteristics of text matches in a single test set, on material that is usually unrelated to the enterprise domain of interest. 

The issues discussed challenge the notion that single-point scores can really tell you enough about long-term MT quality implications. This is especially true as we move away from the localization use case. Speed, overall agility and responsiveness and integration into customer experience related data flow matters much more in the following use cases. The actual translation quality variance measured by BLEU and Lepor may have little to no impact on what really matters in the following use cases.

The enterprise value-equation is much more complex and goes far beyond linguistic quality and Natural Language Processing (NLP) scores. To truly reflect the business value and impact, evaluation of MT technology must factor in non-linguistic attributes including:

• Adaptability to business use cases
• Manageability
• Integration into enterprise infrastructure
• Deployment flexibility   

To effectively link MT output to business value implications, we need to understand that although linguistic precision is an important factor, it often has a lower priority in high-value business use cases. This view will hopefully take hold as the purpose and use of MT is better understood in the context of a larger business impact scenario, beyond localization.

But what would more dynamic and informed approaches look like? MT evaluation certainly cannot be static since systems must evolve as requirements change. Instead of a single-point score, we need a more complex framework that provides an easy, single measure that tells us everything we need to know about an MT system. Today, this is unfortunately not yet feasible.

A more meaningful evaluation framework

While single-point scores do provide a rough and dirty sense of an MT system’s performance, it is more useful to focus testing efforts on specific enterprise use case requirements. This is also true for automated metrics, which means that scores based on news domain tests should be viewed with care since they are not likely to be representative of performance on specialized enterprise content. 

When rating different MT systems, it is essential to score key requirements for enterprise use, including:

• Adaptability: Range of options and controls available to tune the MT system performance for very specific use cases. For example, optimization techniques applied to eCommerce catalog content should be very different from those applied to technical support chatbot content or multilingual corporate email systems.
• Data privacy and security: If an MT system will be used to translate confidential emails, business strategy and tactics documents, human evaluation requirements will differ greatly from a system that only focuses on product documentation. Some systems will harvest data for machine learning purposes, and it is important to understand this upfront.
• Deployment flexibility: Some MT systems need to be deployed on-premises to meet legal requirements, such as is the case in litigation scenarios or when handling high-security data. 
• Expert services: Having highly qualified experts to assist in the MT system tuning and customization can be critical for certain customers to develop ideal systems. 
• IT integration: Increasingly, MT systems are embedded in larger business workflows to enable greater multilingual capabilities, for example, in communication and collaboration software infrastructures like email, chat and CMS systems.
• Overall flexibility: Together, all these elements provide flexibility to tune the MT technology to specific use cases and develop successful solutions.

Ultimately, the most meaningful measures of MT success are directly linked to business outcomes and use cases. The definition of success varies by the use case, but most often, linguistic accuracy as an expression of translation quality is secondary to other measures of success. 

The integrity of the overall solution likely has much more impact than the MT output quality in the traditional sense: not surprisingly, MT output quality could vary by as much as 10-20% on either side of the current BLEU score without impacting the true business outcome. Linguistic quality matters but is not the ultimate driver of successful business outcomes. In fact, there are reports of improvements in output quality in an eCommerce use case that actually reduced the conversion rates on the post-edited sections, as this post-edited content was viewed as being potentially advertising-driven and thus less authentic and trustworthy.

True expressions of successful business outcomes for different use cases

Global enterprise communication and collaboration

• Increased volume in cross-language internal communication and knowledge sharing with safeguarded security and privacy
• Better monitoring and understanding of global customers 
• Rapid resolution of global customer problems, measured by volume and degree of engagement
• More active customer and partner communications and information sharing

Customer service and support

• Higher volume of successful self-service across the globe
• Easy and quick access to multilingual support content 
• Increased customer satisfaction across the globe
• The ability of monolingual live agents to service global customers regardless of the originating customer’s language 

eCommerce

• Measurably increased traffic drawn by new language content
• Successful conversions in all markets
• Transactions are driven by newly translated content
• The stickiness of new visitors in new language geographies

Social media analysis

• Ability to identify key brand impressions 
• Easy identification of key themes and issues
• A clear understanding of key positive and negative reactions

Localization

• Faster turnaround for all MT-based projects
• Lower production cost as a reflection of lower cost per word
• Better MTPE experience based on post-editor ratings
• Adaptability and continuous improvement of the MT system

A more detailed presentation and webinar that goes into much more detail on this subject is available from Brightalk. 

In upcoming posts in this series, we will continue to explore the issue of MT quality assessment from a broad enterprise needs perspective. More informed practices will result in better outcomes and significantly improved MT deployments that leverage the core business mission to solve high-volume multilingual challenges more effectively.

Again, this is a slightly less polished and raw variant of a version published on the SDL site. The first one focused on BLEU scores, which are often improperly used to make decisions on inferred MT quality, where it clearly is not the best metric to draw this inference.

Understanding MT Quality: BLEU Scores

This is the first in a series of posts discussing various aspects of MT quality from the context of enterprise use and value, where linguistic quality is important, but not the only determinant of suitability in a structured MT technology evaluation process. .

What is BLEU (Bilingual Evaluation Understudy)?

As the use of enterprise machine translation expands, it becomes increasingly more important for users and practitioners to understand MT quality issues in a relevant, meaningful, and accurate way.

The BLEU score is a string-matching algorithm that provides basic output quality metrics for MT researchers and developers. In this first post, we will review and look more closely at the BLEU score, which is probably the most widely used MT quality assessment metric in use by MT researchers and developers over the last 15 years. While it is widely understood that the BLEU metric has many flaws, it continues to be a primary metric used to measure MT system output even today, in the heady days of Neural MT.

Firstly, we should understand that a fundamental problem with BLEU is that it DOES NOT EVEN TRY to measure “translation quality”, but rather focuses on STRING SIMILARITY (usually to a single human reference). What has happened over the years is that people choose to interpret this as a measure of the overall quality of an MT system. BLEU scores only reflect how a system performs on the specific set of test sentences used in the test. As there can be many correct translations, and most BLEU tests rely on test sets with only one correct translation reference, it means that it is often possible to score perfectly good translations poorly.

The scores do not reflect the potential performance of the system on other material that differs from the specific test material, and all inferences on what the score means should be made with great care, after taking a close look at the existing set of test sentences. It is very easy to use and interpret BLEU incorrectly and the localization industry abounds with examples of incorrect, erroneous, and even deceptive use.

Very simply stated, BLEU is a “quality metric” score for an MT system that is attempting to measure the correspondence between a machine translation output and that of a human with the understanding that "the closer a machine translation is to a professional human translation, the better it is" – this is the central idea behind BLEU. Scores are calculated for individual MT translated segments—generally sentences—by comparing them with a set of good quality human reference translations. Most would consider BLEU scores more accurate at a corpus level rather than at a sentence level.

BLEU gained popularity because it was one of the first MT quality metrics to report a high correlation with human judgments of quality, a notion that has been challenged often, but after 15 years of attempts to displace it from prominence, the allegedly “improved” derivatives (METEOR, LEPOR) have yet to really unseat its dominance. BLEU together with human assessment remains the preferred metrics of choice today.

A Closer, More Critical Examination of BLEU

BLEU is actually nothing more than a method to measure the similarity between two text strings. To infer that this measurement, which has no linguistic consideration or intelligence whatsoever, can predict not only past “translation quality” performance, but also future performance is indeed quite a stretch.

Measuring translation quality is much more difficult because there is no absolute way to measure how “correct” a translation is. MT is a particularly difficult AI challenge because computers prefer binary outcomes, and translation has rarely if ever only one single correct outcome. Many “correct” answers are possible, and there can be as many “correct” answers as there are translators. The most common way to measure quality is to compare the output strings of automated translation to a human translation text string of the same sentence. The fact that one human translator will translate a sentence in a significantly different way than another human translator, leads to problems when using these human references to measure “the quality” of an automated translation solution.

The BLEU measure scores a translation on a scale of 0 to 1. The measurement attempts to measure adequacy and fluency in a similar way to a human would, e.g. does the output convey the same meaning as the input sentence, and is the output good and fluent target language? The closer to 1, the more overlap there is with a human reference translation and thus the better the system is. In a nutshell, the BLEU score measures how many words overlap, giving higher scores to sequential words. For example, a string of four words in the translation that match the human reference translation (in the same order) will have a positive impact on the BLEU score and is weighted more heavily (and scored higher) than a one or two-word match. It is very unlikely that you would ever score 1 as that would mean that the compared output is exactly the same as the reference output. However, it is also possible that an accurate translation would receive a low score because it uses different words than the reference used. This problem potential can be seen in the following example. If we select one of these translations for our reference set, all the other correct translations will score lower!

How does BLEU work?

To conduct a BLEU measurement the following data is necessary:

1. One or more human reference translations. (This should be data which has NOT been used in building the system (training data) and ideally should be unknown to the MT system developer. It is generally recommended that 1,000 or more sentences be used to get a meaningful measurement.) If you use too small a sample set you can sway the score significantly with just a few sentences that match or do not match well.
2. Automated translation output of the exact same source data set.
3. A measurement utility that performs the comparison and score calculation for you.

• Studies have shown that there is a reasonably high correlation between BLEU and human judgments of quality when properly used.
• BLEU scores are often stated on a scale of 1 to 100 to simplify communication but should not be confused with the percentage of accuracy.
• Even two competent human translations of the exact same material may only score in the 0.6 or 0.7 range as they likely use different vocabulary and phrasing.
• We should be wary of very high BLEU scores (in excess of 0.7) as it is likely we are measuring improperly or overfitting.

A sentence translated by MT may have 75% of the words overlap with one translator’s translation, and only 55% with another translator’s translation; even though both human reference translations are technically correct, the one with the 75% overlap with machine translation will provide a higher “quality” score for the automated translation. This is somewhat arbitrary. Random string matching scores should not be equated to overall translation quality. Therefore, although humans are the true test of correctness, they do not provide an objective and consistent measurement for any meaningful notion of quality.

As would be expected using multiple human reference tests will always result in higher scores as the MT output has more human variations to match against. The NIST (National Institute of Standards & Technology) used BLEU as an approximate indicator of quality in its annual MT competitions with four human reference sets to ensure that some variance in human translation is captured, and thus allow more accurate assessments of the MT solutions being evaluated. The NIST evaluation also defined the development, test, and evaluation process much more carefully and competently, and thus comparing MT systems under their rigor and purview was meaningful. This has not been true for many of the comparisons done since, and many recent comparisons are deeply flawed.

 

Why are automated MT quality assessment metrics needed?

Automated quality measurement metrics have always been important to the developers and researchers of data-driven based MT technology, because of the iterative nature of MT system development, and the need for frequent assessments during the development of the system. They can provide rapid feedback on the effectiveness of continuously evolving research and development strategies.

Recently, we see that BLEU and some of its close derivatives (METEOR, NIST, LEPOR, and F-Measure) are also often used to compare the quality of different MT systems in enterprise use settings. This can be problematic as a “single point quality score” based on publically sourced news domain sentences is simply not representative of the dynamically changing, customized, and modified potential of an active and evolving enterprise MT system. Also, such a score does not incorporate the importance of overall business requirements in an enterprise use scenario where other workflow, integration, and process-related factors may actually be much more important than small differences in scores. Useful MT quality in the enterprise use context will vary greatly, depending on the needs of the specific use-case.

Most of us would agree that competent human evaluation is the best way to understand the output quality implications of different MT systems. However, human evaluation is generally slower, less objective, and likely to be more expensive and thus not viable in many production use scenarios when many comparisons need to be made on a constant and ongoing basis. Thus, automated metrics like BLEU provide a quick and often dirty quality assessment that can be useful to those who actually understand its basic mechanics. However, they should also understand its basic flaws and limitations and thus avoid coming to over-reaching or erroneous conclusions based on these scores.

There are two very different ways that such scores may be used,

• R&D Mode: In comparing different versions of an evolving system during the development of the production MT system, and,
• Buyer Mode: In comparing different MT systems from different vendors and deciding which one is the “best” one.

The MT System Research & Development Need: Data-driven MT systems could probably not be built without using some kind of automated measurement metric to measure ongoing progress. MT system builders are constantly trying new data management techniques, algorithms, and data combinations to improve systems, and thus need quick and frequent feedback on whether a particular strategy is working or not. It is necessary to use some form of standardized, objective and relatively rapid means of assessing quality as part of the system development process in this technology. If this evaluation is done properly, the tests can also be useful over a longer period to understand how a system evolves over many years.

The MT Buyer Need: As there are many MT technology options available today, BLEU and its derivatives are sometimes used to select what MT vendor and system to use. The use of BLEU in this context is much more problematic and prone to drawing erroneous conclusions as often comparisons are being made between apples and oranges. The most common error in interpreting BLEU is the lack of awareness and understanding that there is a positive bias towards one MT system because it has already seen and trained on the test data, or has been used to develop the test data set.

Problems with BLEU

While BLEU is very useful to those who build and refine MT systems, it’s value as an effective way to compare totally different MT systems is much more limited and needs to be done very carefully, if done at all, as it is easily and often manipulated to create the illusion of superiority.

“CSA Research and leading MT experts have pointed out for over a decade that these metrics [BLEU] are artificial and irrelevant for production environments. One of the biggest reasons is that the scores are relative to particular references. Changes that improve performance against one human translation might degrade it with respect to another… Approaches that emphasize usability and user acceptance take more effort than automatic scores but point the way toward a useful and practical discussion of MT quality. “

Excerpt from a CSA Blog on BLEU Misuse, April 2017

There are several criticisms of BLEU that should also be understood if you are to use the metric effectively. BLEU only measures direct word-by-word similarity and looks to match and measure the extent to which word clusters in two sentences or documents are identical. Accurate translations that use different words may score poorly since there is no match in the human reference. 

There is no understanding of paraphrases and synonyms so scores can be somewhat misleading in terms of overall accuracy. You have to get the exact same words used in the human reference translation to get credit e.g.

"Wander" doesn't get partial credit for "stroll," nor "sofa" for "couch."

Also, nonsensical language that contains the right phrases in the wrong order can score high. e.g.

"Appeared calm when he was taken to the American plane, which will to Miami, Florida" would get the very same score as: "was being led to the calm as he was would take carry him seemed quite when taken".

A more recent criticism identifies the following problems:

• It is an intrinsically meaningless score
• It admits too many variations – meaningless and syntactically incorrect variations can score the same as good variations
• It admits too few variations – it treats synonyms as incorrect
• More reference translations do not necessarily help

These and other problems are described in this article and this critical academic review. The core problem is that word-counting scores like BLEU and its derivatives - the linchpin of the many machine-translation competitive comparisons - don't even recognize well-formed language, much less real translated meaning. Here is a more recent post that I highly recommend, as it very clearly explains other metrics, and shows why it also still makes sense to use BLEU in spite of its many problems.

For post-editing work assessments there is a growing preference for Edit Distance scores to more accurately reflect the effort involved, even though it too is far from perfect.

The problems are further exacerbated with the Neural MT technology which can often generate excellent translations that are quite different from the reference and thus score poorly. Thus, many have found that lower (BLEU) scoring NMT systems are clearly preferred over higher scoring SMT systems when human evaluations are done. There are some new metrics (ChrF, SacreBLEU, Rouge) attempting to replace BLEU, but none have gathered any significant momentum yet and the best way to evaluate NMT system output today is still well structured human assessments.

What is BLEU useful for?

Modern MT systems are built by “training” a computer with examples of human translations. As more human translation data is added, systems should generally get better in quality. Often, new data can be added with beneficial results, but sometimes new data can cause a negative effect especially if it is noisy or otherwise “dirty”. Thus, to measure if progress is being made in the development process, the system developers need to be able to measure the quality impact rapidly and frequently to make sure they are improving the system and are in fact making progress.

BLEU allows developers a way “to monitor the effect of daily changes to their systems in order to weed out bad ideas from good ideas.” When used to evaluate the relative merit of different system building strategies, BLEU can be quite effective as it provides very quick feedback and this enables MT developers to quickly refine and improve translation systems they are building and continue to improve quality on a long term basis.

What is BLEU not useful for?

BLEU scores are always very directly related to a specific “test set” and a specific language pair. Thus, BLEU should not be used as an absolute measure of translation quality because the BLEU score can vary even for one language depending on the test and subject domain. In most cases comparing BLEU scores across different languages is meaningless unless very strict protocols have been followed. Because of this, it is always recommended to use human translators to verify the accuracy of the metrics after systems have been built. Also, most MT industry leaders will always vet the BLEU score readings with human assessments before production use.

In competitive comparisons, it is important to carry out the comparison tests in an unbiased, scientific manner to get a true view of where you stand against competitive alternatives. The “test set” should be unknown (“blind”) to all the systems that are involved in the measurement. This is something that is often violated in many widely used comparisons today. If a system is trained with the sentences in the “test set” it will obviously do well on the test but probably not as well on data that it has not seen before. Many recent comparisons score MT systems on News Domain related test sets that may also be used in training by some MT developers. A good score on news domain may not be especially useful for an enterprise use case that is heavily focused on IT, pharma, travel or any domain other than news.

However, in spite of all the limitations identified above, BLEU continues to be a basic metric used by most, if not all MT researchers today. Though, now, most expert developers regularly use human evaluation on smaller sets of data to ensure that they indeed have a true and meaningful BLEU. The MT community have found that supposedly improved metrics like METEOR, LEPOR, and other metrics have not really gained any momentum. BLEU and its flaws and issues are more clearly understood, and thus more reliable, especially if used together with supporting human assessments. Also, many buyers today realize that MT system performance on their specific subject domains and translatable content for different use cases matters much more than how generic systems might perform on news stories.

In upcoming posts in this series, we will continue to explore the issue of MT quality from a broad enterprise needs perspective. More informed practices will result in better outcomes and significantly improved MT deployments that leverage the core business mission to solve high-volume multilingual challenges more effectively.

A cleaner, possibly more polished, and shorter studio version of this post is available here.