The Trouble With Competitive MT Output Quality Evaluations

The comparative measurement and quality assessment of the output of different MT systems is a task that has always been something that is difficult to do right. Right, in this context means, fair, reasonable, and accurate. The difficulty is closely related to the problems of measuring translation quality in general, that we discussed in this post. This difficulty is further aggravated when evaluating customized and/or adapted systems, since doing this requires special skills and real knowledge of each MT platform, in addition to time and money. The costs associated with doing this properly make it somewhat prohibitive.

BLEU is the established measurement of choice, we all use, but it is easy to deceive yourself, deceive others, and paint a picture that has the patina of scientific rigor, yet be completely biased and misleadingly false. BLEU, as we know is deeply flawed, but we don't have anything better, especially for longitudinal studies, even though if you use it carefully, it can be useful in providing some limited insight in a comparative evaluation.

In the days of the NIST competitive evaluations, the focus was on Chinese and Arabic to English (News Domain) and there were some clear and well-understood rules on how this should be done to enable fair competitive comparisons. Google was often a winner (i.e. highest BLEU score), but they sometimes "won" by using systems that took an hour to translate a single sentence, because they evaluated 1000X as many translation candidate options as their competitors, to produce their best one. Kind of bullshit, right? More recently, we have the (WMT16) that attempts to go beyond the news domain, does more human evaluations, evaluates PEMT scenarios, and again controls the training data used by participants, to attempt to fairly assess the competitors. Both of these structured evaluation initiatives provide useful information if you understand the data, the evaluation process, the potential bias, but both are also flawed in many ways, especially in the quality and consistency of human evaluations.

One big problem for any MT vendor in doing output quality comparisons, with Google, is that for a test to be meaningful, it has to be with something that the Google MT system does not already have in its knowledge database (training set).  Google crawls news sites extensively (and the internet in general) for bilingual text (TM) data, so the odds of finding data they have not seen are very low. If you give a college student all the questions and the answers that are on the test, before they take the test, the probability is high that they will do well on that test. This is why Google generally scores better on news domain tests against most other MT vendors, as they likely have 10X to 1,000X the news data that anybody except for Microsoft and Baidu has. I have also seen MT vendors and ignorant LSPs show off unnaturally high BLEU scores, by having an overlap between the training and test set data. The excitement dies quickly, once you get to actual data you want to translate, that the system has not seen before.

Thus, when an MT technology vendor tells us that they want to create a lab to address the lack of independent and objective information on quality and performance, and create a place where “research and language professionals meet,”one should be at least a little bit skeptical, because there is a conflict of interest here, as Don DePalma pointed out. But, after seeing the first "fair and balanced" evaluation from the labs, I think it might not be over-reaching to say that this effort is neither fair nor balanced, except in the way that Fox News is. At the very least, we have gross self-interest pretending to be in the public interest, just like we now have with Trump in Washington D.C. But sadly, in this case, they actually point out that, even with customization/adaptation, Google NMT outperforms all the competitive MT alternatives, including their own. This is like shooting yourself in the hand and foot at the same time with a single bullet!

A List of Specific Criticisms

Those who read my blog regularly know that I regard the Lilt technology favorably, and see it as a meaningful MT technology advance, especially for the business translation industry. The people at Lilt seem to be nice, smart, competent people, and thus this "study" is surprising. Is this deliberately disingenuous, or did they just get really bad marketing advice to do what they did here?

Here is a listing of specific problems that would be clear to any observer who did a careful review of this study and its protocol.

Seriously, This is Not Blind Data.

The probability of this specific data being truly blind data is very low. The problem with ANY publicly available data is, that it has a very high likelihood of having been used as training data by Google, Microsoft, and others. This is especially true for data that has been around as long the SwissAdmin corpus has been. Many of the tests typically used to determine if the data has been used previously are unreliable, as the data may have been used partially, or only in the language model. As Lilt says:  "Anything that can be scraped from the web will eventually find its way into some of the (public) systems" and any of the things I listed above happening, will compromise the study. If this data or something very similar is being used by the big public systems, it will skew the results and result in erroneous conclusions. How can Lilt assert with any confidence that this data was not used by others, especially Google? If Lilt was able to find this data, why would Google or Microsoft not be able to as well, especially since the SwissAdmin corpus is described in detail in this LREC 2014 paper? 

Quality Evaluation Scoring Inconsistencies: Apples vs. Oranges

• The SDL results and test procedure seems to be particularly unfair and biased. They state that, "Due to the amount of manual labor required, it was infeasible for us to evaluate an “SDL Interactive” in which the system adapts incrementally to corrected translations." However, this unfeasibility does not seem to prevent them from giving SDL a low BLEU score.  The "adaptation" that was conducted was done in a way that SDL does not recommend for best results, thus publishing such sub-optimal results is rude and unsportsmanlike conduct. Would it not be more reasonable to say it was not possible, and leave it blank?
• Microsoft and Google released their NMT systems on the same day, November 15, 2016. (Click on the links to see). But Lilt chose to only use the Google NMT in their evaluation.
• SYSTRAN has been updating their PNMT engines on a very regular basis and it is quite possible that the engine tested was not the most current or best-performing one. At this point in time, they are still focusing on improving throughput performance, and this means that lower quality engines may be used for random, free, public access for fast throughput reasons. 
• Neither SYSTRAN nor SDL seems to have benefited from the adaptation, which is very suspicious, and should they not be given an opportunity to show this adaptation improvement as well?
• Finally, one wonders how the “Lilt Interactive” score is processed. How many sentences have been reviewed to provide feedback to the engine? I am sure Lilt took great care to put their own best systems forward, but they also seemed to have been less careful and even seem to have executed sub-optimal procedures with all the others, especially SDL. So how can we trust the scores they come up with?

Customization Irregularities

This is still basically news or very similar-to-news domain content. After making a big deal about using content that "is representative of typical paid translation work" they basically choose data that is heavily news-like. Press releases are very news-like and my review of some of the data suggests it also looks a lot like EU data, which is also in the training sets of public systems. News content is the default domain that public systems like Google and Microsoft are optimized for, and it is also a primary focus of the WMT systems. And for those who scour the web for training data, this domain has by far the greatest amount of relevant publicly available data. However, in the business translation world, which was supposedly the focus here, most domains that are relevant for customization are exactly UNLIKE News domain. The precise reason they need to develop customized MT solutions is because their language and vocabulary are different, from what public systems tend to do well (namely news). The business translation world tends to focus on areas where there is very little public data to harvest, either due to domain-specificity – medical, automotive, engineering, legal, eCommerce etc. or due to company-specific terminology. So, basically testing on news-like content does not say anything meaningful about the value of customization in a non-news domain. What it does say is that public generic systems do very well on news, which we already knew from years of WMT evaluations which were done with much with more experimental rigor and more equitable evaluation conditions.

Secondly,  the directionality of the content matters a lot. In “real life”, a global enterprise generates content in a source language where it is usually created from scratch by native speakers of that language and needs it translated into one or more target languages. Therefore, this is the kind of source data that we should test if we are trying to recreate the localization market scenario.  Unfortunately, this study does NOT do that (and to be fair this problem infects WMT and pretty much the whole academic field – I don’t mean to pick on Lilt!). The test data here started out as native Swiss German, and then was translated into English and French. In the actual test conducted, it was evaluated in the English⇒French and English⇒German direction. Which means that the source input text was obtained from (human) translations, NOT native text. This matters. Microsoft and others have done many evaluations to show this. Even good human translations are quite different from true native content. In the case of English⇒French, both the source and the reference is translated content. 

There is also the issue of questionable procedural methodology when working with competitive products. From everything I gathered in my recent conversations with SDL, it is clear, that adaptation by importing some TM into Trados is a sub-optimal way to customize an MT engine in their current product architecture. It is even worse when you try and jam a chunk of TM into their adaptive MT system, as Lilt also admitted. One should expect very different, and sub-optimal outcomes from this kind of an effort since the technology is designed to be used in an interactive mode for best results. I am also aware that most customization efforts with phrase-based SMT involves a refinement process, sometimes called hill-climbing. 

Just throwing some data in, and taking a BLEU snapshot, and then concluding that this is a representative outcome for that platform is just wrong and misleading. Most serious customization efforts require days of effort at least, if not weeks to complete, prior to a production release.

Another problem when using human translated content as source or reference is that in today’s world, many human translators start with a Google MT backbone and post-edit. Sometimes the post-edit is very light. This holds true whether you crowd-source, use a low-cost provider such as unBabel (which explicitly specifies that they use Google as a backbone), or a full-service provider (which may not admit this, but that is what their contract translators are doing with or without their permission). The only way to get a 100% from-scratch translation is to physically lock the translator in an internet-free room! We already know for the multi-reference data sets, that there are many equally valid ways to translate a text. When the “human” reference is edited based on Google, the scores naturally favor Google output. 

Finally,  the fact that the source data starts as Swiss German, rather than regular German may also be a minor problem. The differences between these German variants appear to be most pronounced when it is spoken rather than written, but Schriftsprache (written Swiss German) does seem to have some differences with standard high German. Wikipedia does state that: "Swiss German is intelligible to speakers of other Alemannic dialects, but poses greater difficulty in total comprehension to speakers of Standard German. Swiss German speakers on TV or in films are thus usually dubbed or subtitled if shown in Germany."

 Possible Conclusions from the Study

All this suggests that it is rather difficult for any MT vendor to conduct a competitive evaluation in a manner that would be considered satisfactory and fair to, and by, other MT vendor competitors. However, the study does provide some useful information:

• Do NOT use News domain or news-like domain if you want to understand what the quality implications are for "typical translation work".  
• Google has very good generic systems, which are also likely to be much better with News domain than with other specialized corporate content.
• Comparative quality studies sponsored by an individual MT vendor are very likely to have a definite bias, especially on comparing customized systems.
• According to this study, if these results were indeed ACTUALLY true, there would little point to using anything other than Google NMT.  However, it would be wrong to conclude that using Google would be better than properly using any of the customized options available since except for Lilt, we can presume they have not been optimally tuned. Lilt responded to my post comment on this point saying, "On slightly more repetitive and terminology-heavy domains we can usually observe larger improvements of more than 10% BLEU absolute by adaptation. In those cases, we expect that all adapted systems would outperform Google’s NMT."
• Go to an independent agent (like me or TAUS) who has no vested interest other than to get accurate and meaningful results, which also means that everybody understands and trusts the study BEFORE they engage. A referee is necessary to ensure fair play, in any competitive sport as we all know from childhood.
• It appears to me (only my interpretation and not a statement of fact) that Lilt's treatment of SDL was particularly unfair. In the stories of warring tribes in human literature, this usually is a sign that suggests one is particularly fearful of an adversary.  This intrigued me, so I did some exploration and found this patent which was filed and published years BEFORE Lilt came into existence.  The patent summary states: "The present technology relates generally to machine translation methodologies and systems, and more specifically, but not by way of limitation, to personalized machine translation via online adaptation, where translator feedback regarding machine translations may be intelligently evaluated and incorporated back into the translation methodology utilized by a machine translation system to improve and/or personalize the translations produced by the machine translation system."  This clearly shows that SDL was thinking about Adaptive MT long before Lilt. And, Microsoft was thinking about dynamic MT adaptation as far back as 2003. So who really came up with the basic idea of Adaptive MT technology? Not so easy to answer, is it?
• Lilt has terrible sales and marketing advisors if they were not able to understand the negative ramifications of this "study", and did not try to adjust it or advise against publicizing it in its current form. For some of the people I talked to in my investigation, it even raises some credibility issues for the principals at Lilt.

 I am happy to offer Lilt an unedited guest post on eMpTy Pages if they care to, or wish to, respond to this critique in some detail rather than just through comments. In my eyes, they attempted to do something quite difficult and failed, which should not be condemned per se, but it should be acknowledged that the rankings they produced are not valid for "typical translation work".  We should also acknowledge that the basic idea behind the study is useful to many, even if this particular study is questionable in many ways. I could also be wrong on some of my specific criticisms, and am willing to be educated, to ensure that my criticism in this post is also fair. There is only value to this kind of discourse if it furthers the overall science and understanding of this technology, and my intent here is to question experiment fundamentals, and get to useful results, not bash on Lilt. It is good to see this kind of discussion beginning again, as it suggests that the MT marketplace is indeed evolving and maturing.

Peace.

P.S. I have added the Iconic comments as a short separate post here to provide the perspective of MT vendors who perform deep, careful, system customization for their clients and who were not included directly in the evaluation.

The Missed Opportunity of Translation Management Systems

Recently, I had been having a discussion with a TMS vendor about potential new directions for  Translation Management Systems to evolve in, when Luigi Muzii, in an independent and unrelated interaction, suggested this new post based on some of his own observations about TMS systems. So based on this unplanned synchronicity, I felt it would be good to air this. I invite others, especially from the TMS vendor community to come forward with their perhaps more informed views.

It has seemed to me that TMS systems in general, focus largely on the translation management problems of yesteryear, rather than the current ones, let alone emerging needs where there is a much greater need for intelligent data technology, beyond translation memory. The data in most TMS systems is in what I would call a dumb repository. The data is tagged by project and customer, but rarely by more extensive metadata that would be valuable to leverage it for future translation projects. No TMS today can help you extract, for example, all the TM related to hybrid automobiles in your total data repository or identify key terminology for hybrid automobiles from the total repository. Look at what is possible with Linguee.  Why are these TMS systems not able to do this, at least on your own data? Data could be catalogued at various levels of granularity by PMs who are not chasing errant freelancers down so relevant chunks can be retrieved when needed. This is what I mean by intelligent data, which understands the relative importance of phrases, some semantics and even relative overlap of ALL data resources with new project work. It should be able to respond to reasonable questions that involve a combination of words/phrases as any supportive automation should. The kinds of tools we use will confine us to certain kinds of work (software and documentation translation) if you want to solve big translation problems like eBay, Microsoft, Facebook is doing, you are going to need better tools. 

MT involves the use and management of large volumes of translation memory and other linguistically relevant data and it would be very useful for TMS systems to provide linguistic data analysis, manipulation, and transformation capabilities. Many of these corpus tools already exist in open source as Juan Rowda has pointed out, but it would be good, and of great value to any global enterprise and large agency users, to have these kinds of tools closely integrated so that they can easily interact with large-scale TM repositories.  The future TMS system needs to be much more useful to, and supportive of, large-scale translation projects that involve tens of millions of words, and both MT and post-editor feedback, as these kinds of projects, will become much more prevalent. No TMS tools I know would be helpful to do normalization and standardization of MT training data even though they keep all of the TM. It would also be very useful for next-gen systems to also provide a sense regarding the relative similarity or dissimilarity of subsets of data

It also seems somewhat archaic that TMS systems still need to break large TEP projects into smaller translation packages to hand off to individual freelancers and does not have a more elegant form of collaborative translation where this disassembly and re-assembly is NOT necessary. SmartCAT is refreshing and interesting because it solves this very basic problem and is also free. The translation collaboration outlook is still pretty bleak but I think better collaboration is one area that TMS vendors should really focus on and make more easily implementable. 

 I don't speak German so not really sure what this says, but I really like the question being asked.😁


What are the most important areas for Translation management systems to develop in future?  Here is a list I came up with as potential areas for improvement, which is easy for me to dream up since I don't have to actually do it.

• Better database management so that systems can handle tens of millions of segments and can slice and dice these segments in every which way as needed.
• Data anonymization and randomization capabilities so that TM can be leveraged for new kinds of translation problems without compromising data security or privacy.
• NLP capabilities to help understand data characteristics at a corpus level and do the kinds of things that Linguee and Google allow you to e.g. predictive and auto-guessing words and phrases.
• Translation collaboration that reduces the burden on project managers and actually frees them from policing roles to handling policy and overall quality driving processes.
• Better integration with MT so corpus can be modified and prepared to raise quality of results and so that PEMT can be better integrated into advanced learning processes
• More Flexible Workflow Design capabilities so that every client is not forced into a one size fits all mode of operation.
• More automation of basic project management functions so human involvement is focused only on special exception situations.
• More measurement metrics on job efficiency and expanding Key Performance Indicators by JobID, PM, language, client etc..

 The emphasis below in Luigi's post is all mine.

-----------------------------------------
↓↓↓↓↓↓↓↓↓↓

I have observed that people in the translation community, no matter how high they rank in their functional area or organization, are generally reactive rather than proactive. The latest financial events where Private Equity is moving in (i.e. Moravia, ULG, and LIOX) are a further confirmation of this attitude, although the recent history of the industry should have enough to tell, for any unbiased reader.

This reactive attitude results in a situation that industry outsiders are forced to introduce innovation into a field other than their core business because of the inability of their translation business partners to do it. And thus we see that these outsiders end up driving the "translation industry" forward and eventually the industry is ruled by customers rather than the original industry players.

While basic business and production processes have remained nearly unaltered for centuries,  technological development has been brought in and implemented following the request, when not actually the directive, of the most affluent, and tech-knowledgeable customers. In most cases, the most significant advancements, and indeed improvements, have been pioneered by those customers; technology providers have followed suit, rarely with striking results. In almost three decades, progress in translation has mostly consisted of some trivial outcome of the amelioration frenzy of software applications exploiting long-established technologies, with the claim, every time, to revolutionize the industry and the profession.

An examination of the translation management systems (TMS) in the market is quite representative of this behavior. TMSs are supposed to be the equivalent of workflow management systems (WFMSs), which date back to the early 1990s if you look at the broader business market. Curiously, or possibly not, TMSs generally lack most of the features of a typical mature full-fledged WFMS that can be found in virtually any business of some significance.

Lionbridge’s current CEO, Rory Cowan, summarized the at best, disappointing, NASDAQ experience of the company he has been leading for quite a few years, with a statement that is as simple as it is sad: “The U.S. markets really do not appreciate the translation industry.”

As a matter of fact, none of the top-ranking translation companies is an innovation champion, nor is it yet remotely comparable, financially, technologically, structurally, and operational sophistication, to even mid-level companies in most other service industries.

Typically, a WFMS is an application framework for building, managing and automating, as much as possible, a set of tasks forming one or more business processes. Some tasks may require human intervention, such as the development of content and/or its approval. Also, the system must allow users to introduce new tasks or describe new processes and typically comes with some form of process flow designer to create new workflow processes and applications through it.

On this basis, WFMS must allow users to define different workflows for different types of jobs or processes, associate one individual or group for any specific task at each stage in a workflow and establish dependencies for process management from top to lower level.

WFMSs should also allow for application integration to form a middleware framework.

Finally, WFMS should also consist of a project management module to help plan, organize, and manage estimation and planning, scheduling, cost control and budget management, resource allocation, collaboration, communication, decision-making, quality management and administration. Similarly, any TMS should enable translation businesses and customers to control their projects and monitor every task according to an established and agreed upon workflow.

A TMS is supposed to allow for automating all or at least most of the translation process while maximizing its efficiency by having repeatable and non-essential work be done by software. However, most state-of-the-art TMSs only provide some basic translation management functions. The thing is that no two people in the translation community would agree on what exactly the creative part of the process is, that can’t be left to machines.  And sadly, no two linguists can even come to an agreement and be able to describe and define the special skills and abilities that define and dignifies themselves as knowledge professionals.

Contrary to what happens in almost every other type of business, no TMS still exists with all the features described above. Specifically, all lack a comprehensive graphical workflow designer. Only one or two provide some very basic capabilities.

So, forget about Leibnitz[*].

As a matter of fact, most players in this industry are generally oblivious or at least uncomfortable with simple diagrams like activity diagrams and even flowcharts, which are graphical representations of workflows or processes. Although pertaining at large to the field of software engineering, modeling languages have long been established this approach as a standard way to visually describe the business and operational step-by-step activities in a system. However, most translation people are still incapable of envisaging a slightly more articulate process than the typical middleman-based agency model.

If this were not bad enough, translation project management is way far afield from real project management, and, not surprisingly, only a small bunch of translation/localization project managers hold a PMI certification.

The supposed and long-touted peculiarity of translation, together with the utter and thick rejection of any real business focus, despite the beloved acquaintance with rates and any relevant discussion related to these rates, is possibly a reason why TMSs are so estranged to WFMS and ERP.

To allow users always to find the best solution and resource as fast as possible a TMS should first be a central repository for language data and the vendor database. The aim of a TMS should be helping users cut out middlemen from the procurement process while enabling project managers to plan the best schedule and budget and pick the best resources. In such a way, TMSs are also expected to reduce overhead and hence costs and time.

Therefore, any current TMS with one-fits-for-all process model is somewhat problematic at least. Customers should be able to design their workflow and control where, when and how much human involvement is requested or needed for differing project needs.

Indeed, TMSs might have a dramatic impact when collaborative translation is effectively implemented, through parallelized process, but, for the same reasons why workflow graphical design features are almost absent from TMSs, real collaborative translation features are yet to come too.

Finally, most TMSs have already moved or are being moved into the cloud; however, while reducing capital expenditure and allowing for faster software deployment, the SaaS model presents some serious drawbacks in lower flexibility and higher customization costs. Of course, the features offered by TMSs depend on the level of sophistication, but the technical intricacies of such TMSs preclude them to be adopted by most LSPs and even by most customers.

TMSs might have been the means to lead customers rather than letting them dictate, but the chance has been wasted. So far.

[*] It is unworthy of excellent men to lose hours like slaves in the labor of calculation which could safely be relegated to anyone else if machines were used.

---------------------------

Luigi Muzii has been in the "translation business" since 1982 and has been a business consultant since 2002, in the translation and localization industry through his firm. He focuses on helping customers choose and implement best-suited technologies and redesign their business processes for the greatest effectiveness of translation and localization related work.

This link provides access to his other blog posts.

A Closer Look at SDL's Adaptive MT Technology

I recently sat with Samad Echihabi, VP Research & Product Development at SDL's MT Research facility in LA to be briefed on their Adaptive MT technology. The team in LA originates from the core Language Weaver group, but the complete team also includes members in Cambridge, UK, and Cluj, Romania. The research team that Samad leads has always been on the forefront of MT research, in spite of multiple rounds of upheaval and management changes around the team. In this post I have gathered product specific information from Samad and his team, and strategy and vision related information from public presentations, given by the new SDL senior management, to piece together a more complete view of what this new technology initiative might mean.

Senior Management Commitment & Support

Earlier this year, I had watched the "100 days" presentation given by Adolfo Hernandez and the "new SDL" management team in Fall 2016, and this piqued my interest in SDL MT technology again. It is always refreshing to see somebody look at an old challenge with new eyes. It was clear that the new management saw leading-edge MT technology being a core business driver, and seem to be serious about putting SDL's MT back on track, and correcting the MT marginalization that previous management had created. It was interesting to hear the new team respond to investment analysts who questioned this new emphasis on MT when the actual direct MT revenue was so shrunken. They repeatedly explained that they had "a different view of MT than the old regime".  However, as the following slides from that "interim results" presentation show, the new team sees a bigger and foundational role for MT and see it as an enabling technology for forward business momentum.

 

Current SDL MT Technology

The SDL MT team has a whole suite of NLP tools ready for action, in addition to their MT technology. These tools can be used to prepare and transform linguistic data for varying language-related project needs that may or may not be related to MT. Samad provided an overview of their existing MT technology which has both on-premise (SDL ETS) and cloud-based (SDL BeGlobal) solutions that are robust and field-tested by a largely enterprise and government user base. He demonstrated how easily a remote console could setup a single server or a 100 servers within minutes. This was easily the most straightforward setup and MT server installation process I have seen. Some installations can take days to set up, and require vendor technical experts to even attempt it.

The cloud architecture is also very flexible and built to be scalable and elastic to adapt to widely varying workloads e.g. when large knowledge bases are re-translated with a major engine update. The architecture is also containerized. (Containers are a solution to the problem of how to get software to run reliably when moved from one computing environment to another. This could be from a developer's laptop to a test environment, from a staging environment into production and perhaps from a physical machine in a data center to a virtual machine in a private or public cloud. I am told this is important and this matters.)

 

Adaptive Machine Translation

Ever since I first saw a translator working with Lilt, early in 2016, I have been a fan of Adaptive MT and felt this was an approach that made so much sense, since so many MT projects stumble at the customization phase. In Adaptive MT engines, generally, the engines are learning as you translate, not via a batch training process managed by an MT vendor, like in most other MT systems. I think it is fair to say it is a next generation approach to post-editing as well, as corrective edits teach the system and the editing task is not mindlessly repetitive. It is also a much better solution for those agency projects that are data-starved, or just needed a quick turnaround without a long drawn out customization process that can take months to complete and still fail.

I am confident that Adaptive MT is a better approach for an agency or freelancer than any do-it-yourself Moses solution, and I think it could become the MT technology that is the most pervasive in the professional translation world because it learns dynamically, is linguistically steered, and improved by ongoing human corrective feedback. Also, unlike NMT it is not a resource hog. There is always the possibility that a highly responsive neural MT technology could emerge and prevail, but I am doubtful that this is possible in the near term (no sooner than 2018 at the earliest).  The Adaptive MT user experience is more like having a virtual assistant who learns to be a better assistant as you work together. The SDL Adaptive MT will work with generic (baseline), vertical (industry specific like IT or Life Sciences) and unique domain-adapted custom engines. This ability to shape and control the starting point Adaptive MT baseline, is a powerful feature, as it theoretically accelerates the system learning curve, and users can reach higher productivity thresholds faster. If properly implemented and used, adaptive MT engines will continue to improve continuously with ongoing use.

 

The SDL Eco-System Advantage?

Adaptive MT is available today from Lilt and SDL, and soon from ModernMT, an EU-funded open source initiative with heavy involvement from Translated.net. After years of being dormant and only showing small incremental progress and many failures from Moses madness, the MT market is humming again with Neural MT and Adaptive MT led momentum. Interestingly, SDL was awarded patents in 2015 for online domain adaptation, and for personalized MT via online adaptation, so it is possible that they were thought leaders on the basic idea of Adaptive MT who just did not get around to implementing a working solution to be the first player in the market, possibly because of differing priorities in the senior management team at the time.

SDL's Adaptive MT is tightly integrated with the new Trados Studio 2017 that provides a  richer interface into the MT engine than Lilt. This includes upLIFT Fragment Recall and upLIFT Fuzzy Repair, plus all the legacy Trados functionality to leverage and support the translator user who can also employ a learning MT engine to support him/her.  (upLIFT is a powerful concordance search that unlocks productivity gains from no-matches and fuzzy-match segments in new ways.) Massimo Ghislandi gave me a demo showing how power Trados features seamlessly integrate with the Adaptive MT, including something called Autosuggest, that leverages available data to assist with sub-segment pre-translation options. In some ways, this is similar and equivalent to the interactive MT component of Lilt. SDL will have both an Enterprise and an Independent Freelancer approach to their Adaptive MT technology which at this moment is limited to 5 languages (more coming soon as Adaptive MT requires different supportive data/software infrastructure than standard SMT). You get an Adaptive MT engine for free with a new Trados license. At the moment you do need Trados to interact with the Adaptive MT engine but they are looking at other options including integration with third party interfaces.  

Massimo suggested that the feature rich Trados would be a much-preferred interface for the kinds of power translators who would most likely engage with Adaptive MT. The Lilt interface is elegant and light but it does require that a translator adapts to a new kind of translation interface and approach. A new approach makes it really hard to get widespread adoption, (as Lilt knows), as old translator dogs don't learn new tricks easily. Translation Memory is really the only CAT tool that has widely taken root with translators to date. To date, nobody has succeeded in selling MT in volume to freelancers, and I am not sure the initial approach SDL is taking is going to be successful either, even though the technology does indeed make sense for a power translator, or any translator who learns to build and use the Adaptive MT productivity boost. The project management infrastructure that surrounds Trados also allows it to be used for large-scale enterprise projects, but I believe there is a need for a cloud-based post-editor interface that can allow easy consolidation of distributed editing work and supervision and monitoring of groups working on multi-million-word projects. Yesterday's tools don't always work automatically for today's problems.

The Adaptive MT Benefit

Samad described some evaluation tests he ran to assess how much benefit the Adaptive MT provided over and above any leverage provided by Standard MT. His careful test process and conservative estimates to isolate the Adaptive MT benefit suggested that this number is from 5% to 25% for the very specific tests he described. The 5% to 25% is improvement on top of Standard MT without taking into account any productivity gain from other Trados features. While most in the MT industry tend to overstate benefits of the technology, here is a case where the benefit is understated, and careful not to over-promise, in contrast to the recent Google hyperbole. I am also going to bet that the SDL human evaluations done by in-house translators, are much more meaningful and accurate than the Google mechanical turk based human evaluations. Based on how conservative and strict his test procedure was, I would expect that the real benefit to a customer is much higher. The latest Trados version produces a table (shown below) that quantifies the benefit of TM and Adaptive MT but the calculation is kind of a mystery, and SDL will hopefully clearly explain how they come up with this number if they want it to be trusted and taken seriously.

SDL has the potential to be a dominant player in this segment of the market i.e. the enterprise and professional translation segment with their MT product suite and especially the Adaptive MT. Some reasons why I say this include:

• SDL is today translating 20B words a month for customers with MT and 100M words a month via traditional TEP. (This means that 95% of what SDL translates on a monthly basis is done by computers). This suggests that they know how to do both, and they already have established that in-house MT use improves margins (2% in fact as they reported on 12/6/2016 though I suspect it is higher). 
• They have a deeper reach into the enterprise than any other MT vendor and could possibly identify NEW high-value MT opportunities more easily.
• They have a market leading TM product for freelancers, with a large installed base that could be engaged to actively use Adaptive MT with the right inducements e.g. free.
• They have the most complete translation production eco-system that potentially could be integrated into a formidable whole, even though they have been incapable of doing this historically. ( I imagine Lionbridge and TransPerfect will disagree, but I am quite certain the SDL MT systems are of a higher caliber and LIOX and Transperfect have other concerns to attend to at this time). This eco-system includes content management software, translation management systems, MT, NLP and translation productivity tools in addition to competence with managing human translation projects at scale. 

• While SDL cannot claim that all their software components are best-of-breed, one would expect that SDL does have a deep understanding of how key components need to tie together to build real market-leading translation process automation that customers value. The new management team also seems much more open to creating "connectors" to allow content to flow between content creation to/from translation processes and on to dissemination. Robust support of data ingestion from other (non-SDL e.g. Adobe) software environments is a critical requirement for many global enterprises and one that can often be a deal breaker when they select core business systems. Hopefully, this openness to connectors will also extend to MemoQ, Memsource and across, as this level of openness would benefit and encourage many potential customers and leverage the SDL MT offerings in particular. "Openness" that requires that you only use SDL components is not really open, and as Microsoft learned, does not work. The translation industry is sorely in need of real openness so that enterprises can easily link translatable content to a variety of translation-related processes.

However, historically we have also seen that SDL does not have a good track record of being able to tie it all together into an elegant, properly integrated market-dominating solution.  Historically their software integration has been messy and problematic from what I could gather. The management team has changed and so the possibility of real synergy exists again, but it is actually too early to tell. Software acquisitions are messy for any company, as each software product often comes with a confining worldview and architectural barriers that make it hard or even impossible to build bridges and create a new more complex, combined and powerful structure. The new team has been in place for barely 9 months so again too early to tell but the story they are telling is more believable (to me at least).

It is said that the companies that are able to build a brand image that is "liked" tend to rise and win market share simply because they are authentic, transparent, reliable, cool and likable e.g. FaceBook, Apple, Harley Davidson, Google, Sony, Amazon, BMW, Tom's. Here is a ranking that perhaps explains why this matters and compare it to the Forbes list which ranks by value and has companies like IBM (the creators of the  FUD approach) in their top 10. (When was the last time you heard somebody say, "Boy, I sure loooves me some IBM." ?)   Historically, I have noticed that SDL has the opposite problem, they are often actively disliked and distrusted by customers, partners, competitors and others in the industry. This is a problem that the new management has to fix as well, as it also is, in my opinion, an important determinant of future success. To me, this means to communicate in an authentic voice, not corporate marketing-speak which has some information value but does not generally build enthusiastic customers and partners who also sell effectively on your behalf.

SDL at one time had a strong MT technology operation that briefly had the largest enterprise MT revenue base in the market (i.e. > SYSTRAN), which was, unfortunately, close to decimated by mismanagement and neglect by previous senior management. Some might even say that the MT industry has had greater degree of mismanagement and poor commercialization strategy challenges than technology related ones.

SDL looks well positioned with their Adaptive MT technology, and I think that both agency and freelance clients could benefit from aggressive use (not casual, occasional use) of this technology. Both Adaptive MT and NMT have the potential to help build enduring production efficiencies that will be difficult to match with older technology, and I expect that those LSPs and freelancers who learn to properly use these technologies will emerge victorious in some way. SDL is the ONLY LSP with the expertise and a bench deep enough, in my opinion, to handle developing NMT solutions (but I am sure many other LSPs will also waste time and resources, and produce crappy sub-optimal systems).  However, SDL also has a history of losing or lagging in games where they seemed to be clear favorites. The "new SDL" management is much easier to listen to, and hopefully they execute as well as they talk. I will continue to share information about the evolution of this technology as I am able to gain from my vantage point, and hopefully, SDL will remain as open to share as they have been thus far. People buy technology they understand, and thus far the MT vendor industry has not done a great job of explaining the what, why, where, and have focused mostly on overly technical descriptions of  the "how" of this technology.

Anyway, 2017 looks like it will be an exciting and memorable year for MT. Wishing you all great success in your endeavors.

-----------------------------

Samad Echihabi is Vice President of Research and Product Development at SDL. He heads the SDL Machine Translation Research and Development group and manages the end-to-end development life cycle of SDLs MT products with a focus on customer-driven technology and science advancement. Samad has extensive knowledge in MT and over 15 years of experience in Software Development at the intersection of Machine Learning and Data Science. Samad is a Fulbright scholar and a graduate of University of Southern California.