Localization and Language Quality

This is a guest post by David Snider, Globalization Architect at LinkedIn - Reprinted here with permission. I thought the article was interesting because it points out that MT quality is now quite adequate for several types of Enterprise applications, even though MT might very well be a force that influences and causes the "crapification" (a word I wish I had invented) of  overall language quality. While this might seem like horror to some, for a lot of business content that has a very short shelf life, and value only if the information is current, this MT quality is sufficient for most of the people who have in interest in the specific content. While David thinks that the language quality will improve, I doubt very much if much of this MT content will improve much beyond what is possible by the raw technology itself. Business content that has value for a short time and then is forgotten simply cannot justify the effort to raise it the level of "proper" written material.

If you go to the original post there are several comments that are worth reading as well.

-------

People have been complaining recently about the decline of language quality (actually, they’ve been complaining for decades – or make that centuries!)  I have to admit that I sympathize: I’m from a generation that was taught to value good writing, and I still react with horror when I see obvious errors, like using “it’s” instead of “its”, or confusing ‘their’, ‘there’ and ‘they’re’.  (I’m even more horrified when I make mistakes myself, which happens more than I like to admit.)
But for my son’s generation? Not so much. Grammar, spelling, and punctuation aren’t that important to them; what matters is whether the other person understands them, and vice versa. My son is already 25 (wow, time flies!), so there’s another generation coming up behind him that’s even less concerned about ‘good’ writing; in fact, this new generation is so accustomed to seeing bad writing that for the most part they don’t even realize there are errors.  This makes for a vicious circle: people grow up surrounded by bad writing, so they, in turn write badly, which in turns exacerbates the problem. I’ve heard this referred to as the ‘crapification of language’.

 

Why is this happening?

 

Ease of publishing: in the old days, the cost of publishing content - typesetting it, grinding up trees and making paper, printing the content onto the paper, binding it, shipping it to a store and selling it - was immense. For this reason most published content was thoroughly edited and proofread, as there was no second chance. So if you read printed content like books, magazines and newspapers, you were generally exposed to correct grammar, spelling and punctuation. Since most of what people read was correctly written (even if not always well-written), people who read a lot generally learned to write well. But now anyone can create and publish content, with no editing or proofreading. The result is just what you’d expect.

Informal communications: email, texting, twitter – they all favor speed, and when people are in a hurry quality usually suffers.

Machine-generated content: this includes content that’s created by computers – for example, Machine Generated support content created by piecing together user traffic about problems – as well as Machine Translated content. Machine Generated content, and especially MT content is, as we localization people know, often of very poor quality.

What does this mean for Localization?

 

Being in the localization business myself, I want to tie this in to the effect on localization. In some ways this ‘crapification’ works against us: garbage in garbage out, after all, and if the source content is badly written then it’s harder for the translators to do a good job, be they humans or machines. But at the same time, this can work for us – especially when it comes to Machine Translation, where there are a couple of things that are making even raw MT more acceptable:

MT engine improvements: MT quality has steadily improved over the past 50 years (yes it’s been around at least that long!) Major improvements, like statistical MT and now neural MT, seem to occur every 10 years or so. Perfect human-quality MT is still ‘only 5 years out’ and will undoubtedly continue to be so for a long time, but quality is steadily improving.

User expectations: The good news for MT is that due to the crapification of language the expectations bar has been coming down, and people are much more willing to accept raw MT, warts and all. Despite the quality problems, more & more people are using web-based MT services like Google Translate, Bing Translator, etc., to read and write content in other languages.  As with texting above, they’re more concerned with content than with form: they’re OK with errors as long as they can understand the content or at least get the gist of it. This seems to be true even for countries that have traditionally had a high bar for language quality, like Japan and France. As shown in the chart below, we’ve already passed the point that raw MT is acceptable for some types of content. (Note that this chart is purely illustrative and is not based on hard data.)

Of course the bar remains high for things like legal documents, marketing content and of course your own personal homepage, but it’s getting lower for many other types of content, especially for things like support content (which many companies have been MTing for years), as well as for blogs and other informal content. In fact, the graph could be redrawn something like this:

 

Is there any hope for language quality?

 

As the quality of machine-generated and machine-translated content improves and as editing and proofing tools become better and more ubiquitous, the quality of all content will improve, until we approach the days of professionally edited and proofread books and magazines. As bad writing disappears and people grow accustomed to seeing well-written content, I think even unedited human language quality will start to curve back up again. (I’ve tried to capture this in the graphs above.)
So yes, I believe the crapification of language will slow and eventually reverse itself (hmm, unpleasant plumbing image there)! This doesn’t mean languages won’t continue to evolve, fortunately. That’s one of the things that make them so fascinating – and so challenging to translate.

Some key excerpts from the comments at the original post are listed below:

David Snider: The crapification 'helps' localization teams by allowing them to put more assets in the 'good enough' bucket. The real question for raw MT is: "is my customer better off having to fight their way through a badly-translated MT article and maybe get the help they need, or are they better off not getting the help at all?"

Jorge Russo dos Santos :I disagree that we will see content revert to a golden age of quality. I think, that we see today, there are different quality levels for content and that will continue to appear. If anything, the tolerance to poorly written content will probably increase, as people consume more and more content, but there will pockets where people will require premium content and will be willing to pay for it, either in the original languages or on the localized language(s) and this will not be only for legal.

Please go to the original posting to see all the comments at this link.

  David Snider, Globalization Architect at LinkedIn

Comparing Neural MT, SMT and RBMT – The SYSTRAN Perspective

This is the second part of an interview with Jean Senellart (JAS) , Global CTO and SYSTRAN SAS Director General. The first part can be found here: A Deep Dive into SYSTRAN’s Neural Machine Translation (NMT) Technology .

Those translators who accuse MT vendors of stealing or undermining their jobs should take note that SYSTRAN is the largest independent MT vendor. A position it has held for most of its existence, but has never generated more than €20 million in annual revenue. Which to me suggests that MT is mostly being used for different kinds of  translation tasks and is hardly taking any jobs away. The problem has been much more related to unscrupulous or incompetent LSPs who used MT improperly in rate negotiations. MT is hugely successful for those companies who find other ways to monetize the technology, as I pointed out in  The Larger Context Translation Market.This does suggest that MT has huge value in enabling global communication and commerce, and even in its less than perfect state is considered valuable by many who might otherwise need to acquire human translation services. If anything, MT vendors are the ones that are trying hardest to develop technology that is actually useful to professional translators as the Lilt offering shows and as this new SYSTRAN offering also promises to be. The new MT technology is reaching a point where it is becoming rapidly responsive to corrective feedback and thus much more useful to professional translation use case scenarios. The forces affecting translator jobs and work quality are much more complex and harder to pin down as I have also mentioned in the past.

In my conversations with Jean, I realized that he is one of the few people around in the "MT industry", who has deep knowledge and production use experience with all three MT paradigms. Thus, I tried to get him to share both his practical experience-based and philosophical perspectives about the three approaches. I found his comments fascinating and thought that it would be worth highlighting them separately in this post. Jean (through SYSTRAN) is unique in being one of the rare practitioners around, who has produced commercial release versions, of say French <> English MT systems, using all three MT paradigms. More if you count the SPE and NPE “hybrid” variants where more than one approach is used in a single production process.

Again in this post, I have tried as much as possible to keep Jean Senellart’s direct quotes in here to avoid any misinterpretation. My comments are in italics when they do occur within his quotes.

 

Comparing The Three Approaches; The Practical View

Some interesting comparative comments made by Jean about his actual implementation experience with the three methodologies (RBMT, SMT, NMT):

“The NMT approach is extremely smart to learn the language structure but is not as good at memorizing long lists of terminology as RBMT or SMT was. With RBMT, the terminology coverage was right, but the structure was clumsy – with SMT or SPE, we had an “in-between” situation where we got the illusion of fluency, but sometimes at the price of a complete mistranslation, and a strange ability to memorize huge lists but without any consideration of their linguistic nature. With NMT, the language structure is excellent, as if the neural network really deeply understands the grammar of the language – and introducing [greater] support of terminology was the missing link to the previous technologies.”

(This inability of NMT to handle large vocabulary lists is considered one of the main weakness of the technology currently. Here is another reference discussing this issue. However, it appears that SYSTRAN has developed some kind of a solution to address this issue.)

“What is interesting with NMT is it seems far more tolerant than PBSMT (Phrase-Based SMT) to noisy data. However, it does need far less data than PBSMT to learn – so we can afford to provide only data for which we know we have a good alignment quality. Regarding the domain or the quality [of MT output of] the languages, we are for the moment trying to be as broad as possible [
rather than focusing on specialized domains].”

In terms of training data volume, JAS said: “This is still very empirical, but we can outperform Google or Microsoft MT on their best language pairs using only 5M translation units – and we have a very good quality (BLEU score about 45** ) for languages like EN>FR with only 1M TU. I would say we need 1/5 of the data necessary to train SMT. Also, generic translation engines like Google or Bing Translate are using billions of words for their language models, here we need probably less than 1/100th.”

(**I think it bears saying that I fully expect that the BLEU here is measured with great care and competence, unlike what we see so often with Moses practitioners and LSPs in general who assume scores of 75+ are needed for the technology to be usable.

The ability of the new MT technology to improve rapidly with small amounts of good quality training data and small amounts of corrective feedback suggests that we may be approaching new thresholds in the use of MT for professional use.)

 

Comparing RbMT, SMT, and NMT: The Philosophical View

When I probed more deeply into the differences between these MT approaches, (since really SYSTRAN is the only company who has real experience in all 3), JAS said: “I would more compare them in terms of what they are trying to do, and on their ability to learn.” His explanation reflects his long-term experience and expertise and is worth careful reading. I have left the response in his own words as much as possible.

“RBMT, fundamentally (unlike the other two), has an ulterior motive: it attempts to describe the actual translation process. And by doing that, it has been trying to solve a far more complicated challenge than just machine translation; it tries to decompose [and deconstruct and analyze] and explain how a translation is produced. I still believe that this goal is the ultimate goal of any [automated translation] system. For many applications, in particular, language learning, but also for post-editing, the [automated ] system would be far more valuable if it could produce not only the translation but also explain the translation.”

“In addition, RBMT systems are facing three main limitations in language [modeling] which are:

1) the intrinsic ambiguity of language for a machine, which is not the same for a human who has access to meaning [and a sense for the underlying semantics],

2) the exception-based grammar system, and

3) the huge, contextual and always expanding volume of terminology units.”

“Technically, RBMT systems might have different levels of complexity depending on the linguistic formalism being used, making it hard to compare with the others (SMT, NMT), so I would rather say that one of the main reasons for the limitations of a pure RBMT system lies in its [higher reaching] goal. The fact is that fully describing a language is a very complicated matter, and there is no language today for which we can claim a full linguistic description.”

“SMT came with an extraordinary ability to memorize from its exposure to existing translations – and with this ability, it brought a partial solution to the first challenge mentioned above, and a very good solution to the third one – the handling of terminology – however, it mostly ignores the modeling of the grammar. Technically, I think SMT is the most difficult of the 3 approaches, it combines many algorithms to optimize the MT process, and it is hard work to deal with the huge database of [relevant training ] corpus.”

“NMT has access to the meaning and is dealing well with modeling of human language grammar. In terms of difficulty, NMT engines are probably the simplest to implement, a full training of an NMT engine involve only several thousands of lines of code. The simplicity of implementation is, however, hiding the fact that we/nobody knows why it is so effective.”

“I would use an analogy of a human learning to drive a car to explain this more fully:

- The Rule-based approach will attempt to provide a full modeling of the car dynamic, on how the engine is connected to the wheel, on the effect of acceleration in the trajectory, etc. This is very complicated. (And possibly impossible to model in totality.)

- The Statistical approach, will use data from past experience and will try to compare a new situation with a past situation and will decide on the action based on this large database [of known experience]. This is a huge task and very difficult to implement. (And can only be as good as the database it learns from.)

- The Neural approach, with a limited access to the phenomenon involved, or with limited ability to remember, will build its own “thinking” system to optimize the driving experience, it will actually learn to drive the car, build reflexes – but will not be able to explain why and how such decisions are being made, and will not be able to leverage local knowledge – for instance that at a specific bend on the road in very specific weather condition, it had to anticipate [braking] because it is particularly dangerous, etc... This approach is surprisingly very simple and thanks to computation power evolution have become more accessible.”

“Today, this last approach (NMT) is clearly the most promising but will need to integrate the second (SMT) to be more robust, and eventually to deal with the first one (RBMT) to be able to not only make choices but also explain them.”

 

Comparing NMT to Adaptive MT

When probed about this, JAS said: “Adaptive MT is an innovative concept based on the current SMT paradigm – it is, however, a concept that is quite naturally embedded in the NMT paradigm; of course, there will be work needed to be done to make it work as nicely as the Lilt product does it. But, my point is that NMT (and not just from Systran) will bring a far more intuitive solution to this issue of continuous adaptive learning, because it is built for that: on a trained model, we can tune the model without any tricks with feedback of one single sentence – and produce a translation which immediately adapts to user input.”

The latest generation MT technology, especially NMT and Adaptive MT look like a major step forward to enabling the expanding use of MT in professional translation settings. With continuing exploration and discovery in the fields of NLP, artificial intelligence and machine intelligence, I think we may be in for some exciting times ahead as these discoveries benefit MT research. Hopefully, the focus will shift to making new content multilingual and solve new kinds of translation challenges, especially in speech and video. I believe that we will see more of the kinds of linguistic steering activities we are seeing in motion at eBay and that there will always be a role for competent linguists and translators.

Jean Senellart, CEO, SYSTRAN SA

The first part of the SYSTRAN interview can be found at A Deep Dive into SYSTRAN’s Neural Machine Translation (NMT) Technology 

Post Script: There is a detailed article that describes the differences between these approaches on the SYSTRAN website  released a few weeks after this post was published.

A Deep Dive into SYSTRAN’s Neural Machine Translation (NMT) Technology

One of the wonderful things about my current independent status is the ability to engage deeply with other MT experts who were previously off limits because competing MT vendors don't usually chat with open hearts and open cloaks. MT is tough to do well and I think the heavy lifting should be left to people who are committed for the long run, and who are willing to play, invest and experiment in spite of regular failure. This is how humans who endure and persist, learn and solve complex problems.

This is Part 1 of a two-part post on the SYSTRAN NMT product announcement. The second part will focus on comparing NMT with RBMT and SMT and also with the latest Adaptive MT initiatives. It can be found here: Comparing Neural MT, SMT and RBMT – The SYSTRAN Perspective

Press releases are so filled with marketing-speak as to be completely useless to most of us. They have a lot of words but after you read them you realize you really don't know much more than you got from the headline. So, I recently had a conversation with Jean Senellart , Global CTO and SYSTRAN SAS Director General, to find out more about their new NMT technology. He was very forthcoming and responded to all my questions with useful details, anecdotes, and enthusiasm. The conversation only reinforced in my mind that "real MT system development" is something best left to experts, and not something that even large LSPs should dabble with. The reality and complexity of NMT development push the limits of MT even further away from the DIY mirage.

In the text below, I have put quotes around everything that I have gotten directly from SYSTRAN material or from Jean Senellart (JAS) to make it clear that I am not interpreting. I have done some minor editing to facilitate readability and "English flow" and added comments in italics within his quotes where this is done.

 

The New Product Line

JAS clarified several points about the overall evolution of the SYSTRAN product line.
SYSTRAN intends to keep all the existing MT system configurations they have in addition to the new NMT options. So they will have all of the following options:

• RBMT :- the rule-based legacy technology
• SMT :- Moses-based generation of engines that they have released for some language pairs over the last few years
• SPE :- Statistical Post-Editing translation engines - that were introduced in 2007 as the first implementation combining Rule-Based plus Phrase-Based Statistical systems.
• NMT:- is the purely neural machine translation engines that they just announced.
• NPE :- stands for « Neural Post-Editing » and it is the replication of what they did in SPE using Phrase-Based machine translation, but now using Neural Machine Translation instead of SMT for the second step in the process. They are now using a neural network to correct and improve the output of a rule-based engine.

They will preserve exactly the same set of APIs and features (like the support of a user dictionary) around these new NMT modules so that these historical linguistic investments are fully interchangeable across the product line.

JAS said: "From my intuition, there will still be situations where we will prefer to continue to offer the older solutions: for instance, when we will need high-throughput on a standard CPU server, or for low-resource languages for which we already have some RBMT solution, or for customers currently using heavily customized engines." However, they expect that NMT will proliferate even in the small memory footprint environment, and even though they expect that NMT will eventually prevail, they will keep the other options available for their existing customer base.

The NMT initiative focused on languages that were most important to their customers, or was known to be difficult historically, or was currently presenting special challenges not easily solved with the legacy solutions. So as expected the initial focus was on EN<>FR, EN<>AR, EN<>ZH, EN<>KO, FR<>KO. All of these already show promise, especially the KO <> EN, FR combinations which showed the most dramatic improvements and can be expected to improve further as the technology matures.

However, DE<>EN is one of the most challenging language pairs, as Jean said: "we have found the way to deal with the morphology, but the compounding is still problematic. Results are not bad, though, but we don't have the same jump in quality yet for this language pair."

 

The Best Results

So where have they seen the most promising results? As Jean said: "The most impressive results I have seen are in complicated language pairs like English-Korean, however, even for Arabic-English, or French-English the difference of quality between our legacy engines, online engines, and this new generation is impressive.

What I found the most spectacular is that the translation is naturally fluent at the full sentence level - while we have been (historically used) to some feeling of local fluency but not sounding fully right at the sentence level. Also, there are some cases, where the translation is going quite away from the source structure - and we can see some real "rewriting" going on."

Here are some examples comparing KO>EN sentences with NMT, SYSTRAN V8 (the current generation) and Google:

 

And here are some examples of where the NMT seems to make linguistically informed decisions and changes the sentence structure away from the source to produce a better translation.

 

The Initial Release

When the NMT technology is released in October, SYSTRAN expects to release about 40 language pairs (mostly European and Major Asian languages related to English and French) with an additional 10 still in development to be released shortly after.

As JAS stated: "We will be delivering high-quality generic NMT engines that will be instantly ready for "specialization" (I am making a difference with customization (which implies training) because the nature of the adaptation to the customer domain is very different with NMT)."

Also very important for the existing customer base is that all the old dictionaries developed over many years for RBMT / SMT systems will be useful for NMT systems. As Jean confirmed: "Yes - all of our existing resources are being used in the training of the NMT engines. It is worth noting that, dictionaries are not the only components from our the legacy modules we are re-using, the morphological analysis or named entity recognition are also key parts of our models."

With regard to the User Interface for the new NMT products, JAS confirmed: "the first generation will fully integrate into the current translation infrastructure we have - we had to replace of course the back-end engines, but also some intermediate middle components. However, the GUI is preserved. We have started thinking about the next generation of UI which will fully leverage the new features of this technology, and will be targeting a release next year."

The official SYSTRAN marketing blurb states the following:

"SYSTRAN exploits the capacity NMT engines have to learn from qualitative data by allowing translation models to be enriched each time the user submits a correction. SYSTRAN has always sought to provide solutions adjusted to the terminology and business of its customers by training its engines on customer data. Today SYSTRAN offers a self-specialized engine, which is continuously learning on the data provided."

 

Driving MT Engine Improvements

Jean also informed me that NMT has a simple architecture but the number of options available to tune the engines are huge and he has not found one single approach that is suitable for all languages. Options that can make a significant difference include, "type of tokenization, the introduction of additional features for instance for guiding the alignment, etc...

So far we have not found one single paradigm that works for all languages, and each language pair seems to have its own preference. What we can observe is that unlike SMT where the nature of the parameters was numerical and not really intuitive, here it seems that we can get major improvements by really considering the nature of the language pair we are dealing with."

So do these corrective changes require re-training or is there an instant dictionary-like capability that works right away? "Yes - this is a cool new feature.We can introduce feedback to the engine, sentence by sentence. It does not need retraining, we are just feeding the extra sentence and the model instantly adapts. Of course, the user dictionary is also a quick and easy option. The ability of an NMT engine to "specialize" very easily and even to adapt from one single example is very impressive."

 

Detailed MT Quality Metrics

"What is interesting is that we get major score improvement for systems that have not been tuned for the metrics they are evaluated against - for instance, here are some results on English-Korean using the RIBES metric."

 

"In general, we have results in the BLEU range of generally above 5 points improvement over current baselines."

"The most satisfying result, however, is that the human evaluation is always confirming the results - for instance for the same language pair shown below - when doing pair-wise human ranking we obtained the following results. (RE is human reference translation, NM is NMT, BI is Bing, GO is Google, NA is Naver, and V8 our current generation). It reads "when a system A was in a ranking comparison with a system B - or reference), how many times was it preferred by the human?"
 

"What is interesting in the cross comparison is that when we rank engines by the pair - When we blindly show a Google and V8 translation we see which one the user prefers. The most interesting row, however, is the second one:

         RE             BI       GO       NA      V8

NM  46.4           74.5    73.9    72     63.1

When comparing NMT output with the human reference translation, 46% of the time NMT is preferred (which is not bad, that means about one sentence out of two, the human does not prefer the Reference HT over NMT!), when comparing NMT and Google - 74% of the time, the preference goes to NMT, etc..."

 

The Challenges

The computing requirements have been described by many as a particular challenge. Even with GPUs, training an NMT engine is a long task. As Jean says: "and when we have to wait 3 weeks for a full training, we do need to be careful with the training workflow and explore as many options as possible in parallel."

"Artificial neural networks have a terrific potential but they also have limitations, particularly to understand rare words. SYSTRAN mitigates this weakness by combining artificial neural network and its current terminology technology that will feed the machine and improve its ability to translate."

"It is important to point out that graphic processing units (GPUs) are required to operate the new engine. Also, to quickly make this technology available, SYSTRAN will provide the market with a ready-to-use solution using an appliance (that is to say hardware and software integrated into a single offering). In addition, the overall trend is that desktops will integrate GPUs in the near future as some smartphones already do (the latest iPhone can manage neural models). As [server] size is becoming less and less of an issue, NMT engines will easily be able to run locally on an enterprise server."

As mentioned earlier there are still some languages where the optimal NMT formula is still being unraveled e.g. DE <> EN but these are still early days and I think we can expect that the research community will zero in on these tough problems, and at some point at least small solutions will be available even if complete solutions are not.

 

Production User Case Studies

When asked about real life production use of any of the NMT systems Jean provided two key examples.

"We have several beta-users - but two of them are most significant. For the first one, our goal is to translate a huge tourism related database from French to English, Chinese, Korean, and Spanish. We intend to use and publish the translation without post-editing. The challenge was to introduce support for named entity recognition in the model - since geographical entities were quite frequent [in the content] and a bit challenging for NMT. The best model was a generic model, meaning that we did not even have to adapt to a tourism model - and this seems to be a general rule, while in previous generation MT, the customization was doing 80% of the job, for NMT, the customization is only interesting and useful for slight final adaptation.

The second [use case]- is about technical documentation in English>Korean for an LSP. The challenge was that the available "in-domain" data was only 170K segments, which is not enough to train a full engine, but seems to be good enough to specialize a generic engine."

From everything I understand from my conversations, SYSTRAN is far along the NMT path, and miles ahead in terms of actually having something to show and sell, relative to any other MT vendor . They are not just writing puff pieces about how cool NMT is, to suggest some awareness of the technology. They have tested scores of systems and have identified many things that work and many that don't. Like many innovative things in MT, it takes at least a thousand or more attempts before you start developing real competence.They have been carefully measuring the relative quality improvements with competitive alternatives, which is always a sign that things are getting real. The product is not out yet, but based on my discussions so far, I can tell they have been playing for awhile. They have reason to be excited, but all of us in MT have been down this path before and as many of us know, the history of MT is filled with empty promises. As the Wolf character warns us (NSFW link, do NOT click on it if you are easily offended) in the movie Pulp Fiction after fixing a somewhat impossible problem, let's not get carried away just yet. Let's wait to hear from actual users and let us wait to see how it works in more production use scenarios before we celebrate.

The goal of the MT developer community has always been to get a really useful automated translation, in a professional setting, since perfection it seems is a myth. SYSTRAN has seriously upped their ability in being able to do this. They are getting continuously better translation output from the machine.  If I were working with an enterprise with a significant interest in CJK <> E content, I would definitely take a closer look, as I have also gotten validation from Chris Wendt at Microsoft on their own success with NMT on J <>E content. I look forward to hearing more feedback about the NMT initiative at SYSTRAN, and if they keep me in the loop I will share it on this blog in future. I encourage you to come forward with your questions as it is a great way to learn and get to the truth, and Jean Senellart seems willing and able to share his valuable insights and experience.

 

Jean Senellart, CEO, SYSTRAN SA

Comparing MT Based Translation Errors with Human Translation Errors

This is an important subject and one that needs ongoing examination and continuing study to get to real insight and find greater process efficiency. I hope that this post by Silvio Picinini will trigger further discussion and I also invite others who have information, and opinions to share on this subject, to step forward. In my observation of MT output over time, I have seen that MT produces a greater number of actual errors, but the types of errors most often generated by MT are easy to spot and clean up. Unlike the incorrect or inconsistent terminology and sometimes misunderstood source errors, that may often be hidden in the clear grammar and flow of a human translation.  These human errors are much harder to find, and not as easy to correct without multiple stages of review by independent reviewers. 

-----

It is common to see a focus on the errors that machine translation makes. But maybe there are also strengths in MT that can benefit translators and customers. So I wrote an article in Multilingual magazine comparing errors made by MT and human translators. The article is below, reproduced here with permission. 

Please find the original article published on Multilingual magazine, July-August 2016, “Errors in MT and human translation”, pg. 46-50.

Post-editing could be defined as the correction of a translation suggested by a machine. When a translator receives a suggested translation from Google, Bing or another machine translation (MT) engine, the translator is working as a post-editor of that suggestion. 

If instead the translator receives a suggestion from a translation memory (TM), the suggestion for that segment was created by a human translator in the past. If there is no suggestion from a TM, human translation is the creation of a translation by a human. If we now assume that a human translator and an MT engine “think” differently about the translation of a sentence, we can explore how a human translator makes different errors compared to a statistical machine translation process. If we make translators and post-editors more aware of the types of errors that they will find, they will be able to improve their translation quality. 

Inconsistency 

A human translator will likely go through a text and consistently translate the word “employee” with the same words in the target language, keeping his or her favorite word in mind and using it all the time in the translation. The MT engine, on the other hand, has no commitment to consistency. Statistical machine translation (SMT) is a popularity contest. So, in one sentence, the translation for “employee” may be the usual one. But the corpus used to train the engine may have come from governments, or from the European Parliament, for example. And governments may like to call employees “public servants.” If that is popular enough, the MT may choose this translation for some sentences. The translation will be inconsistent and could look something like “Company employees may only park on the east parking lot. If that is full, the public servant may park on the west parking lot.” 

Thus, MT is more likely than humans to translate inconsistently. 

However, here are two more thoughts on this. First, humans create inconsistencies also. If the TM contains translations made by several translators with different preferences for words, or contains translations created over a long period of time, chances are that there are inconsistencies in the TMs. Those inconsistencies will be accepted by the human translator. 

Second, we are not weighing glossaries to one side or the other. The human translators could follow the glossary, or run glossary consistency checks and their translations would be consistent. But the same applies to post-editing. 

Mistranslations 

Many words have more than one meaning, and are defined as polysemous. The word “holder,” for example, may mean a person who owns a credit card (as in credit card holder), but it may also be a stand designed to hold an object, such as a plate holder. If you are thinking of items on eBay, you are most likely expecting the translation that means plate holder. A human translator will easily know the correct translation. The machine, however, may have a lot of training data related to finance or laws, and the most popular translation there could be the person. The MT could choose this most popular meaning and the result could be “fashion sponge holder with suction cup” translated as “fashion sponge card holder with suction cup.” 

Thus, MT is more likely than humans to make mistranslation substitutions. 

Words that are not to be translated 

For a human translator, it is easy to know that Guess, Coach and Old Navy are brands, and therefore should not be translated into most target languages. As you know, SMT is a popularity contest, and the very common word “guess” likely appears quite frequently in the corpus that trained the SMT engine. The consequence — you guessed it — is that the MT is likely to translate the word instead of leaving it untouched because it is a brand.
This may happen with product names as well. Air Jordan sneakers could have the word Air translated. It could happen with the brand Apple versus the fruit apple, but since it is a popularity contest, the fruit may now be left untranslated instead of the iPhone having a fruit to go with it. 

Thus, MT is more likely than humans to translate words that are not supposed to be translated.

Untranslated Words 

The MT leaves out of vocabulary (oov) words untranslated, and humans will translate them. This will favor humans depending on how many oov words are present in the content to be translated. If the content is specific and different from the corpus used to train the engine, it is more likely that some words will not be known by the MT engine. But if the MT engine is well trained with the same kind of subject that is being translated, then the MT engine will minimize the number of untranslated words. On the other hand, MT takes the collective opinion into account. It may not translate words that are now commonly used untranslated, while a translator could be more traditional or old-fashioned and would translate the word. Would you translate “player” in “CD player” in your language? The word “player” used to be translated a few decades ago, but the usage changed and the English “CD player” is common now in many languages. The MT will learn from the corpus the most current and frequent usage, and may do better than a human translator. Overall, this issue still slightly favors the human side.
Thus, MT will leave more wrongly untranslated words than humans. 

Gender and Number Agreement 

The MT engine may learn from the corpus the correct translation for “beautiful skirt,” and skirt is a feminine word in many languages. However, the first time the source contains the combination “beautiful barometer,” it will pick from what it knows and it may translate beautiful as a feminine word. If barometer is masculine in the target language, this creates an error of gender agreement. The MT is more likely to make this error than a human translator, who intuitively knows the gender of objects. The same applies to singular and plural. English uses invariant adjectives for both, as in “beautiful skirt” and “beautiful skirts.” Thus, the MT engine may pick the singular translation for the adjective next to a plural noun. The MT is more likely to make a number agreement error than a human translator, who knows when singular or plural is needed. 

Thus, MT will make more grammar errors than humans. 

So far we have seen several examples of situations where humans translate better than MT engines. Now we will look at how a “self-correcting” property of MT, created from the popularity of a translation, can often do a better job than humans. A statistical MT engine can be seen as a “popularity contest” where the translation that is suggested is the most popular translation for a word or group of words present in the “knowledge” (corpus) that trained the MT engine.

Spelling

There are two types of spelling errors: the ones that create words that don’t exist (and can be caught by a spellchecker) and the errors that turn a word into another existing word. You may have turned “from” into “form” and “quite” into “quiet.” The first type, a pure spelling error made by MT would require that you have in the corpus more instances of the error than of the correction. Can you imagine a corpus that contains “porduct” 33 times and “product” only 32 times? So MT almost never makes a spelling error of this kind.

For the second type, humans turn words into other words, and the spellchecker will miss it. The MT engine will not make this error because it is not likely that the corpus will contain the misspelled word more frequently than the correct word for that context. This would require having “I am traveling form San Francisco to Los Angeles” more frequently in your corpus than you would have “I am traveling from San Francisco to Los Angeles” and which one is more likely to be popular in a corpus? The correct one. This is why MT will almost never make this kind of spelling error, while it is easy for a human translator to do so.

Thus, humans are more likely than MT to make spelling errors of any kind. 

False Friends 

False friends are words that look similar to a word in a different language, but mean something different. One example is the word “actual,” which means “real” in English. In Portuguese, the word atual means current or as of this moment. So a presentation mentioning “actual numbers” could be translated as “current numbers,” seriously changing the meaning and causing confusion. A human translator may make this error, but the MT would require the wrong translation for “actual” to be more popular in the corpus than the correct translation. You would need “actual numbers” to be translated more frequently as “current numbers” than as “real numbers.” Do you think this would happen? No, and that is why MT almost never falls for a false friend, while a human translator falls for it occasionally.

Thus, humans are more likely to make false friend errors than MT. 

Fuzzy Match

There are several errors that result from the use of TM. These memories offer the human translator suggestions of translation that are similar to the segment they are translating. Similar does not mean equal, so if the suggested translation is a fuzzy match, the human translator must make changes. If they don’t make any change and accept the fuzzy match as it is, they risk making errors. There are three sub-types of errors to mention here:

Different terms. Think of a medical procedure where the next step is “Administer the saline solution to the patient.” If a fuzzy match shows “Administer the plasma to the patient,” this might risk a person’s life.

Opposite meaning. Think of “The temperature of the solution administered to the patient must stay below XX degrees.” If a fuzzy match shows “must stay above XX degrees,” this might risk a person’s life. For an eCommerce environment, this type of error could be a major issue: “This item will not be shipped to Brazil” versus “This item will be shipped to Brazil.” 

Numbers that don’t match. Fuzzy matches from a year before may offer the translator a suggested translation of “iPhone 5” because that was the model from a year ago. The new model is the iPhone 6. If a fuzzy match is accepted with the wrong number, the translator is introducing an old model.

Thus, humans are much more likely to make errors for accepting fuzzy matches than MT.

Acronyms 

MT may leave acronyms as they are, because they may not be present in the corpus. The MT engine has the advantage of having the corpus to clarify if an acronym should be translated as the same acronym as in the original, if it should be translated as a translated acronym or if it should be translated using the expanded words from the meaning of the acronym. Human translators may make errors here. If they do research, and the research does not clarify the meaning, the original acronym may be left in the translation. So this is an issue that favors the MT over humans, although not heavily.
The best solution for both humans and MT is to try to find the expanded form of the acronyms. This will help MT and humans produce a great and clear translation. 

Thus, humans are slightly more likely to make errors translating acronyms than MT. 

Terminology 

MT may handle terminology remarkably better than a human translator. If an engine is trained with content that is specific to the subject being translated, and that has been validated by subject matter experts and by feedback from the target audience that reads that content, the specific terminology for that subject will be very accurate and in line with the usage. Add to this the fact that multiple translators may have created those translations that are in the corpus, and it becomes easy to see how an MT engine can do a better job with terminology than a single human translator, who often translates different subjects all the time and cannot be a subject matter expert on every subject. 

Consider the following example: 

English: “In photography and cinematography, a wide-angle lens refers to a lens whose focal length is substantially smaller than the focal length of a normal lens for a given film plane. This type of lens allows more of the scene to be included in the photograph.”

Portuguese machine translation: “Em fotografia e cinematografia, uma lente grande angular refere-se a uma lente cuja distância focal é substancialmente menor do que a distância focal de uma lente normal para um determinado plano do filme. Este tipo de lente permite que mais da cena a ser incluída na fotografia.” 

In this example about photography, the MT already proposed the translation of wide-angle as grande angular, which is the term commonly used to refer to this type of lens. This translation means approximately “large angular.” A human translator knows the translations for the words wide and angle. The translator could then be tempted to translate the expression wide-angle literally as “wide angle” lenses (lente de ângulo amplo), missing the specific terminology used for the term. The same could happen for focal length. Portuguese usually uses distância focal, which means “focal distance." A human translator, knowing the translation for focal and length, would be tempted to translate this as comprimento focal and would potentially miss the specific terminology distância focal. 

The quality of the terminology is, of course, based on the breadth and depth of the corpus for the specific subject. A generic engine such as Google or Bing may not do as well as an MT engine custom-trained for a subject. But overall, this is an issue that could favor the MT over humans. 

Thus, humans are more likely to make errors for inappropriate terminology than MT.

 Figure 1: MT and human translation errors

Emerging technologies for post-editing 

Now that we are aware of the issues, which are summarized in Figure 1, we are in a better position to look at emerging technologies related to post-editing. Post-editing work has one basic requirement: that the translator is able to receive MT suggestions to correct, if need be. This technology is now available integrated on several computer-aided translation (CAT) tools. 

Considering the above, the next application of technology is the use of quality assurance (QA) tools to find MT errors. The technology itself is not new and has been available in CAT tools and in QA tools such as Xbench or Okapi CheckMate. What is new is the nature of checks that must be done with these tools. One example: in human translation you use a glossary to ensure the consistent translation of a term. In MT, you could create a check to find a certain polysemous word and the most likely wrong translation for it. Case frequently has the meaning of an iPhone case, but it is often wrongly translated with the meaning of a legal case. Your glossary entry for MT may say something like “find case in the source and legal case in the target.” This check is very different from a traditional check for human translation that looks for the use of the correct translation instead of the use of the “probably wrong” one.

After doing post-editing and finding errors, the last area of application of this technology is in the measurement of the post-editing, since what makes post-editing most attractive is the promise of increasing the efficiency of the translation process. We will briefly mention some of the main technologies being used or researched: 

Post-editing speed tracking. The time spent post-editing can be tracked at a segment level. These numbers can then be compiled for a file. Some examples of use of this technology include the MateCat tool, the iOmegaT tool and the TAUS DQF platform.

MT confidence. Another technology worth mentioning is MT confidence scores. Based on several factors, an MT engine can express how confident it is on the translation of a certain word. If this confidence can be expressed in terms of coloring the words in a segment, this feature will help the post-editor focus on words with less confidence that are therefore more likely to require a change. This feature appeared in the CASMACAT project, as illustrated in Figure 2. 

 Figure 2: The CASMACAT project shows MT color-coded according to what is most likely to need post-editing

Edit distance. A concept that is not new but could be more used more often is the concept of edit distance. It is defined as the number of changes — additions, deletions or substitutions — made to a segment of text to create its final translated form. Comparing the final form of a post-edited segment to the original segment that came out of the MT engine provides a significant indication of the amount of effort that went into the post-editing task. The TAUS DQF platform uses edit distance scores.

We use the concept of edit distance in a broader sense here, indicating the amount of changes. This includes the “raw” number of changes made, but also includes normalized scores that divide the number of changes by the length of the text being changed, either in words or characters. The TER (Translation Edit Rate) score is used to measure the quality of MT output, and is an example of a normalized score. 

The final quality that needs to be achieved through post-editing defines levels of “light post-editing” and “full post-editing.” There are discussions to define and measure these levels. The scores based on edit distance may provide a metric that helps in this definition. It is expected that the light post-editing should require fewer changes than a full post-editing, therefore the scores based on edit distance for a light post-editing should always be lower than the score for a full post-editing. Figure 3 below shows a hypothetical example with numbers.

Figure 3: Edit distance shows how much editing is required for a given machine-translated file.

Scores based on edit distance can be an important number in the overall scenario of measuring post-editing, combined with the measurements of speed. The TAUS DQF efficiency score proposed a combination of these measurements.

Silvio Picinini is a Machine Translation Language Specialist at eBay since 2013. With over 20 years in Localization, he worked on quality-focused positions for an LSP, and as an in-house English into Brazilian Portuguese translator for Oracle. A former quality engineer, Silvio holds degrees in electrical and electronic/software engineering.

LinkedIn: https://www.linkedin.com/in/silviopicinini

Overview of Expert MT Systems - Iconic Translation Machines

This is a third post in a series on expert MT systems developers. Iconic fits my definition of expert much more closely (as does tayou) as they do-it-for-you rather than give you a technology platform to do-it-yourself. (DIY does not work well for those who do not really know what they are doing.) They bring deep expertise to bear on the very specific translation problem that you bring to them, and may create a unique MT solution for each customer. In the case of Iconic, the components used to build an MT solution, may also vary from customer to customer, as they assemble sub-components to address unique customer problems, which can vary by content, language and quality expectation. In general, the interaction with an expert is going to much more consultative and customer specific and less off-the-shelf. As MT is an evolving technology, it is especially useful for an expert developer to have close ties with an academic institution that is engaged in ongoing research in MT, and related technology like machine learning and artificial intelligence. This allows them to continue to enhance the technology and most expert developers will have this kind of a relationship in place. This is especially critical at this juncture, as there is a lot of leverage able work being done in the machine intelligence and neural network based learning field. This post developed from a conversation with John Tinsley. Most of the emphasis in the post below is mine.

-------------------------------------

 

From the lab to the market

The concept and technology behind Iconic originated from a large scale research and development project at Dublin City University (DCU). The state-of-the-art MT technology developed by a number of MT PhDs, who are still with the company today, was adapted to meet the translation requirements at the European Patent Office. On the back of this, the company spun-out of the university and our first commercial MT engine, for English to Portuguese patent translation, went into production in July 2010. We haven’t looked back since. 

Our original focus was exclusively on patent translation and we worked with a number of LSPs and information providers who specialised in this field. We’ve since evolved into a turnkey machine translation software and solutions provider that specializes in custom solutions tailored with subject matter expertise for specific industry sectors and content types. These are areas where our sophisticated MT technology has significant value to add over off-the-shelf solutions.

MT: a complex technology

There are two established paradigms of MT: rule based, and statistical (though Neural MT is clearly on the horizon, more on that later). Statistical MT is by far the predominant approach, and extensions to this include the incorporation of rules or linguistic information which are often referred to as “hybrid” MT. Despite this, there is no single approach or configuration that works best across all language pairs, content types, and writing styles.

Our approach, from a technological perspective, tries to address the fact that there’s no “one-size-fits-all” machine translation solution. The best approach completely depends on the job at hand. We have developed an Ensemble Architecture^TM which combines 100’s of processes across different paradigms that can be combined to produce the best translation for a given task. 

Some of these processes are specific to a particular language, to a certain content type (e.g. legal, financial), or to a particular writing style (e.g. patents, contracts, annual reports). Depending on a given input document or client project, our platform takes an on-the-fly decision as to the most appropriate combination of processes to use. In fact, in many cases, different sections of a single document can be translated using completely different processes. The goal is to use the most effective set of tools at our disposal for any given task, so that our end users get the best possible translation.

The Ensemble Architecture: a sample of just a few of the processes available to enhance translation quality

From a client customisation perspective, this model is super extensible. It allows us to develop and add new processes to the ensemble architecture as needed on a case by case basis, making it a continuously evolving architecture.

One of the biggest challenges in maintaining such a complex technology is the need for deep expertise in the underlying MT methodologies. This places a lot of importance on the team, which is something we’ve put a lot of resources into building at Iconic. Having a mix of experts in the science underpinning MT technology, and language experts, combined with software development skill in the team allows us to transfer the subject matter expertise in our team, directly into our clients’ applications.

This is crucial when it comes to ongoing engine development and customisation. When adding data to an engine, quality can hit a ceiling quite quickly. End users will highlight critical issues in the output that cannot be resolved by simply “retraining” and adding more data. Even if we have valuable post-edited data from translators, it is still often not enough. Having the expertise in the team, in a) knowing where to look “under the hood” to identify the cause of an issue, and b) having the skill to implement a fix, is crucial to success. In fact, we recently ran a video series introducing some of the members of our team which you can watch here. 

Applying MT successfully across industry sectors

Iconic’s first product was IPTranslator, a suite of MT engines adapted for patent and intellectual property related content. Patents are challenging enough to read in your native language, never mind trying to develop MT software to handle them. The complexity of the documents was one of the key motivators in developing our Ensemble Architecture - we needed the capability for the MT engine to be able to dynamically adapt to the content domain, be it a chemical, engineering, or electronic patents. Similarly, it needs to be able to treat the different patent sections in different ways, e.g. the stunted telegraphic style of the title vs. the long-winded “legalese” of the claims. 

Since then, this architecture has grown to inherently cover other areas, and verticals that pose similar technological challenges to MT, such aseDiscovery, the financial and life sciences industries, and, in particular, user-generated content in the hospitality and e-commerce industries. 

From the language perspective, like most approaches to MT, our technology is language independent and can be applied to any combination of languages. However, given the extensibility and the ability to combine domain-adapted approaches with language-specific processes, we’ve been able to tackle harder projects had have great success with traditionally more challenging languages such as Chinese and German, to name just two. 

These are areas we’ve chosen to focus on because we have significant value to add with our technology and expertise. MT has been rolled out generally with relative success in industries such as IT and for languages like French, which tend to lend themselves better to language technology in general. But when it comes to more difficult languages and content types, a smarter approach is needed and that’s often where we come into play. 

MT for multiple buyers

We are the MT partner of choice for some of the world’s largest translation companies, information providers, and government and enterprise organisations, each of whom can have very different requirements from an MT technology solution. We have worked very closely with Welocalize over a number of years, in particular Park IP, to bring post-edited MT into their workflow for patent translation across a number of languages. Similarly, we have worked extensively with RWS in the same field. 

Over the past 18 months, in collaboration with the ADAPT research centre, we have been providing English to Irish MT to the Irish government, also for post editing. To the best of our knowledge, is the first successful commercial deployment of MT for this language combination.

Outside of traditional language services use cases, Iconic’s MT is used in a number of areas where translation is just one part of a much bigger picture. For instance, over the course of the last year we have been working on large scale digitization projects, taking millions of documents of archive content stretching back over the past 150 years (in various challenging formats as you can imagine!), converting them, cleaning them, machine translating them, and then making the available in multilingual searchable databases. Billions of words of historical analog content have passed through our engines and are now accessible to a global audience. 

In an increasing number of these cases, MT is producing quality output that is fit for purpose as is without the need for post editing and this trend is only going to continue as the volume of content being produced grows further.

Business models for complex technology

In the same way that there’s no single “best” approach to MT, it is also not an “out of the box” type of software. Yes, there are use cases and instances where it can work in this way, but they are the arguably the exception rather than the rule. It can work very well for anecdotal or spot usage, but when it comes to complex real-world problems, a deeper level of expertise and experience is required.

This has probably been distorted by (fantastic, ground-breaking) tools like Moses which have done a great job of lowering the bar to entry to machine translation from a software development perspective. But doing things in this way will only get you so far and, ultimately, without understanding the underlying complexities of the software, of the languages being translated, and of how to compliment and extend upon Moses with additional natural language processing (NLP) techniques, performance will plateau. 

It’s possible to provide interfaces for users to upload their own training data, terms, post-editing rules, and so on, which has been done extensively and often to good effect. Ultimately, however, that’s really over-simplifying the technology and basically taking it out of expert hands. Doing that, as a software provider you have no recourse when someone misuses your technology and it doesn’t meet their needs.

5 steps to adopting machine translation

Software with a Service

To overcome this, we provide MT as a fully managed cloud-service with a different take on the standard business model. We call this model Software with a Service (as opposed to Software as a Service) and what it does is combine technology automation (the software) with specialized expert labor (the service) to deliver a complete solution to a business problem. It’s as much about expert people-powered customer service as it is about code-powered efficiency. The result is awesome customer experience that can only be delivered with a human touch.

MT engine development and maintenance is costed on a professional services basis as required, which serves the purpose of ruling out unnecessary “retraining” when it is not going to bring any value. Usage of the engines, cloud-based or otherwise, is costed as normal based on usage either on a subscription basis or pay-as-you-go. This can change to standard license model when the software needs to be installed on local servers, which is typically done for security reasons.

Buyers of MT are smart, and they are increasingly aware that assessing MT based on output samples or bake-offs (comparisons) between providers doesn’t make sense. MT is never going to be as good as it can be straight out-of-the-box. It takes time, and requires expert guidance to develop, incorporate, and maintain. This is why we’ve had a good response to this model - it just makes sense. 

When should MT be used in the first place?

As I mentioned, MT is not a one-size-fits-all solution and, in fact, in some cases it’s not suitable at all. That’s why the first step in all of our engagements is to assess whether it’s feasible to use MT in the first place. Never mind other business models; the worst business model is accepting projects that are destined to fail from the outset! The industry has been plagued by false dawns and lack of expectation management for a long time, so MT providers need to become more transparent about when their solutions should and should not be used. 

How do we assess feasibility? At Iconic, we look at 8 factors that can influence things in different ways when it comes to MT. In order for us to have a clear picture in our mind as to the best way to proceed in each case, we need to understand these factors, interpret them, and then apply our experiential-driven expertise in order to communicate to the customer how they impact things. We do this even if it means turning down a project because it wasn’t suitable. We discussed these factors in a webinar recently and we’re currently running a blog series on our website where we’re diving even deeper on each factor. 

The 8 factors that influence machine translation projects

The Next Frontier? The Neural Frontier?

So, where do we go from here? For all MT providers, there’s no hiding from the fact that neural MT is not just a fad and is most likely here to stay, and Iconic is no exception. That being said, there has been a lot of talk and hype about it, but relatively little action outside of academia (yet), in part due to the fact there are a still a few hurdles to overcome before it becomes a practicable commercial solution.

We pride ourselves on our deep MT expertise and our ability to develop complex language solutions, so we are very keen and excited to begin working with neural MT. To that end, we’ve been working on an exciting project with ADAPT research centre in Dublin, who have been pioneers in MT research over the last 15 years. We will be in a position to reveal more later this year, so stay tuned! 

To contact Iconic this link provides several alternatives. 

 

5 Tools to Build Your Basic Machine Translation Toolkit

This is a second post from the MT Language Specialist team at eBay, by Juan Martín Fernández Rowda . I have often been asked by translators about what kinds of tools are useful when working with MT. There is a lot of corpus level data analysis, preparation and editing going on around any competent MT project. While TM tools have some value, they tend to be segment focused and do not scale, there are much better tools out there to do the corpus pattern analysis, editing and the comparison work that is necessary to build the best systems. We are fortunate to have some high-value tools laid out very clearly for us here by Juan who has extensive direct experience working with large volumes of data and can provide experience-based recommendations.

----------------------------------------------------------------------------------

If you are a linguist working with Machine Translation (MT), your job will be a lot easier if you have the right tools at hand. Having a strong toolkit, and knowing how to use it, will save you loads of time and headaches. It will help you work in an efficient manner, as well. 

As a Machine Translation Language Specialist at eBay, I use these tools on a regular basis at work, and that is why I feel comfortable recommending them. At eBay, we use MT to translate search queries and listing titles and descriptions into several languages. If you want to learn more, I encourage you to read " The Next Big Thing You Missed: Why eBay, Not Google, Could Save Automated Translation ". 

1. Advanced Text Editors

Notepad won't cut it, trust me. You need an advanced text editor that can, at least:

• deal with different file encoding formats (UTF, ANSI, etc.)
• open big files and/or with unusual formats/extensions
• do global search and replace operations with regular expressions support
• highlight syntax (display different programming, scripting or markup languages -XML, HTML, etc.- with color codes)
• have multiple files open at the same time (tabs)

This is a list of my personal favorites, but there are a lot of good editors out there.
Notepad ++ : My editor of choice. You can open virtually any file with it, it's really fast, and it will keep your files in the editor even if you close it. You can easily search and replace in a file or in all open files, using regular expressions or just extended characters (control characters like \n or \t). It's really easy to convert from/to different file encodings and save all opened files at once. You can also download different plugins, like spellcheckers, comparators, etc. It's free and you can download it from here. 

Sublime : This is another amazing editor, and a developers' favorite. Personally, I find it great to write scripts. You can do many cool things with it, like using multiple selections to change several instances of a word at once, split a selection of words into different lines, etc. It supports regular expressions and tabs, as well. It has a distraction-free mode if you really need to focus. It's also free, and you can get it here. 

EmEditor : Syntax highlighting, document comparison, regular expressions, handles huge files, encoding conversion… Emeditor is extremely complete. My favorite feature, however, are the scriptable macros. This means, you can create, record, and run macros within EmEditor - you can use these macros to automate repetitive tasks, like making changes in several files and/or saving them with different extensions. You can download it from here. 

2. QA Tools

Quality Assurance Tools assist you in automatically finding different types of errors in translated content. They all basically work in a similar way: 1) you load files with your translated content (source + target); 2) you optionally load reference content, like glossaries, translation memories, previously translated files or blacklists; 3) the tool checks your content and provides a report listing potential errors. Some of the errors you can find using a QA Tool are:

• terminology: term A in the source is not translated as B in the target
• blacklisted terms: terms you don't want to see in the target
• inconsistencies: same source segment with different translations
• differences in numbers: source and target numbers should match
• capitalization
• punctuation: missing or extra periods, duplicate commas, etc.
• patterns: certain used defined patterns of words, numbers and signs, which may contain regular expressions to make them more flexible, expected to occur in a file.
• grammar and spelling errors
• duplicate words, tripled letters, and more.

Some QA Tools you should try are:
Xbench allows you to run the following QA Checks: find untranslated segments, segments with the same source text and different target text, and segments with the same target text and different source text, find segments whose target text matches the source text (potentially untranslated text), tag mismatches, number mismatches, double blanks, repeated words, terminology mismatches against a list of key terms, and spell-check translations. Some linguists like to add all their reference materials in Xbench, like translation memories, glossaries, termbases and other reference files, as the tool allows you to find a term while working on any other running application with just a shortcut.

Xbench also has an Internet Search tab to run searches on Google. The list is pretty limited but there are ways to expand it, maybe I'll share that in the future. You can get Xbench here. 

Checkmate is the QA Tool part of the Okapi Framework, which is an open source suit of applications to support the localization process. That means, the Framework includes some other tools, but Checkmate is the one you want to perform quality checks on your files. It supports many bilingual file formats, like XLIFF, TTX, and TMX. Some of the checks you can run are: repeated words, corrupted characters, patterns, inline codes differences, significant differences in length between source and target, missing translations, spaces, etc. The patterns section is especially interesting; I will come back to it in the future. Checkmate produces comprehensive error reports in different formats. It can also be integrated with LanguageTool,an open source spelling and grammar checker. You can get Checkmate here. 

3. Comparison Tools 

Why do you need a comparison tool? Comparing files is a very practical way to see in detail what changes were introduced, e.g. which words were replaced, which segments contain changes, or whether there is any content added or missing. Comparing different versions of a file (for example, before and after post-editing) is essential for processes that involve multiple people or steps. Beyond compare is, by far, the best and most complete comparison tool, in my opinion. 

You can also compare entire folders. If you work with many files, comparing two folders is an effective way to determine if you are missing any files or if a file does not belong in a folder. You can also see if the contents of the files are different or not. 

4. Corpus Analysis Tools

As defined by its website, AntConc is a freeware corpus analysis toolkit for concordancing and text analysis. This is, in my opinion, one of the most helpful tools you can find out there when you want to analyze your corpus or content, regardless of the language. AntConc will let you easily find n-grams and sort them by frequency of occurrence. It is a very practical way to identify the highest frequency n-grams in your corpus. Obviously, you want the most frequently used terms to be translated as accurately as possible. In most texts, words like prepositions or articles are the most common ones, so you can use a stop-word list to filter them out when they don't add any value to the task at hand.

AntConc is extremely helpful when it comes to find patterns in your content. Remember - with MT, you want to fix patterns, not specific occurrences of errors. It may sound obvious, but finding and fixing patterns is a more efficient way to get rid of an issue than trying to fix each particular instance of an error. With AntConc you can select the minimum and maximum sizes of the n-grams you want to see, as well as the frequency. 

AntConc can create a list of each word occurring in your content, preceded by the number of hits. This can help you get a deeper insight on your corpus for terminology work, like which terms you should include in your glossary. These words can also tell you what your content is about - just by looking at the most frequent words, you can tell if the content is technical or not, if it belongs to any specific domain, and even which MT system you can use to translate it, assuming you have more than one customized systems.
There are many things you can use this tool for and it deserves its own article.
Check AntConc out here. 

5. CAT Tools

CAT Tools make a great post-editing environment. Most modern tools can be connected to different machine translation systems, so you get suggestions both from a TM and from an MT system. And you can use the TM to save your post-edited segments and reuse them in the future. If you have to use glossaries or term bases, CAT tools are ideal, as they can also display terminology suggestions. 
 
When post-editing with a CAT tool, there are usually 2 approaches: you can get MT matches from a TM (of course, they need to be added to it previously) or a connected MT system, or you can work on bilingual, pre-translated files and store in your TM post-edited segments only. 
 
If you have never tried it, I totally recommend Matecat. It's a free, open source, web-based CAT tool, with a nice and simple editor that is easy to use. You don't have to install a single file. They claim you will always get up to 20% more matches than with any other CAT tool. Considering some tools out there cost around 800 dollars, what Matecat has to offer for free can't be ignored. It can process +50 file types; you can get statistics on your files (like word counts or even how much time you spent on each segment), split them, save them on the cloud, and download your work. Even if you never used a CAT tool before, you will feel comfortable post-editing in Matecat in just a few minutes. 
 

Another interesting free, open-source option is OmegaT. Not as user-friendly as Matecat, you will need some time to get used to it, even if you are an experienced TM user. It has pretty much all the same main features commercial CAT tools have, like fuzzy matching, propagation, it supports around 40 different file formats, and it boasts an interface to Google Translate. If you never used it, you should give it a try. 

If you are looking into investing some money and getting a commercial tool, my personal favorite is MemoQ. It has tons of cool features and, overall, is a solid translation environment. It probably deserves a more detailed review, but that is outside of the scope of this post. You can learn more about MemoQ here.

Juan Rowda

Staff MT Language Specialist, eBay

https://www.linkedin.com/in/juan-martín-fernández-rowda-b238915

Juan is a certified localization professional working in the localization industry since 2003. He joined eBay in 2014. Before that, he worked as translator/editor for several years, managed and trained a team of +10 translators specialized in IT, and also worked as a localization engineer for some time. He first started working with MT in 2006. Juan helped to localize quite a few major videogames, as well. 

He was also a professional CAT tool trainer and taught courses on localization.

Juan holds a BA in technical, scientific, legal, and literary translation.