Thursday, April 23, 2015

How Translators Can Assess Post-Editing MT Opportunities

With the continued growth in the use of MT, it has become increasingly important for translators to understand better when it is worth getting involved, and when it is wise to stay away from post-editing opportunities that come their way. 

This is still a very fuzzy issue for most translators and I think it might be useful to share some information with them to highlight some of the key variables they could use to determine the most rational action given the facts at hand. For some, post-editing will never be palatable work, but for those who look more closely and see that PEMT is now just another variant of professional translation work that is much like other translation work, which can be economically advantageous when one is working with the right partners and the right technology in this case.  

We have seen that in the early days of MT use that there has been much cause for dissatisfaction all around, especially for translators who have been asked to post-edit sub-standard MT output for very low rates. Translators do need to be wary since many LSPs deploy MT technology without really understanding it, with the sole purpose of reducing costs, and with no understanding on how to produce systems that actually enable this lower cost scenario or interest in engaging translators in the process. Thus it is worth translators learning some basic discrimination skills to determine and establish some general guidelines to understand the relative standing of any PEMT opportunity that they are presented with.

The following checklist is a useful start (IMO) that every translator should consider when deciding what kinds of PEMT opportunities are worth working on.
  • Understand the very specific MT output that you will be working with as every MT engine is unique and assessments need to be made in reference to the actual output you will be working with.
  • Determine if the LSP understands what they are doing with the MT technology and can respond to feedback on error patterns. There are many “upload and pray” efforts nowadays that create very low-quality systems that are very hard to control and challenging for translators to work with.
  • Understand the MT technology that is being used as not all MT is equal. There are many variants and you should know what the key differences are. Systems that allow feedback and have more controls to correct errors after the MT engine has been built and accept ongoing corrective feedback will generally be better to work with.
  • Have a basic understanding of the MT methodology which means at least an overview of the rules-based and statistical approaches. This can give you a sense of what kind of feedback you can provide and also help you understand error patterns.
  • Understand that MT engine development is an evolutionary process rather than an instant solution that Google has led some of us to believe. Professional MT deployment is a molding process that evolves in quality through expert iteration and is typically done to tune an engine for a specific business purpose to help an ongoing high-volume translation production need. MT makes much less sense for random one-time use.
  • Understand the basic quality assessment metrics used with MT. BLEU scores are often bandied about with MT systems and often interpreted incorrectly. If you understand them you will always have a better sense of the reality of a situation as incompetent practitioners use and interpret these scores incorrectly all the time. The BLEU scores are only as good as the Test Sets used and so try and understand what makes a good Test Set as described in the link.
It is wise to use technology when and only if there is a clear benefit, and this is especially true with MT. An LSP should have a clear sense that the productivity of the translation project will be improved by using the technology otherwise it is detrimental in many ways. This means that there needs to be a clear idea of what typical translation project throughput is before and after the use of MT. And a trusted way to measure how MT might impact this productivity. 

  • Thus MT only makes sense when it boosts productivity or when it makes it possible to provide some kind of translation for material that would just not get translated otherwise.
  • Translators should also understand that lower rates are not necessarily bad if their throughput is appropriately higher.
  • Finally, MT error patterns tend to be consistent so it makes sense to approach corrections at a chunk level rather than an individual segment level. 

Much of the dissatisfaction with PEMT work is related to compensation. My post on PEMT compensation remains the most read post on this blog even though it is now 3 years old. But I think if you understand the specific MT output you are dealing with and it’s impact on your throughput you can make an informed decision.

It is wise to remember that a lower rate does not necessarily mean less overall compensation as the following totally hypothetical chart explains. (The productivity benefits are more likely to be shared less generously). The best LSPs will have an open and transparent process in setting this rate and translators will be involved to ensure that the rate is fair and reasonable and based on actual MT output quality rather than some arbitrarily lower rate “since we are suing MT”. Also, expect Romance language rates to be lower than tough-for-MT languages like Japanese and Korean if editing effort is used as a criterion for setting the rate.

Much of what I have covered here was presented in a Proz presentation that is still available as video (slides with voice) for those who want to see and hear more details of the summary presented in this post.

As a complete aside this is for those who think that Genetically Modified foods are harmless. Here is a quote from a biotech company leader that you might want to consider the next time you eat corn from a US supermarket:
“We have a greenhouse full of corn plants that produce anti-sperm antibodies.” ~ Mitch Hein, president of Epicyte, a California-based biotechnology company.

And to end on a cheery note, I was very impressed by the musicality of this song and thought others might want to hear it too.


  1. Jeannette Stewart

    Thanks Kirti for sharing your valuable insights and knowledge! We all need to learn a lot more about MT and PE to be able to make informed decisions in accepting work and to also take advantage of available technology.


  2. As bright as usual.

  3. maxi Schwarz-BastamiMay 1, 2015 at 10:42 AM

    maxi Schwarz-Bastami

    Some thoughts after reading the article:
    * As a translator who might be asked to do this kind of work, I don't think it is necessary to understand how the system works, when it's suitable etc. That is only necessary for the people who run text through MT so that they don't make poor choices resulting in a poor "translation" that will require extensive PE. What WE need to understand is that many clients asking for this kind of work don't understand the technology they are using, and could be sending us junk and expect magic from the technology, and miracles from us.

    * As translators, what we need to do when asked to do PEMT is relatively simple. Look at the entire text and if it's a garbled mess, reject the work.

    * In regards to compensation, translators should charge a per-hour rate. Editing poorly translated text takes much longer than well translated text, so this policy is fair for all concerned. It also goes toward preventing the abuse where someone runs a text through MT for free, then gets it "edited" for .03/word and gets a translation for a very small price, while the translator works extremely hard for very long hours for just about nothing - and has helped reduce proper translation work in the market.

    - TRANSLATORS should charge a per hour rate. The customer has no role in setting these rates. It is not a question of the customer "giving a say" to the translator, or allowing the translator to negotiate. The professional being hired is the one who sets rates. I don't hire an accountant and tell her what I will pay her. She tells me her fee

    1. Thank you for your feedback and perhaps the hourly rate is the simplest way to handle the compensation issue. Though even with an hourly rate there would be throughput expectations, (if one translator does twice the volume of work that another does at the same hourly rate for example) so I think it is not so easy to walk away from this issue.

      LSPs are using a collaborative process in many cases to set a rate that involves at least a few translators if not all who will work on a project. This is when the rate is set based on the actual quality of the MT output. Because MT quality can vary so widely any scheme that doe snot somehow account for this is likely to create dis-satisfaction somewhere in the production chain.

    2. maxi Schwarz-BastamiMay 1, 2015 at 11:24 AM

      Kirti, do you yourself work as a translator, and is this coming from your experience?

      An LSP is a customer, and what the customer uses only goes as far as the vendor's policies. When a client comes to me, I state my rate. If the client wants to "work out" a rate with me based on his expectations, I restate my rate. In regards to how the fee is calculate, what kinds of estimates are made, that is MY job as the professional. I have been translating for about 25 years and that is my extensive experience.

      Since you are advising, it is important to know where you are coming from, which is why I asked whether you do this work yourself as a translator, or from which perspective.

    3. maxi, I am not a translator, so you are correct I have no real basis to advise translators what to do in this case. I am saying what I say as an MT system developer, based on what I have seen my LSP clients do. That is, they explain the MT output assessment process to translators so everybody can see that the rate is fair even though in absolute terms the rate may be lower.

      You policy makes a great deal of sense for a traditional translation job, but what would you recommend to the LSP or client who is faced with paying two translators the same hourly rate but sees that there is a significant discrepancy in work output?

      Business translation is changing in many ways and some in the industry are trying to determine the best way to address compensation equitably in view of the changing production models. . It is my opinion, that translators can greatly enhance the quality of the MT systems output once they understand why certain things happen with MT, also that translators should be compensated more for correcting bad translations and thus understanding the kind of MT output one is dealing with matters. I understand that some or many translators do not care to get involved with either of these situations and prefer a clear situation where the rate is set or the hourly rate is set. So my comments are addressed to translators who might care to dig into these changing parameters or want to be more involved in future and rather than any kind of absolute recommendation I am trying to trigger dialogue that might move us to better ways to do this.

  4. John Fossey

    I recently finished a job where the client, a large agency, sent the source document pretranslated through MT. As a colleague said who also reviewed the text, "It makes me stupid". I was not under any obligation to use the MT text and soon after beginning I deleted the MT targets and worked from scratch, once I realized that not only was the MT text wasting my time, it was actually taking longer to "correct" the target than to translate it all over again.

    MT engineers seem to feel they are obliged to tell translators how their systems work. However, they don't seem to understand how translators work. Take the output of an MT engine and while it may bear a resemblance to a translation - grammatically more or less correct, dictionary meanings transferred approximately - it's still just not the way it would be said by a native speaker of the target language. Not only that, but buried in the half-sense can be some very serious errors - negatives transferred as positives, nuances completely missed, etc.

    A professional translator, upon reading the source text of a segment, generally forms an immediate picture of how the target should read. There may be some details of terminology that need research, but usually the target segment is immediately apparent. When a MT target text is then presented, it is almost never the way the translator would say it and it plays games with the translator's mind. This tends to make it even harder for the translator to faithfully render the source into the target, harder than if the MT target had not been presented in the first place.

    Another factor that comes into play is the physical act of touch-typing. To touch-type a target segment from start to finish generally takes much less time than reading the MT target, putting the cursor to the right of incorrect words or phrases, deleting them by backspacing and retyping them, as is required to PE a MT text. PEMT interferes with the normal flow of a translators work.

    I question the statement about two translators with a significant discrepancy in work output. In this case I believe you are comparing apples and oranges, because if you take two translators at the same skill level, the output should be pretty similar. The lower producing translator is inevitably less-skilled - their skill is not going to suddenly increase by MT. It was the computer industry that invented the term GIGO - garbage in, garbage out.

    This is not to say that MT doesn't have its place. In "gisting" high volumes of text, such as when a company wants to monitor blog comments about itself in another language, it is useful. But where the result has to be presented in any serious application, say marketing or technical documents, it's another story.

    1. John,

      Just as it does not make sense to lump every translator out there into one single group and consider them all equivalent, it does not make sense to consider that the MT experience you describe above is typical of every MT experience out there..Clearly this is one scenario where you have incompetent practitioners but there are others also possible..

      There are "good" MT systems and many more bad MT systems out there especially amongst the instant do-it-yourself scenarios. A "good" MT system is one which would leverage a translation project and one which would be useful since it would actually produce output that would help work get done faster and that would be helpful to a translator.

      Even good MT systems can only solve certain types of problems where the volumes are large and the material is somewhat repetitive. I do not advocate any old MT as a way to get work done. Good MT takes care and effort and if you follow one of the links on my original post you will see much more detail on this.


    2. Well, I couldn't comment on whether this was a "good" or a "bad" MT system, only that it's the product of a great deal of investment by a global LSP, not a generic system generally available to the public. Certainly, it was not worth the investment for my use.

  5. maxi Schwarz-BastamiMay 1, 2015 at 4:43 PM

    Kirti, let's try to unravel this. It should begin with a trained professional in whatever field. As the expert he may decide at some point that the tools of his trade could use refinement, and he might call in another expert - a tool maker - and together they'd find a solution. The tool maker learns about the craftsman's work and needs, and uses his skills to create the perfect tool. This makes sense. Supposing that a customer, knowing nothing of the craftsman's skills, goes to a toolmaker, who also knows nothing about it, and has him design a tool, and then they try to persuade the craftsman to use that tool. That does not make sense. I have seen the latter happen a few times now in our industry.

    As a programmer, you have been hired by customers of translators, to create a tool for us to use. The customer has not been trained in our area of expertise, but has hired you anyway. You are not to blame, because you have expertly followed their instruction, and you have become caught up in a complex situation. You can only go by what you are observing as a non-translator from the vantage point of the customers of translators, and then trying to make sense of it all. You then write, explaining to us, and the results get all mixed up.

    Thank you for answering my first question, by the way. The thing is that your initial article comes across as giving advice to translators and yet it did not appear to be written by anyone who has knowledge of the profession, which was the case. Your next advice to John has a similar feeling: giving advice from your vantage point - but it's all skewed because you don't actually understand the translation profession. You understand the product that you were asked to create by customers of translation, who don't necessarily understand translation. This is the perfect time for us to give you information, so that you can catch up to that side of things. :) I'll attempt it in a couple of posts.

  6. maxi Schwarz-BastamiMay 1, 2015 at 4:46 PM

    There is an overall situation that you may not be aware of. There are attempts to cut out the expert and replace him or her. This is generally done by people who have no training in the field. Another thing that has happened is that with the Internet, a lot of people have seen an opportunity to make quick money out of other people's work. They form "agencies" but without anyone actually understanding the field, and then seek to automate everything. Some automation is legitimate, when it is suitable to the task, and usually trained translators will be able to define this. But a lot of it has become hopelessly twisted. We get weird scenarios like mega-companies manned by clueless folks using automated stations, hiring non-experts who may have become jobless in their own fields or want to make some extra money, because these will work for next to no money .... The translation industry is a big mess, but at the center there are still genuine professional experts who know what they are doing, watching this whole thing, and knowing what translation is. But they are not the most visible. There is so much nonsense out there, proclaimed so loudly, that sometimes we just watch the whole thing with a sense of futility.

    You can only describe what you see, but you are not a translator, and you are seeing what a segment of our customers are doing, which is also not the whole picture, and very far from the true picture.

  7. Maxi, Kirti is a marketing professional, not a developer. I long ago found it funny that many of those preaching the invincibility of MT were not from the translation industry, but who nevertheless were defining the future of translation. I have to say that Kirti is a lot more informed about the trade now than three years ago, and has toned down his rhetoric accordingly. Still, you do not expect him to talk from the perspective of a translator and understand all the nuances in translation. Nobody can unless he/she is an expert in the field.

  8. Couple of thoughts about some of the comments here:
    - Hourly payment.
    Paying translation by the hour would make a lot of sense. The fact that individual translators don't work with the same speed is not really an issue -- it's the same when you take your car to garage A or B. And this difference also holds for quality. So if as a customer you're not happy with the amount of time your translator bills you for (or with the quality of their work), look for one who bills for less (or delivers better work). The main obstacles for hourly billing are of a different type: the lack of predictability of the cost, especially in a supply chain where the one facing the end customer would have a hard time to quote without volume-based pricing, and the looseness of the supply chain. It's more natural to trust a car mechanic, doctor, lawyer, accountant etc. who you will actually meet than one of possibly thousands of freelance translators in distant locations.
    - Increased productivity.
    MT does typically and on average increase translators' productivity. Numerous studies have shown that. One issue there is however that how much it increases, and even if it does at all in a specific situation, depends on a lot of factors, of which the individual translator is not surprisingly the most important (if you take any two translators, their speed difference is likely to be bigger than any potential gain thanks to MT; in other words, good vendor management will have more impact on an organization's success than the choice of one MT system over another -- in terms of productivity).
    Another issue is that the typical productivity gains while significant are not in the order what would qualify post-editing as a "disruptive" innovation. The move to post-editing is complex and still comes with a good share of risks and unknowns (which all represent cost), and it's not so obvious at all that any organization will see a return on investment, if the gains aimed at are only those related to producitivity. So the inventive to go to post-editing is not so strong after all -- for the time being, well-run traditional LSPs don't have to fear being pushed out of the market by their post-editing competition.
    Because of the above I would have expected that post-editing as a practice would be more successful with LSPs that work with in-house staff, and create an incentive for those who don't to consider hiring translators. The other scenario where post-editing seems to be succeeding is in the enterprise, where there is large and consistent flow of translatable content in a fairly restricted domain.
    Now, I'm of course an MT believer. But I think that we need to focus more on other benefits of MT than mere productivity. For instance, if an LSP can guarantee their translators that they don't have to do any terminology research because the MT has taken care of that, translators may accept different rates. Or if the LSP uses MT as a way to automate quality control, there are important cost and more importantly process time savings to be made. Or if LSPs produce MT that is good enough to work with less-skilled translators (e.g. crowdsourced post-editing), etc.

  9. maxi Schwarz-BastamiMay 3, 2015 at 11:50 AM

    On the other side there is also the wish to do away with translators, with the professionals, and replace them. This is where MT comes along. This is not our first scenario, where the professional hires a programmer to make things more efficient, starting with his or her knowledge and experience in the field.

    Our first buck-making bunch come into the MT scene. One thing being played out is this: You run a text through MT for free, and then you hire a translator to "edit" rather than translate, and since editing is often done for a very low per-word price, you get something done for 0.03/word instead of 0.15/word - so 5,000 words cost you $150 instead of $750, but you can still sell it for .20/word, and make a nifty profit. Your own work consists of persuading your client, collect the translation, and get paid. The translator has to work 5X as long, and produce a lesser product, for a resulting per-hour pay which is dismal.

    Meanwhile, what we are seeing both in CAT tools (which are closer to the realities of translation) and MT, is a trend that asks for the source text to be written in a way that will suit the program. That's akin to me saying, "Please use only baby words, because I don't understand big words, but I'll still be your interpreter today." In some scenarios this might be apt for end clients who have very specific needs. But a lot of text (maybe the majority) doesn't lend itself to that. And an engineer responding to a call to tender, a lawyer making his closing arguments, a doctor describing a patient's symptoms and treatment, is not going to make such allowances.

    * One of your opening observations was about MT not being used well by people who don't understand when to use it and when not to use it, and how. A lot of your article is more suitable to customers of translators who think their role is to impose these systems on translators, and at least telling them when to use them. I think you will have seen a lot of such scenarios, because the people who understand translation and quality are not likely to resort to them.

    It is from these angles that I understand the sum total of things, and I am hoping that what I wrote can be somewhat a bridge of understanding. Because before you get that picture, we will talk past each other in regards to any of the details that have been mentioned.

  10. maxi Schwarz-BastamiMay 3, 2015 at 11:54 AM

    That being gotten out of the way, actual responses:

    >>"Though even with an hourly rate there would be throughput expectations, (if one translator does twice the volume of work that another does at the same hourly rate for example) so I think it is not so easy to walk away from this issue. ..... [in another post] ... but what would you recommend to the LSP or client who is faced with paying two translators the same hourly rate but sees that there is a significant discrepancy in work output?"<<
    The translation companies I work for hire professionals, and they usually start with small projects to get a feel for things. A lot of the modern entities advertise 50,000 word jobs in specialized fields which they give to the lowest bidder who are absolute strangers. If this LSP also doesn't understand translation and uses MT of any quality willy nilly, he'll get that huge variation - it becomes a crap shoot. Some of these entities may also not be able to tell about the quality and suitability of the translation once they receive it. There is a bigger picture here.

    But I think your question is actually an argument that a per word rate *will* give control. Not really - because we need quality, not just quantity. And a translator who is being paid a low per word fee to fix garbage tediously (as it happens too often) will not take as much care - or if he is competent and ethical, won't touch the work - then who is left to do it?

    Btw, I'm not that comfortable with the term LSP. We translators are Language Service Providers - that is what we do - and the middlemen are our customers who find translators for their end clients to provide the translations. I think that translation company and translation agency is a more honest term. ;-)

    >>..."LSPs are using a collaborative process in many cases to set a rate that involves at least a few translators if not all who will work on a project. This is when the rate is set based on the actual quality of the MT output.... <<
    They try to. And yes, "collaborative" is a buzz word that I get. Do you go for this "collaborative process" when someone hires you to do programming? That is - do they tell you how much you are going to be paid, and you accept that you will be able to negotiate this? Or do you have a fee that you set for your project? I expect it will be the latter.

    >>.. "There are "good" MT systems and many more bad MT systems out there especially amongst the instant do-it-yourself scenarios. A "good" MT system is one which would leverage a translation project and one which would be useful since it would actually produce output that would help work get done faster and that would be helpful to a translator .... <<
    That would be good. But has anyone actually talked to translators about what they need? I can indeed see that there would be varying qualities of MT systems.

    >>... "Even good MT systems can only solve certain types of problems where the volumes are large and the material is somewhat repetitive. .... <<
    TOTALLY AGREE. But then - what about CAT tools? Wouldn't they be more suitable? Or might you envision a clone of both?

  11. maxi, thank you for your detailed comments. While I am not a translator, I do speak 3 languages and have worked in a marketing and product development capacity with MT for 10 years. In this time I have seen many successful MT deployments working with experts, and many unsuccessful ones from hucksters and especially incompetent agencies. All my MT experience is working initially with large global enterprises and more recently with translation agencies (not translators) but I have interacted often with translators over the years in the MT engine development process. My blog is an information service where I share my experience and opinions and I do not claim any absolute mastery (as stated there) – often I write to generate dialogue and I say what I say in as professional and ethical a manner as I am able to.

    Assuming that most business translation is done to facilitate and enhance international commerce, in MT, much of what we focus on is content that would never go to a standard translation process because of sheer volume and time constraints e.g. knowledge base content, community content and information triage is needed like in eDiscovery where MT is used to determine which 1,000 documents from a million need to go to competent human translators. As an example you can see the relative information volumes of community and knowledge based content in this post: Given the short shelf life (utility to their customer) of a lot of corporate information it is imperative for many global enterprises to use MT.

    As the technology has evolved some of the MT developers have reached a level of competence where we know that the MT technology can also help in getting product documentation work done more efficiently. For example in the Omnilingua case study: 52% of the raw MT output from the Language Studio™ custom MT engine had no errors at all, and could thus be published as is. I have seen many other such cases but I will admit they are much less frequent than the scenario you describe with terrible output with many less competent MT developers out there.

    While you make a strong and very reasonable case for an hourly rate there are many agencies who think otherwise and I am simply presenting scenarios where it may make sense to use an approach other than an hourly rate. Because MT has more uncertainty in final outcome, it often requires more cooperation between the key players on things like schedule, compensation etc.. Everybody needs to make adjustments including the MT developers. This dialogue is an important one as it changes some basic rules we all know and again I understand that MT is something many translators will want to keep as far away as possible. It is not for everyone and the challenges are much greater in some language combinations than others.

    I have addressed (I think) many of the questions you raise in my blog and you can look there if you feel so moved. Thank you again for your comments.

    1. maxi Schwarz-BastamiMay 3, 2015 at 1:33 PM

      If I can summarize what I see from our exchange, MT is intended for particular specialized situations, and also not all MTs are alike. In actual fact, they are misused in many ways, and what typically comes across our desks are these cases of misuse. Novices should beware and be careful. Hopefully this exchange will help them.

      In regards to many agencies "thinking otherwise", I think that my extensive experience directly in the field for several decades might suggest differently.

      And again I would ask you whether you accept your customers dictating what fees you charge. I strongly advise my fellow translators to do what professionals in other fields do. If you can tell your accountant what you will pay him, then maybe that scenario makes sense - but I have a feeling that this doesn't happen.

    2. maxi I have covered this subject with a slightly different perspective a few years ago and you may be interested in the evolving perspective :

    3. maxi Schwarz-BastamiMay 5, 2015 at 10:25 AM

      I saw the lone comment to that particular post.
      I skimmed the article (it is lengthy). It has the same error in it that does not recognize that when a professional is hired, that professional states his or her fee. Btw, you have not answered my question about your own work. Do your customers set your fee, or do you set your fee?

      The other article states that translators have "more power than they realize". I have no idea what that is supposed to be about. It is a very simple thing. A customer approaches me with a project. I examine the project, and give my fee, after assessing if it's feasible. It has nothing to do with "attitudes" toward MT (suggested in the article), stating one would "never" do this or that thing. I examine each project on a case by case basis. That's it. Very simple. So far what has crossed my desk has been garbage. If non-garbage crosses my desk I will edit it at my usual per hour fee.

    4. maxi, I did answer your question above: " Because MT has more uncertainty in final outcome, it often requires more cooperation between the key players on things like schedule, compensation etc.. Everybody needs to make adjustments including the MT developers."

      This means that everybody including the translation buyer needs to adjust compensation policies as situations may require.

      The power that translators have is to refuse to work with bad output and be able to quickly understand this based on samples.

      Also if you go the original post (not just in LinkedIn) you will see many more comments.

    5. maxi Schwarz-BastamiMay 5, 2015 at 10:44 AM

      And as an experienced translator who actually works in this field, and is trained in the field, I disagree with your statement. Translators do not need to make adjustments of any kind whatsoever.

      You still have not answered a question I deem important --- In your own work, who determines your fee? Do you determine your fee, or do your clients tell you what they plan to pay you? Similarly, do you change your manner of doing your work, because a client has developed a system, without understanding your profession? I expect you will say no to both counts. Ditto here

  12. Hi there

    Very nice information you have updated here , i am biggest fan of your site.
    Thanks for sharing , I have enjoyed a lot your post.

  13. Wow such a great information, we all need a great interpretation to give our views to others.
    Really helpful.