Friday, May 8, 2015

Compensation for Post-Editing MT All Comes Down to Compromise

This is a guest post to provide an alternate and independent perspective on issues I have discussed in this blog. It is unedited and may or may not be consistent with my own views, but I think it is always useful to hear other views on these issues. I invite others who may wish to share their opinions on these MT related issues to also voice their opinions, especially those with very different views. I have added very few clarifications (in italics). My own views are outlined in this recent post and this older post which is still one of the most popular posts on this blog.

With thanks to @MattBramowicz for this submission.
If you ask a professional translator, chances are they will tell you that they’ve been requested at one time or another to provide post-editing or revisions for machine translated text. When this happens, more often than not many translators either begrudgingly accept or refuse the project altogether.
This can be attributed to 2 main reasons:

  1. The machine translated text is not very accurate to begin with, resulting in sometimes more work and time to decipher the text than it would take to translate it outright.
  2. The rates to post-edit are oftentimes significantly less than the translator’s normal translation rates.
As technology progresses, so do the methodologies for translation practices. Of course, technology is not just instantaneously perfect. It takes time to develop, improve, and advance until it is if not perfect, at least to a level that is efficiently satisfactory. At this moment in time, machine translation is in its “needs improvement” phase. While the accuracy has come a long way since its inception decades ago, with some languages being translated more accurately than others, it has still not reached a level that would be deemed “good”. However, for it to improve, it must still be a technology that is utilized so that there is a need to warrant the time and energy it would take to improve its process.

While many translators may refuse to provide post-editing services, the fact remains the need for the service is at an all-time high. Whether there are platforms created for mobile app translations, website content management systems, or even standardized document content formatted for machine translation, there are many resources that are utilizing the ubiquitous nature of web development to promote the post-editing or “hybrid” method of translation services. Therefore, translators could and should be more open to the process.

That being said, in no way should translators be “exploited” for their services either. There should be a compromise between translation service providers and translators to ensure fair pay is given for quality work.

For a compromise to take place, both sides must understand the other’s concerns and needs. Since we’ve already established the translator’s concerns, let’s go over the translation service providers’:
Hybrid translation is usually offered as a cost-saving method to the client to provide translation services for items that either would not have been translated in the first place
  1. due to the content not being of utmost importance, or there is too much text and the client wouldn’t be able to afford professional translation for it.
  2. Hybrid translation is usually offered as less than perfect translation method, not as good as professional translation, but still much better than mere (raw) machine translation.
  3. For reasons 1 and 2, the rate they charge the client is much less than for professional translation, so the budget for translators is much lower.
Now that we know both sides, the trick is to find a solution so that all parties’ needs are met.
For starters, we can look at post-editing translation jobs as opportunities for translators to make a little extra money on projects that for all intents and purposes came about thanks to hybrid translation being an option. Therefore, translation service providers can categorize these types of projects as such, and create a separate list of available translators who are willing to make themselves available to work on these from time to time as a means to earn some extra income. That way, TSP’s are only contacting translators who are willing participants in these projects. Likewise, TSP’s can and should keep more than the usual amount of translators on file for these types of projects, so if a translator is too busy to work on a certain project, they can decline without feeling pressured into taking it because the TSP has no one else available.

Second, while translators will have to work at a lower rate than ordinary for these projects, they should still be given a rate commensurate with the work involved. Translation Service Providers should set a rate at or close to the proofreading rate of translators. While the margins may not as be as profitable as professional translation service orders, the quantity of orders placed by clients should make up for it, provided the outcome is good enough quality that they want to order again. The best way to ensure that is the case is to pay the translators enough so that they feel compelled to do the best job they can. 

Also, deadlines should be reasonably set so that translators are able to work on these projects in between their traditional and more lucrative projects. However, many TSP’s tend to offer hybrid translation as a “speedy” translation service. While this can still be true, especially when formatting is taken out of the equation, they should still set reasonable deadlines with the client. If the client is in need of a rush delivery, TSP’s can offer a rush delivery surcharge option to the client, and pay the translator a little bit more for that particular project. If that isn’t an option, the project could be split between 2 or more translators in order to meet the deadline on time without putting too much strain on one translator.

Another option is for TSP’s to set up some sort of reward package for translators who consistently accept these projects and provide good services. For instance, for every 15 projects completed, let’s say, the translator receives a $50 bonus. While the amount may be relatively small, it still provides an incentive for the translators to accept as many projects as possible and it shows that the TSP appreciates their loyalty and hard work. 

Perhaps TSP’s could also pay translators an hourly rate instead of a per word rate. That way, for segments of text that are exceedingly difficult to decipher the meaning from, the translator is compensated fairly for their time. The only caveat for this option is that the end user is usually charged a per word rate (since there’s no way of knowing how much time to charge them up front). So there is a potential for the project to go over-budget if the time it takes to complete the translation is much longer than anticipated and surpasses the per word rate charged to the client. A possible solution to that issue would be to have the client agree to a clause that states the price they are given up front is simply an estimate, and the final amount may be more. This is the traditional practice for services like auto repairs, home repairs, etc., and most other service-based industries, which translation services are most certainly a part of as well. However, for some reason many clients don’t view it as such, and are reticent to agree to such terms up front.

While post-editing jobs may pay less than traditional translation projects in terms of word count, in some cases (depending on how the platform is set up and quality of machine translation), the process of post-editing can be completed much more quickly, and therefore result in a higher rate per hour than initially anticipated. In such cases, depending on the source text quantity, a translator can earn close to, if not the same rate, as traditional projects in a given time span. 

In the end, it all comes down to how the platform and post-editing process is set up by the TSP and the level of cooperation that can be achieved between the TSP and the translators.

About the Author
Matt Bramowicz is a content writer and graphic designer for Translation Cloud LLC (, a leading professional translation company located in Jersey City, NJ. You can follow him on Twitter: @MattBramowicz 


Thursday, April 23, 2015

How Translators Can Assess Post-Editing MT Opportunities

With the continued growth in the use of MT, it has become increasingly important for translators to understand better when it is worth getting involved, and when it is wise to stay away from post-editing opportunities that come their way. 

This is still a very fuzzy issue for most translators and I think it might be useful to share some information with them to highlight some of the key variables they could use to determine the most rational action given the facts at hand. For some, post-editing will never be palatable work, but for those who look more closely and see that PEMT is now just another variant of professional translation work that is much like other translation work, which can be economically advantageous when one is working with the right partners and the right technology in this case.  


We have seen that in the early days of MT use that there has been much cause for dissatisfaction all around, especially for translators who have been asked to post-edit sub-standard MT output for very low rates. Translators do need to be wary since many LSPs deploy MT technology without really understanding it, with the sole purpose of reducing costs, and with no understanding on how to produce systems that actually enable this lower cost scenario or interest in engaging translators in the process. Thus it is worth translators learning some basic discrimination skills to determine and establish some general guidelines to understand the relative standing of any PEMT opportunity that they are presented with.


The following checklist is a useful start (IMO) that every translator should consider when deciding what kinds of PEMT opportunities are worth working on.
  • Understand the very specific MT output that you will be working with as every MT engine is unique and assessments need to be made in reference to the actual output you will be working with.
  • Determine if the LSP understands what they are doing with the MT technology and can respond to feedback on error patterns. There are many “upload and pray” efforts nowadays that create very low quality systems that are very hard to control and challenging for translators to work with.
  • Understand the MT technology that is being used as not all MT is equal. There are many variants and you should know what the key differences are. Systems that allow feedback and have more controls to correct errors after the MT engine has been built and accept ongoing corrective feedback will generally be better to work with.
  • Have a basic understanding of the MT methodology which means at least an overview of the rules-based and statistical approaches. This can give you a sense for what kind of feedback you can provide and also help you understand error patterns.
  • Understand that MT engine development is an evolutionary process rather than an instant solution that Google has led some of us to believe. Professional MT deployment is a molding process that evolves in quality through expert iteration, and is typically done to tune an engine for a specific business purpose to help an ongoing high volume translation production need. MT makes much less sense for random one-time use.
  • Understand the basic quality assessment metrics used with MT. BLEU scores are often bandied about with MT systems and often interpreted incorrectly. If you understand them you will always have a better sense for the reality of a situation as incompetent practitioners use and interpret these scores incorrectly all the time. The BLEU scores are only as good as the Test Sets used and so try and understand what makes a good Test Set as described in the link.
It is wise to use technology when and only if there is a clear benefit, and this is especially true with MT. An LSP should have a clear sense that the productivity of the translation project will be improved by using the technology otherwise it is detrimental in many ways.This means that there needs to be clear idea of what typical translation project throughput is before and after the use of MT. And a trusted way to measure how MT might impact this productivity. 

  • Thus MT only makes sense when it boosts productivity or when it makes it possible to provide some kind of translation for material that would just not get translated otherwise.
  • Translators should also understand that lower rates are not necessarily bad if their throughput is appropriately higher.
  • Finally, MT error patterns tend to be consistent so it makes sense to approach corrections at a chunk level rather than an individual segment level. 


Much of the dissatisfaction with PEMT work is related to compensation. My post on PEMT compensation remains the most read post on this blog even though it is now 3 years old. But I think if you understand the specific MT output you are dealing with and it’s impact on your throughput you can make an informed decision.


It is wise to remember that a lower rate does not necessarily mean less overall compensation as the following totally hypothetical chart explains. (The productivity benefits are more likely to be shared less generously). The best LSPs will have an open and transparent process in setting this rate and translators will be involved to ensure that the rate is fair and reasonable and based on actual MT output quality rather than some arbitrarily lower rate “since we are suing MT”. Also expect Romance language rates to be lower than tough-for-MT languages like Japanese and Korean if editing effort is used a criterion for setting the rate.


Much of what I have covered here was presented in a Proz presentation that is still available as video (slides with voice) for those who want to see and hear more details of the summary presented in this post.


As a complete aside this is for those who think that Genetically Modified foods are harmless. Here is a quote from a biotech company leader that you might want to consider the next time you eat corn from a US supermarket:
“We have a greenhouse full of corn plants that produce anti-sperm antibodies.” ~ Mitch Hein, president of Epicyte, a California-based biotechnology company.

And to end on a cheery note, I was very impressed by the musicality of  this song and thought others might want to hear it too.

Monday, December 1, 2014

Machine Translation Humor Update

It has been sometime since I first wrote a blog post about MT humor primarily because I really have not been able to find anything worth the mention, until now, and except for some really lame examples about how MT mistranslates (sic) I have not seen much to laugh heartily at. It seems a group of people on the web have discovered the humorous possibilities of MT in translating song lyrics which might be difficult even for good human translators. (It really seems strange to be saying “human translator”.) 

I should point out that in all these recent cases one does have to work at degrading the translation quality by running the same text through a whole sequence of preferably not closely related languages.

It has often surprised me that there are some in the MT industry who use “back translation” as a way to check MT quality, as from my vantage point it is an exercise that can only result in proving the obvious. MT back translation by definition should result in deterioration since to a very great extent MT will almost always be something less than a perfect translation. This point seems to evade many who advocate this method of evaluation, so let me clarify with some mathematics as math is one of the few conceptual frameworks available to man where proof is absolute or pretty damned certain at least.

If one has a perfect MT system then the Source and Target segments should be very close if not exactly the same. So mathematically we could state this as:

Source (1) x Target (1) = 1

since in this case we know our MT system is perfect ;-)

But in real life where humans play on the internet and you have DIY MT systems being used to determine what MT can produce, the results are less likely to equal 1 which is perfect as shown in the example above.

So lets say you and I do a somewhat serious evaluation of the output of various MT systems (each language direction should be considered a separate system) and find that the following table is true for our samples by running 5,000 sentences through various MT conversions and scoring each MT translation (conversion) as a percentage “correct” in terms of linguistic accuracy and precision.

Language Combination Percentage Correct
English to Spanish 0.8 or 80%
Spanish to English 0.85 or 85%
English to German 0.7 or 70%
German to English 0.75 or 75%

So if we took 1,000 new sentences and translate them with MT we should expect that the percentage shown above would be “correct” (whatever that means). But if we now chain the results by making the output of one, the input of the other, we will find that results are different and and get continually smaller e.g.

EN > ES > EN = .8 x .85 = 0.68 or 68% correct

EN > DE > EN = .7 x .75 =  0.525 or 52.5% correct

So with MT we should expect that every back test will result in a lower or degraded results as we are multiplying the effect of two different systems. Since computers don’t really speak the language one cannot assume that they have equal knowledge going each way and if you provide a bad source from system A to system B you should expect a bad target as computers like some people, are very literal.

So now if we take our example and run it through multiple iterations we should see a very definite degradation of the output as we can see below.

EN > ES > EN(from MT) > DE > EN = .8 x .85 x .7 x .75 = 0.357 or 35.7%

So if you are trying to make MT look silly you have to run it through multiple iterations to get silly results. It would help further if you chose language combinations like EN to Japanese to Hindi to Arabic as this would cause more rapid degradation to the original English source. Try it and share your results in the comments. 

So here we have a very nicely done example and you should realize it takes great skill for the lead vocalist to mouth the MT words as if they were real lyrics and still maintain melodic and rhythmic integrity so be generous in your appreciation of their efforts.

This video shows very effectively how using multiple languages very quickly can degrade the original source as you can see when they go to 64 languages. Somehow words get lost and really strange.

And here is one from a vlogger who really enjoys the effect of multiple rounds of MT on a songs lyrics. She is a good singer and is able to maintain the basic melody without breaking into a smile so I found it quite enjoyable  and I would not be surprised that some might believe that these were indeed the lyrics of the song. She has a whole collection of recordings and has what I consider are high production values for this kind of stuff.

And she produces wonderful results on this Disney classic "When you paint the colors of your air can" which used to be a favorite of my daughter. I actually think the song from the Little Mermaid is much funnier and was done by just running it only through four iterations in Google Translate, but since I could not embed it here directly I have given the link.

 Here is another person who has decided that 14 iterations is enough to get to generally funny with this or any pop song. I'm not sure how funny this really is since I don't know the original song.

 So it appears that we are going to see a whole class of songs that are re-interpreted by Google Translate and it is possible to get millions of views as MKR has, and probably even make a living doing this.  So here you see one more job created by MT.

So anyway if somebody suggests doing a back test with MT you should know the cards are clearly stacked against the MT monster and the results are pretty close to meaningless. A human assessment of a targeted sample set of sentences is a much better way to understand your MT engine.

Hope you all had a good Thanksgiving vacation and are not feeling compelled to shop too fervently now. 

In this time of strife and distrust in Ferguson it is good to see spontaneous goodwill and instant musical camaraderie between these amateur musicians. 


My previous posts on MT humor for those who care are:
Machine and Human Translation Based Humor

Translation Humor & Mocking Machine Translation