Pages

Friday, November 8, 2013

Translator Strategies For Dealing With PEMT

I recently had the opportunity to speak to a group of translators and interpreters about machine translation and how it increasingly impacts their work lives. Given that more and more agencies are using MT nowadays,  it is now much more likely that a translator might be approached to do post-editing work and thus my message to the translators at the event focused on how to assess these opportunities (or hazards) and maximize the benefit of any interaction.

Translators have much more power than they realize and I predict that they will eventually learn to separate the wheat from the chaff. Translation agencies will hopefully figure out that while MT technology will proliferate, the shortage of “good” translators will only intensify in a future where global companies want to translate 10X or more the volume of information they do today. 

We see many examples of MT use by agencies today but very few of these would qualify as skillful and appropriate and even fewer would be considered fair to the post-editors. It is my sense that MT technology will only offer long-term competitive advantage to those who use it with skill and real expertise and have skilled translators involved in the process. It is very easy to dump data into an instant MT portal and get some kind of an engine, but not so easy to get an engine that provides a long-term cost and efficiency production advantage.

If you are one of those translators who feels that they will NEVER do post-editing work or have decided that you simply don’t want to do it because you have plenty of “regular” work then this post will probably not be of any interest. I am one of those people who believe that MT will continue to gather momentum and that it is useful to translators to understand why and determine when to get involved or not. (And this is not just because I am involved with the sales and marketing of this technology.) It just simply makes sense at a common sense level. The first thing to understand is that all MT engines are not equal and that free online MT is not the best example of professional use of this technology even though it can be surprisingly good in some languages.


image
Since there is a great deal of variation in the specific MT output that translators are expected to post-edit,  I think it makes sense for a translator to understand each unique opportunity as it comes along, and determine whether it is worth his/her time and engagement. Some MT opportunities can pay better than standard translation work, if the word rate to MT quality ratios are properly determined, and thus I think it makes sense for translators to understand when this is actually the case. Early experiences with incompetent or unscrupulous MT practitioners have helped PEMT work develop a reputation for being mind-numbing work that is poorly compensated. This IMO is more a reflection of the quality of these early efforts than of the real possibilities of the technology when used with expertise. 

image

Thus I have come up with a simple checklist for a translator to evaluate a potential PEMT “opportunity” and decide whether to engage or not.

1) Compensation is linked to actual work effort

The MT output quality to word rate (financial compensation) relationship is a fundamental issue for translators. It is important to understand the “average” output quality of the MT output and then understand the effort required to fix it to target quality levels, and ensure that it is related to the compensation offered.

Since with PEMT we are generally talking about asking translators to accept a lower rate than they normally charge, it is important that there is a modicum of trust with the agency in question. This would allow a fair and reasonable rate to be established that matches the effort required to get the MT output to required target levels. This subject is dealt with in some detail here and here. The better you understand your own personal productivity with the specific MT output you are dealing with the more informed your decision will be. The specific effort level can be assessed quickly by doing a small test with a “representative sample” of a 100 or so sentences. The throughput measurements you make can then be used to extrapolate and calculate an acceptable rate. 

So if your normal throughput is 2,500 words/day (313 words/hour) and you find that with the test MT output you can expect to do 5,000 words a day, it would be reasonable to accept a rate that is 60% of your normal rate and even 50% might be fair if you feel the sample is very representative and you do not mind this type of work. (I would err on the higher side as the test is only as good and representative as your test sample.)

A critical skill to develop in these scenarios is the quick assessment of the MT output quality and determine what your work throughput and thus acceptable rate is. Remember small grammar and word order errors are much easier to correct than word salad and bad and inconsistent terminology problems which require research. The rapid assessment of the quality of the MT output should be an important part of determining when a project is worth doing or not. Having a basic understanding of BLEU, Edit Distance and other methodologies is useful as this can expedite assessment of the PEMT opportunity. Asia Online offers free software to run BLEU and develop your own error classification based calculations.
image

Some things to be wary of include:
  • Agencies that establish an arbitrary lower word rate independent of language and MT output quality. This is a pretty good clue that they don’t know what they are doing and a sign that there will be dissatisfaction all around.
  • Agencies using DIY MT who don’t really understand what they are doing. Expect great inconsistency and variability in the output quality and usually lower overall quality which means a greater PEMT effort.
  • Agencies that have the same rate for tough languages like Japanese and easier languages like Spanish PEMT work. I would generally expect that that the effort would be greater for tough languages and so they should be paid at higher rate.
  • Agencies that give you MT output that is lower in quality than you could get on your own from Google or Microsoft. This is a sign that they do not understand what they are doing.
  • There are many agencies out there that have very little understanding of the complexities of MT and are only using it as a way to reduce costs. They will give you crappy output to edit and expect you to fix it for a fraction of a reasonable rate. Identify these agencies and let fellow translators know who they are. Avoid working with them.
  • Hourly rates may actually be better for some kinds of MT projects where the translator is expected to only do a partial correction. Research suggests that it is very hard to define how far a partial correction goes.
An example of agencies that do it right and use objective and trusted measures to establish fair compensation include Advanced Language Translation and Omnilingua. It is worth understanding their process.

2) Trust and communication around technological uncertainty

I think that one of the main reasons MT has taken so long to gain momentum is the low levels of trust within the supply chain and unfortunate early experiences with MT where rates were lowered unfairly and translators were expected to bear the brunt of incompetent use of MT technology. The stakeholders all need to understand that the nature of MT requires a higher tolerance for “outcome uncertainty” than most are accustomed to. Though it is increasingly clear that domain focused systems in Romance languages are more likely to succeed with MT, it is not clear very often how good an MT engine will be a priori, and investments to measure this need to be made to get to a point to understand this. 

The stakeholders all need to understand this and work together and each make concessions and contributions to make this happen in a mutually beneficial way. This is of course easier said than done as somebody has to usually put some money down to begin this process. The reward is long-term production efficiency so hopefully enterprise buyers are willing to fund this, rather than go the fast and dirty MT route as some have been doing. Agencies that are new to MT and post-editing are those most likely to get it wrong and translators should seek out agencies that are sensitive to resolving the uncertainty in a fair way.

Some specific things that translators can watch for include:
  • The quality of the dialogue and rapport with the project managers at the agency.
  • Some agencies provide very clear examples of what they expect you to do with different kinds of errors. This is a good sign and helps focus the work in the most efficient way. Some like Hunnect develop an online training course for post-editors to help clarify this.
  • The agencies that are willing to work with translators to deal with this technological uncertainty are the ones to focus on. Again Scott Bass from ALT provides wise words on PEMT Best Practices and provides an example of what a win-win scenario looks like.
3) Ability to interact with and control MT technology 

One of the common complaints about PEMT is about the drudgery of error correction work. This does suggest that not all translators want to do this kind of work or are well suited to it. Many translators are also seeking to provide feedback and steering advice to the MT system to reduce the drudgery, however, not many MT systems can properly use and leverage this type of feedback. Some like the Asia Online Language Studio are designed from the outset to utilize this type of feedback. We are seeing now that many translators do realize that MT can be an aid, much like TM, to get repetitive translation work done faster. MT offers “fuzzy matches” for each new segment that is translated through the system. Good MT systems will produce the equivalent of high quality fuzzy matches and will be much more consistent in output quality than what most of us experience with free online MT (or most DIY efforts) as shown in the second graphic above. Bad MT systems will be inconsistent, unpredictable, produce lower quality output and generally be unresponsive to any corrective feedback, especially when the practitioners are simply dumping data into an instant MT engine making portal.
The following are some characteristics of superior MT platforms:
  • The ability to provide some initial error pattern feedback to reduce mind-numbing correction work.
  • Noticeable improvements in quality with relatively small amounts of corrective feedback.
  • The ability to control the MT output with terminology or repetitive error pattern corrections at run time in addition to the upfront overall training, as this can greatly enhance the speed of the post-editing work.
  • A defined process to take small amounts of corrective feedback to improve the engine BEFORE a production run to reduce the post-editing effort.
  • The ability to control the overall linguistic style of the translations to requirements.
This outlines some kinds of corrections that can be run at the time of running a translation through an existing Asia Online MT engine.

image
Examples of correcting problematic source text to make the post-editing task easier.
image
Example of using preferred terminology in the event that the original training chooses other terms.
image

If you have a good feeling about all three items in the list above PEMT can be just another kind of translation task and can sometimes be one that offers greater financial reward. 


There have been several studies of varying quality that examine how PEMT compares with regular translation approaches and we see mixed results and often experimental bias. I just saw this study on The Efficacy of Human Post-Editing for Language Translation from Stanford that attempts to measure this in as objective manner as possible. I like that they also summarize many previous studies. Some may find fault with this one too because they use oDesk, even though these were translators who had passed a 40 question skill/competence test. IMO the study is perhaps more objective and rigorous than most I have seen from the localization community and I think it is worth noting the key findings and is worth a closer look by anybody interested this issue. They ran a carefully monitored regular vs. PEMT comparison test for 3 languages (English to Arabic, French, and German) and found the following:
  • Most translators found the MT (Google Translate) useful and preferred it to not having a suggestion
  • PEMT reduces the time taken to get the task done
  • Across languages they found that the suggested translations improve final quality
  • Across languages, users provided the following ranking of basic parts of speech in order of decreasing translation difficulty: Adverb, Verb, Adjective, Other, Noun.
“Our results clarify the value of post-editing: it decreases time and, surprisingly, improves quality for each language pair. Our results strongly favor the presence of machine suggestions in terms of both translation time and final quality. If translators benefit from a barebones post-editing interface, then we suspect that more interaction between the UI and MT backend could produce additional benefits.”
I would love to hear what other translators may have to share about their PEMT experiences, both positive, negative and suggestions they might have to improve the process.  I would like even more to hear what they think about an ideal post-editing environment or workbench and recommendations they would have. 

If you are interested in my slides from the MiTiN presentation you can find them here.

37 comments:

  1. In my experience, PEMT is not much different to the process of verifying a translation performed by a non-native. You compare the translated text with the originall and assess whether or not the two documents convey the same message accurately.
    The main criticism I have of MT is that it has the potential to make more disastrous translation errors than is likely with a human translator. The consequence of this is that the correction of the erroneous translation can take as much time as if you had perormed the whole translation yourself.
    I have not had the opportunity to compare different MT processes as I only receive the occasional surprise request to verify a document that troubles one of my clients. Having handled several such tasks, I can only say that the standard has been similar in all. Maybe they are all using the same MT provider.
    By Brian Flack

    ReplyDelete
  2. I'm among those who don't plan on doing PEMT, unless I will be able someday to setup and train my own local MT engine and incorporate it as part of *my* workflow, although I'm not sure how beneficial even this type of solution will be compared to referencing quality self-made TMs, Glossaris, and well, the subject specific expertise for which a professional translator is hired.

    MT is indeed an aid, but in its current form it is not a professional aid, it is an aid for the brokers and generalist, often questionable-quality translation working in the low market segments (as evident by the study cited above); the former abuse it and the latter use it as a hand holding aid. The truth is that currently MT is not being used by merit, but for pure financial benefit of the brokers that operate in the shallow, muddy water of the pond they call the "industry" (i.e. an artificial group of "service providers" that leech off the translation profession just because they manage to make some money). It is a smoke and mirrors game in which the brokers and the technology developers try to exploit both clients and translators in a desperate attempt to survive in the toxic environment that they helped create.

    I thank you for writing this article and for touching the subject of fair compensation that commensurates with the effort (and its abuse), as well as pointing out that most brokers out there have no clue about MT and they just use it arbitrarily to reduce costs (for them) and increase margins (again, for them) with no consideration to ethics and professionalism. The mere talks about "controlled language" (i.e. reducing the quality level of the source to better prepare it for MT) it a testament on the true and probably only motive (financial) of using MT. In the long time those abusive and unprofessional practices will backfire on all the entities involved, only by then who knows how many true professionals, as opposed to brainwashed cogs who are trained to to PEMT according to some "metric" and "standard" or whatever, while having no clue about what translation is really about, will be left in this business.

    Also, I don't quite see the connection between translation and PEMT. Translation required a specific set of skills, while PEMT is a (mind-numbing) process that requires a completely different set of skills. It is like programming and debugging, seem superficially similar, but completely different things in practice.

    One technology that is almost never mentioned in the MT as a productivity aid debate is dictation (text-to-speech), with clear evidence of comparing (in a relatively bad day) and/or exceeding the best MT scenarios in terms of productivity, not to mention the quality. If half the budget and effort put into MT and other snake-oil profit-driven (not for the translators, of course) "solutions" development were invested in optimizing and developing dictation technology. I suspect things were different.

    ReplyDelete
    Replies
    1. Shai,

      I would be careful to characterize all MT as being the same thing. While you may not have interfaced with properly customized systems there are an increasing number of people who are doing this and for them, MT is indeed a professional aid even though it is difficult for an individual translator to get there on his own, except in a limited way with RbMT systems as described by William below.

      The interaction experience with good MT systems is not so different from working with fuzzy matches. Sometimes they are very useful and sometimes they are not but in general the "good MT systems" do make it easier to get the work done faster by giving you many more useful MT segments than you are used to seeing on Google. These "good" systems get much less publicity than Google MT, or bad MT experiences with LSPs who do not know what they are doing.

      A lot of the advances in MT do actually come from research in speech technology and Kevin Lossner has written in detail how he uses STT technology to improve his own productivity even though he stays far away from MT.

      I think that many translators dismiss MT as a failure because they continue to measure it against human translation where some linguistic insight and finesse is required. Indeed it will fail there -- like trying to get a mentally handicapped person to do calculus when you know they can add and multiply with great accuracy at times, but only if you speak nicely to them.

      The real focus of MT has been to improve the efficiency on repetitive texts like user documentation, car service manuals etc.. and then address translation problems that would be ignored without the technology. Especially involving the huge volumes of new information that is created by companies and customers around their product use experiences, particularly knowledge bases and user community feedback.

      I am sure there will be many more bad MT experiences exposed in future, given the many instant MT engines being made out there, but for those who take the trouble to understand what characterizes the better MT experience, the work is not going to be so different from working on documentation translation projects with TM. Good MT experiences require skill and expertise that are not widely available yet but in time may become commonplace.

      Also, you can actually make MT systems now with many Moses tool kits, but you may find that the undertaking is significantly more complicated to do well than many have led us to believe. These tool kits do not eliminate the need to understand what you are doing and many who try this may be better off just cleaning up their TMs and using speech to text technology to improve productivity.

      Delete
    2. Kirti,
      Thank you for the details reply.
      I don't classify all MT engines as the same thing, but currently I would say that the vast majority are generic, snake-oil type of solutions that are used by unscrupulous brokers to cheat people (clients and translators) out of their money.
      I also want to make it clear that I'm not against technology, I'm against its abuse. In the translation marketplace each discussion about technology needs to be divided into two: the technology itself and its abuse. In practice this distinction is becoming harder and harder for many to maintain as technology abuse and obsession it is gradually taking over the large market segments.

      I'm following your blog because you are one of the few from the "MT side" that I consider as honest and professional, and I appreciate perspective. I only think that while comparing using MT as working with an existing TM is true, it cannot be automatically considered a professional tool. Some agencies have huge TMs that are as much garbage as they are big. They are the result of many years of using armature, cheap human translators tht know nothing about the subject matter that they translate nor have great skills as translators. These types of TMs are not something that I will consider using. Actually, I don't work with TMs other than my own (with the occasional exception of a TM from a trusted colleague), so it all depends on the quality of the TM. Therefore I differentiate Incorporated MT into one's own workflow and PEMT (i.e. getting an MT output to review and "correct"). The former is indeed a tool, while the latter is not related to translation. Even when setting aside all the abusive practices that grow around it, in its core this is an editing task, and editing and translation are not the same thing. They require completely different skills, and a good translator is not necessarily a good editor and vice versa. Sadly, not too many understand this either.

      I'm no expert, but I did experience with SMT. Both generic and somewhat custom made. My experience is very limited, certainly compared to yours, but I concluded that in this point in time this technology is not for me, partly probably because I work in fields that are not very suitable for MT at the moment. The only scenario in which I see my self using MT is when running a local customized engine that will yield "fuzzy matches" closer to my own style and voice. I acknowledge that maintaining such an engine is not a trivial task and that it is a constant work that required expertise. I'm no prophet and I never say never, so my point of view might change later on, but I still have trouble seeing the advantages of MT for specialized experienced translator with good reference materials that knows how to properly incorporate and use TEnTs and other translation supporting technology tools in his or her workflow.

      Furthermore, to date, most researches that I saw about MT productivity and quality gain were done in comparison to likely bad and cheap human translation (I saw two researches that used translators from two MLVs - one I know personally to produce appalling quality translations; and now the one using oDesk). I don't consider these to be serious researches because they use poor quality human translations that many times can hardly be called a translation compared to what professional produce as a benchmark, and this is kinda of cheating in my book (especially in light of the lack of transparency about the quality of the human translation).

      So yes, MT is a tool, but as any other tool it is not for everyone and suits a certain type of work better than others. Moreover, PEMT is not translation it is editing, and the fact that translators are mentioned as the target audience for it instead of editors is another testament to the lack of knowledge about the core work and skills needed.

      Delete
    3. Clearly PEMT is not for everybody and many translators will choose not do it.

      From my perspective the best reasons a translator might have to avoid it are:

      1) The pay rate is not related to the effort and difficulty of the work -- i.e. too much work for too little pay
      2) The work is "mind-numbing" or disagreeable at a personal level
      3) Your philosophical objection to "helping" MT get better

      But outside of these reasons I am not sure other objections make as much sense. MT is used because it can help to reduce the cost of getting repetitive material done and get it done faster. Speed and cost have become more important for many kinds of content that do not have a long shelf life. Many agencies use MT as a ploy to push rates down and translators can make each other aware of these kind of initiatives and avoid these agencies. There are also agencies who are unscrupulous in other ways -- avoid them.

      However, I think it makes sense for translators to take a look at the specifics of an individual PEMT "opportunity" and then decide whether to engage or not once the fair and reasonable compensation test has been passed. (Assuming that you would consider it all). It is then possible for PEMT to be just another aspect/kind of translation work -- which some will be willing to do and some won't. So it goes.

      There are still translators who do not use TM and the market is broad and diverse enough that they will continue to find work. I expect that the demand for translators who do not want to do PEMT will continue into the future as long as they have skills and competence that is valued elsewhere.

      I can understand that you are wary of using a lot of TM as well since it if often not what it is said to be. I have been involved in enough MT projects to realize that the TM used is often problematic. It is VERY hard to keep TM consistent and clean over a period of years. I know that on average we find that 15-25% of the TM provided for most SMT customization is not useful because it cannot be trusted for consistency and quality even at an automated assessment level.

      Delete
    4. There is also another reason, in my opinion at least.
      The core translation process is to capture the meaning of a sentence and transfer it into another language. The individual key competencies involved in this process (excellent command of the language, expertise in the subject field, good writing and expression skills, etc.) can be measured empirically, but the elusive core skill that differentiates one translator from another and enables good translators to create good and effective translations, is not just the sum of these parts.
      This is why good translators usually don’t use other people’s TMs, not necessarily because of quality, but mainly because they have their own voice, style, expertise and way of doing things. They have their little “tricks” and secrets to make the text more effective or appropriate to the target audience. They simply don’t care to trade their profession for a generic voice and style, let alone one that was created by a machine that was trained on and is repurposing decades of questionable quality corpuses. As far as I know, most SMT training is done with huge TMs provided by MLVs or other brokers who have interest in this technology. From my experience, these TMs are generally amongst the worst. Even a simple manual can be ruined by a skill-less, cheap translator – as evident in many languages in the form of the manuals that come with many consumer electronics and even some more specialized devices. Many (I would even say the vast majority) of these were translated by the MLVs and are stored in the TMs that are used by MT developers and researches.
      Translation is a cognitive process. When I work on a text I need to work on the entire text, not only on parts of it. I use a TM mostly to maintain consistency, not to repurpose half-random bits and pieces of text that although mathematically might be a ‘match’ of some kind, but in context or purpose could be misleading or not the best fit.
      In a way, MT is only as good as the data put into it, and in many cases this data is not good enough to begin with, partly because of its eclectic nature. My biggest pet peeve about translation supporting technology is that a lot of times the developers have only superficial familiarity with the core competencies involved; they often consult the big brokers in the market, who also don’t really know the profession they supposed to represent, but have the resources and influence to put the technology into (ab)use – which is probably enough for the developers who merely look for a business opportunity.
      You recommended avoiding unethical agencies. First, the translation marketplace is filled with brokers, and brokers are not agencies (= professional practices) there is a huge difference. Secondly, it is a good recommendation but turning not very practical for many, especially newcomers or naïve, as the unethical practices and exploitation continue to grow; supported in a way by the silence demonstrated by the technology developers whose technology is being abused (and some of them are well aware that their technology is being abused). This is the root of the disassociation in the translation marketplace. The technology is developed by “programmers” with a mathematical type of analytical thinking that is different from the analytical thinking that is needed for translation, and at least some of them are developed with the purpose of serving the needs big translation buyers and brokers (i.e. getting a cut from all the money that seem to exchange hands), which a lot of times have a lot to do with business considerations and almost nothing to do with professional ones. This created an industry within the translation marketplace that trades in words as a raw material, and professional translators have nothing to look for there.

      Delete
  3. Great article. As a freelancer, my experience until now with PEMT is not related to companies, but to my workflow for some materials I translate. I use ProMT in a workflow where a small chunk of text is translated, reviewed and then I create glossary entries to improve quality, send this back to ProMT and then translate another chunk, repeating this process until the end of the text. Quality gets better and better and time used in preparing glossary and all other steps becomes irrelevant when you see how fast and precise you can finish the hole text. Of course, I'm talking about a very small amount of text if compared to a company volume, but I do believe companies which use a feedback system to improve MT quality are able to provide great material to be post-edited and I surely would evaluate the possibility to partner with them. It would really be a win-win.

    ReplyDelete
  4. I have summarized my two main points why I'm cautious about MT in my recent blog post here:

    http://www.englishczechtranslator.com/blog/why-i-dont-think-supporting-machine-translation-systems-is-a-good-idea/


    Kirti: "no reason to consider a PEMT job differently from any job that requires you to use available TM"

    I dare to disagree, not only for reasons outlined in my blog. (Of course, those reasons may not be the case with every project, but anyway.) Technical aspects aside, TM contains human translation. Professionally, I find it somewhat derogatory to reprocess/recreate something "soulless", without a human touch (at least as long as the difference is noticeable) just because someone wants to save money or time.

    Up to everybody, but IMHO words are not bricks - I have the impression that PEMT adds or supports the routine/robotic approach to translation (TM might be sometimes perceived as having similar effect, but it is not the same thing) which compromises the genuine creativity and joyful work.

    ReplyDelete
  5. I already said it elsewhere: machine translation is like a machine gun – fast, destructive and devastating for your client’s reputation. In case of PEMT, PE is similarly devastating - for a (former) translator’s writing skills (and mind).

    ReplyDelete
    Replies
    1. Yes, Valerij, concerns about the mental effects of PEMT go back over 30 years (see http://www.mt-archive.info/Aslib-1981-Bevan-1.pdf for example), but none of the snake oil crowd want to talk about these things, much less have them studied and documented more extensively. Stefan Gentz shared an interesting comment from a recent MT discussion at a Romanian conference, in which one post-editor admitted that after a long day's work he felt unsafe operating a motor vehicle. He wasn't joking I think. Of course none of this matters to the world's linguistic sausage producers who are perfectly willing to add the disposable brains of willing monkeys to their gruesome grind.
      I appreciate Kirti sharing his productivity estimates once again. As on previous such occasions, however, I cannot help but note that these are well below the productivity once can expect from a good translator working in a familiar subject area with a well-tuned dictation system. Pursuing these silly MT systems offers no real benefit except to those who pocket the IT consulting fees for set-up and maintenance; they are a scam which remind me very much of my days as an IT consultant at the heart of the Y2K boondoggle.

      Delete
    2. Valerij/Kevin

      Given that the interest and research in MT continues in spite of it's less than stellar historical record suggests to me that this is a problem that many want to solve, for many reasons other than upsetting professional translators like you. I expect this momentum will only build as there are more and more documented successful MT-use case studies.

      As many have stated, there is a shortage of good translators so those who want ever flowing streams of business content translated need to figure out how else large/small translation projects can be done. Where the content is roughly predictable and the domain is known and definable, MT will likely have success. And maybe in your opinion the people who are engaged in this are lesser beings. So it goes.

      Perhaps your viewpoints are based on the quality of German MT which is much lower than that for Romance languages, or perhaps you just dismiss this as an effort that can never succeed.

      As long as PEMT can deliver final translations that are close to (or sometimes even better) than professional translators at a cost that is lower and at a speed that is faster than a traditional human-only approach there will be good reason for people to use the technology. The technology can only build momentum if there is a real ROI for at least a few if not many.

      Much of the corporate content (documentation and manuals in particular) that professional translators have been working on historically is now increasingly viewed as being of lower value and having a very short shelf life - both to the end-customer/consumer and to the overall business mission of enabling international commerce. Thus it makes sense for globally focused enterprises to see if more of this low-value or short shelf-life content can be translated at lower cost using more automation. Many global enterprises who adopted MT technology long before LSPs and translators have, apparently think this makes sense. Perhaps MT is even becoming an imperative as they see that unstructured, short shelf-life content found in social networks is increasingly influencing international business potential and prospects and creating new kinds of business translation challenges that do not fit well in the old production models.

      There are some that will try and engage in constructive dialogue and there will be some who will just dismiss the other viewpoint summarily without hesitation. (In the US we have a mindset generally referenced as the Tea Party, that summarily dismisses, discredits and criticizes any attempt at governance with self-righteous vigor with little if any constructive feedback. Fortunately they are being increasingly marginalized.) Those who summarily dismiss the technology also often tend to know very little about where and when it can be used successfully since of course they already know that it doe snot work. Life can be messy, and change is never easy, even when everything in the new world looks rosy and wonderful.

      I fully expect that some translators will have no interest in working with MT and so those of us who continue on this path of PEMT will have to find others who are willing and interested. We do understand that it is not for everybody.

      The overall growth and interest in translation of business content is expected to grow so I think there will be opportunity for all kinds of translation professionals.

      Thanks for your comments.

      Delete
    3. Kirti, I wouldn't have an issue with machine pseudo-translation translation (MPT) if it actually had a chance of working in the domains that are of interest to me and if the field were not such a playground for frauds and scaremongers. If MT could deal effectively with the contracts, patents and chemical procedures that I translate, I would heave a big sigh of relief and go back to developing software or medical implant materials again or just spend a lot more time fishing and writing poetry. But it's not going to happen in our lifetimes. As Kim Harris pointed out in the GALA panel in Berlin recently, even in the domains where MT may have some relevance, SMEs with turnovers below about 10 million (euros or dollars I don't recall, but I don't think it matters since she was generalizing) really can't afford proper implementation and need "help". You yourself have pointed out often enough that the majority of practitioners are going about it all wrong, applying the tools in the wrong domains or using "gargled" free translations and hiring some poor dopes to clean up the mess. And that I see as the real problem, not somebody tweaking software strings in an automated process or taking the bread off a table somewhere in Mumbai by automating the translation of instructions for use of a toaster that a three year old child could operate intuitively. When I have seen you break down the areas where you feel MT has a proper application and here it does not, I haven't seen a lot to take issue with and probably agree with a good part of it, but then I hear Wiggins shoot his mouth off again with "Get on the MT boat or drown!" or other variations on that theme, or I look at most of the SDL presentation slides on where MT should be applied, and the stench of fraud is overpowering. I don't believe for a moment that you would be willing to put your hand in the fire to swear that what SDL is promoting for MT application is right and proper, or am I mistaken in that?

      Delete
    4. Kevin

      There are many more examples of bad or incompetent use of MT than there are of appropriate, skilled and informed use. In most cases in these sub-par MT situations the translator/editors bear the brunt of the fix up effort of upstream incompetence. In this day of instant Moses systems where "anybody" can build an MT engine if he has a minute to spare, there is an abundance of shoddy MT systems, so to some extent your critical comments are not unfounded. MT is complex and needs real expertise and long-term experience to get results up to professional quality levels and most do not have the patience, skill or interest to do this.

      However, there are also a growing number of expert MT systems (though still a minority) that do in fact provide leverage to both translators and agencies -- I have described some of these systems in my blog posts. I believe that Asia Online is a superior provider because the focus is on getting the best output quality and actual productivity improvement and providing overall control of how an MT system behaves for professional use purposes.

      The MT challenge you pose is indeed a difficult one, but one that can be addressed by working with experts who understand the special issues, e.g. patents will need very special terminology work and the MT engine will need to understand how to handle very long sentences. I have seen German patents that have single sentences with over 400 words. You can see an example of MT output from a JP to EN patent system in my MiTiN slide deck (link in last sentence in blog post). If you do not take special measures to deal with these problems, the odds are very high that you will get garbage output or word salad as it is known in the community.

      As far as SDL is concerned it is my opinion (and only that) that what you get from them is a combination of very mediocre MT systems combined with heavy handed and vague marketing messaging. They depend on captive customers (big enterprises who are trapped by proprietary tools like TMS, Idiom) for much of their customer base. And any rational customer (one not trapped by their proprietary tools) who would compare SDL MT against some other alternatives in the market would likely find that they could get much better quality for much less money elsewhere. In many cases I would not be surprised if one would not do better by simply using Google or Bing Translate. Capitalistic economies have always warned buyers to beware: Caveat Emptor, and marketing-speak and corporate-speak in general are to be regarded with suspicion as social network real-speak has made clear. SDL has nothing but corporate-speak in their messaging (do you know any real people who talk like that?) and to my ears it all rings hollow.

      I also choose to say much of what I do say on my own blog rather than on an Asia Online platform so that I can stay honest without compromising them in any way, but I do HAVE to believe in their technology.

      Once you understand and acknowledge that MT requires special skills, expertise and experience and needs to be tailored to specific translation problems - your approach will be different, more careful, more deliberate. Then the odds of success go up significantly, especially when there is real collaboration between MT technologists, LSPs and translators and you may find that it is even possible to build MT systems that are useful to people who are translating German patents in 2013

      Delete
    5. Even if for the sake of argument we assume that everything you said is correct, you cannot separate the technology abuse from its functionality.
      The MT lobby likes to claim that they focus on technology improvements, but in practice they work on reshaping the perception of quality and "aligning expectations", almost in a social engineering kind of way, as well as spreading FUD and attempting to plant fear; i.e. if we (the TM lobby) have done our preparation and expectation alignment work right, we will provide you a bad quality and service, but you won't see it that way, you will see it as a good quality and even feel good with yourself because you joined the 21st century and received a product of (enter your marketing BS of choice) that saved you nothing in the short-term, but will cost you more in the long-term (not because of the technology, but due to the type of people you are in business with).

      It is enough to watch the following TAUS video: http://www.linkedin.com/redirect?url=http%3A%2F%2Fwww%2Eyoutube%2Ecom%2Fwatch%3Fv%3Db2rg2N-RnoI&urlhash=xFKg&_t=tracking_disc (especially the part between the 2:40 and 6:00 minute marks). It is funny and pathetic to hear that they plan on taking an unethical and unprofessional concept such as CAT discounts and use it as the base for introducing a new abusive concept in the form of PEMT discounts (she even refers to it as new best practice). As if two wrongs make something right again.. A little later she mentions that they need to see how low they can push these discounts (It could be understood as an honest attempt to assess the limits of the technology, but for some reason I have the sneaking feeling that by that she meant testing how low they can push down the poor soles that jump their smoke and mirrors wagon). This is not the only instance in which similar ideas where demonstrated by the technology developers and their brokers disciples who most often lack any understanding or respect to the profession and what it entails, while measuring everything by the number of words and assign to them "fuzzy matches", as if we are using Lego bricks in a prefabricated template kind of design - exposing their true intention: Profit. So, maybe and even probably a true MT research is done in the Academy and even in a commercial setting, and maybe MT can be useful to some extent as a tool, but as long as the developers and researches associate themselves with the greedy and unscrupulous commercial parties who are only interested in MT because they stand to gain money out of abusing it, they cannot expect that the discussion about their technology will focus on merit.

      Delete
    6. Shai

      You and I cannot control how different people will represent the technology. MT is one of those technologies that has a long history of misrepresentation and over promising.

      However, most people can educate themselves so that they can tell good intentions from the bad to some extent at least.

      I think it is always possible to say NO to PEMT jobs when you see that the basic data (pay, MT quality, work required) does not make sense. I think that if translators learn to quickly assess this for each unique MT post-editing job it will be harder for agencies to deceive them. Though perhaps there will always be a few desperate translators who will do this work even though they are paid unfairly. I expect the best translators will learn to discriminate and assess the fair and reasonable opportunities from those that simply do not make sense and refuse to work with those agencies that are not fair. Thus my recommendation is to learn how to tell what kind of MT you are dealing with and also understand what kind of agency you are dealing with.

      While there are some (or even many) who are unscrupulous, I think more often it is just ignorance and a lack of understanding of the human factors and human implications of deploying technology.

      My basic strategy and intent in my blog is to provide education about MT for all the parties involved. It is more difficult to deceive people who are educated and informed and understand the facts at hand. Translators who learn how to make these assessments will not be victims even though they cannot control how different agencies will behave.

      My overall feeling is that good translators are a rare commodity and that is unwise to create long-term ill will with them. The best agencies already understand this and treat translators with respect but there will always be some agencies that you would want to avoid under any circumstance. With MT you would have some different inputs in your decision making on whether to undertake a job or not. My post was an attempt to give translators a way to assess when MT makes sense from their perspective. Though at this point bad examples outnumber the good ones, I think in time this will change and informing translators is one way to do this.

      Delete
    7. Kirti,
      What you are saying about declining work and choosing who to work with is true in principle in a vacuum type of enviroment, but much less so in the current marketplace reality for many. Even some of the more professional translators operating in it are starting to experience difficulties, because the damage and erosion of the profession done overtime by many brokers and some technology developers is not isolated to a specific interaction between an individual and a less-than-stellar broker/client, but far more systematic.

      Good translators are hard to find for several reasons, a major one is the fact that this is just how complicated and skill intensive the profession is. Not anyone can do it, but just about anyone can call themselves a translator, and even more so wedge themselves into the valueless "supply chain". Should the blind obsession with technology abuse continues, their number won't increase but decrease, and with them so do some of the key competencies and expertise.
      To be perfectly honest, the number of potential "good" translators is probably limited at any time point in light of the demand. However, instead of investing in education, true translation supporting technologies and the promotion of the profession to attract more skilled and specialized people, the blinding infatuation with technology and greed result in just the complete opposite. Skilled and specialized translators are leaving the profession, part time or altogether, after being fed up with all the childish, unethical, rude, and abusive conducts, practices, and power plays that are damaging the profession, and only seemed to accelerate in the last decade or so, and they are not replaced by equally skilled and specialized ones. If the MT lobby plan succeeds (and this is really not about the technology and its merit; the master plan is all about social engineering), newcomers will be educated on the knees of PEMT, and the polarity and skill gap that are already present will, probably, increase.

      All of this could be avoided, but it seems that the technology obsessed players are taking the zero-sum game approach (because they feel that they are winning and believe that they can continue to do so with ease), and from power-plays and zero-sum game mentality nothing good can come up in the long term.

      Delete
    8. Shai

      I think it is easy to blame the technology (and we all want somebody or something to blame) but I think the problem is more complex and the changes are broader than you are suggesting. From my viewpoint I see all of the following:

      -- The growth of free online MT
      -- The decreasing importance of corporate marketing messages and the increasing importance of social networking banter
      -- The changing value of traditional translation content (documentation) -- If nobody is reading it then it should be done as cheaply as possible
      -- The increasing value of non-traditional translation work - video, MT tuning, social network sentiment analysis etc...
      -- The growth of collaborative internet hub based work
      -- New volunteer translation models (crowdsourcing, FB, Twitter)
      -- The inability of translation agencies to keep pace with all these simultaneous changes and in most cases technological naivete. If Lionbridge and SDL are considered technology leaders in the translation business than that is a pretty sorry state of affairs.

      I don't think any single force can be blamed exclusively but they all can be together, and the MT vendors are the weakest force of change of all the ones that could be listed.

      I wrote a few times about this e.g. http://kv-emptypages.blogspot.com/2010/04/falling-translation-prices-and.html

      My guess about where this is all heading is that domain specialist, translation tech-savvy translators will be increasingly valued and be paid more than those who who provide 'generic' translation services. Many marginal translation agencies will feel pressure from more efficient competitors and the overall need for translation will grow but not in the SDL (software and documentation localization) area. Also many enterprises will take control of these new translation problems and further undermine those agencies that add very little value beyond project management.

      MT today probably translates 10 Billion words a day (up from 2B in2010) so hopefully we all learn to say more useful things about it than how much Google MT sucks. Many global C-level people think it is fine for a large range of content, and many companies realize that they need to be much more strategic about what they focus their high quality human efforts on.

      Delete
    9. Kirti,
      I didn't blame the technology. I've only stated that the technology is one topic of discussion and its abuse it completely another.

      Again, the MT lobby claims technology improvement, but most of what could be heard and experienced in the field are not attempts to build a dialogue and truly explore the benefits and limitations of the technology for specific purposes, to discuss ethical and honest business cooperation for these cases in which it could be beneficial, instead there is a clear attempt of social engineering to convince both clients and translators that they need - even depend - on MT, a lot of talks about discounts, pushing down rates (for the translators), increasing margins (for the agencies), how to fraud translators into accepting PEMT work (all the above are available on the web as videos and articles from MT advocating bodies), and a lot of vague and arbitrary statistical data and nice-looking graphs to show how great the technology is, but almost never backed up by reliable data (and I will stress again, comparing MT against low quality human translation and claiming it to be a success is not a success story if you ask me).

      Your outlook of the future may become the reality and maybe not. I'm not a prophet so I can't comment one way or the other. What I do know, however, is that not everything is so ethical driven and free of any profit-driven motives.

      Just as an example. One MT trainer that I had a discussion with on a LinkedIn group (in which the majority of the translators crowd are quite influential), boldly claimed that by 2015 not a single word in the world will be translated by humans, and those who will not adapt, and yesterday, will find themselves in a great distress.

      Delete
    10. Shai

      You and I are in much more agreement than may be apparent from an initial glance at our comments.

      I agree that we have reached a point where a much larger dialogue with translators needs to happen and I am hoping that more translators move to place where they themselves define what is acceptable PEMT work and what is not rather than have agencies and MT developers dictate this to them.

      Translators have never learned to talk together as a professional community and so glib agencies and technology developers take over. The translator voice does not need to come from committees and associations -- I am hoping that it actually happens in blogs and the social media where you hear the most authentic voices and are not subjected to arbitrary censorship like in ProZ.

      So far most of the criticism I have heard from translators basically says "MT sucks" and to my mind this is a sure path to victim hood because even bad MT works to some extent. It would be useful for everybody to hear much more specific feedback on what is wrong with how the tech is presented and used, and what can and should be improved. IMO Translators who are educated about what makes sense for them, and what does not are a key factor needed to bring glib talking advocates to their rightful place.

      As far the possibility of MT doing ALL the translation of the world by 2015 -- to put it politely I will say this person is wrong, very wrong and ignorant of what the technology can and cannot do. Avoid working with him. To put it more graphically I will say this kind of statement is equivalent to a verbal shart. (Look up the word in Urban Dictionary if you don't know it, a good video explanation as well for those who are more visually oriented).

      I think it is very much in the interest of translators to understand what makes MT work and determine how to separate the wheat from the chaff. This does mean that you need to go beyond Google and instant DIY and interact with more knowledgeable people, which is still not so easy to do.

      “You never change things by fighting the existing reality.
      To change something, build a new model that makes the existing model obsolete.”
      ― Richard Buckminster Fuller

      Delete
    11. I do not work with that person. That person is a MT trainer working for some MT snake-oil vendor, or so I presume. The only reason that I have interacted with her was call her out on some of her blatant claims about MT, its role, and the future of the human translator.

      Yes, a dialogue is needed and always welcomed, but there are two obstacles:
      1) The majority of current "ambassadors" of MT, at least the one the average translator is exposed do, are not people that you can, or even want, to build a dialogue with. They clearly on a path to destroy the translation profession as much as they for their sole and relatively immediate benefit. Naively enough I tried to engage in conversation with some of them, attempts that failed miserably.

      2) A dialogue is a mutual undertaking. My feeling is that not only translators are hard to reach (and translators are a very professionally and commercially diverse and fragmented bunch to begin with; this is in part why they never really found a common ground as the differences are often worlds apart), the MT-lobby at large is not very responsive or inclined to do so anyway, not in an official capacity at least. In their official capacity they seem to prefer the FUD spreading and zero-sum game approach as means to control the reality and the outcome of their commercial endeavor, while taking advantage of the diversity, ignorance, and detachment, mainly among pseudo-translators, who seem to be their preferred benchmark and target audience.

      Delete
    12. Yes the current situation is quite messy for all the reasons you have pointed out, but I think there are enough reasonable people out there that win-win scenarios can be created for those who find PEMT work acceptable.

      I think as translators become more familiar with MT technology and understand that it can vary greatly, and thus become more informed of when it does.does not make sense to them, we will see fewer negative experiences.

      Today PEMT is often just simply a strategy to reduce rates with no real rationale and with MT output that is quite unusable.

      The basic understanding that is necessary between agency and editor is that the work must be compensated fairly and that there needs to be some transparency on the rate setting process. There also needs to be enough of an understanding that if the facts change that there is some room for negotiation and rates can be adjusted if initial MT quality estimates are not accurate.

      The ALT case study I have written about previously suggests that this is possible.

      Delete
  6. I would also feel unsafe operating a motor vehicle if the road looked like it does here - have a look to the right and to the left of our posts. (Wondering about the telescope above...)

    ReplyDelete
  7. «As long as PEMT can deliver final translations that are close to (or sometimes even better) »

    Hi, Kirti,
    I would like to see statistical evidence of this (not by Common Sense Advisory).
    Needless to say, I am totally aligned with Valerij and Kevin's comments.
    Thank you very much,

    ReplyDelete
    Replies
    1. Aurora,
      The Stanford study cites several sources in http://vis.stanford.edu/files/2013-PostEditing-CHI.pdf. If you look at the writings of Sharon O'Brien and Ana Guberof you will also find additional examples of statistically validated studies. Asia Online has presented several case studies where this has been true for some specific cases -- you can find these by searching on PEMT in my blog.

      There are many cases I am aware of with Asia Online MT systems that provide accurate and compelling consistency on terminology even though MT never gets all the grammar right which is much easier to edit and fix. This is a characteristic of good MT systems.

      Delete
    2. For the sake of accuracy, her name is Ana Guerberof.

      Delete
    3. Sorry Ana - I suspected I had a spelling error and meant to correct it but this blogger comment system does not allow edits once you submit.

      Thanks to Anonymous for correcting us.

      Delete
    4. No worries, I'm used to getting my name misspelled. Thank you, again, for the mention.

      Delete
  8. «The Stanford study cites several sources in http://vis.stanford.edu/files/2013-PostEditing-CHI.pdf.»

    Thank you, Mr. Vashee.
    I'll read it to understand the grounds to say that MpT can sometimes be better than real translation. In fact, even when comparing two real translations an expert in translation studies would find it difficult to define which one is "better." I'll read it. Thank you.

    «If you look at the writings of Sharon O'Brien and Ana Guberof»

    I disagree with most of Ms. Guberof's ideas, but I guess this does not surprise you.

    «Asia Online has presented several case studies where this has been true for some specific cases -- you can find these by searching on PEMT in my blog.»

    If you could give me more specific directions on MpT being better than real translation in your blog, I'd appreciate it.

    ReplyDelete
    Replies
    1. You can find some references to this here:

      http://kv-emptypages.blogspot.com/2011/10/building-momentum-for-post-edited.html In this link you can see that Paypal found that found that “machine-aided human translation” delivers better, more consistent terminology in the first pass and thus they were able to focus more on style and fluency. Deadlines are easier to meet and she also commented that MT can handle tags better than humans.

      In this same post you will also see a presentation by Sajan who found that: The custom engine output delivered higher quality than their first pass human translators especially in Chinese but also for Spanish in a very technical information technology related domain.


      In http://kv-emptypages.blogspot.com/2012/05/omnilingua-profile-of-effective.html the LSP was asked how by their client why the quality had noticeably improved after they saw the final product that was developed using the Asia Online engine. The end customer, a major auto manufacturer, noticed that the overall quality had improved over SDL MT and older human only based production modes.

      http://www.mt-archive.info/10/MTS-2013-Sutter.pdf is an example where it was determined that MT based process produced slightly better output even though very little customization was done.

      Delete
    2. Thank you very much. I'll be back once I finish reading all the information your have pasted here, and what Valerij has pasted below. (Thank you, Valerij).

      Of course there are countless cons to MpT vs. real translation, but I was curious about your comment above.


      Delete
  9. Kirti and Aurora,

    You will find a lot of statistical evidence e.g. here http://www.mt-archive.info/conf/MTS-2013-TOC.htm as well as in many other sources listed in this archive (http://www.mt-archive.info/), to the CONTRARY effect.

    Unfortunately, all the evidence that PEMT CANNOT “deliver final translations that are close to (or sometimes even better)“ will be interpreted as “examples of bad or incompetent USE of MT“ (as Kirti just demonstrated) by those who have an intrinsic interest to promote PEMT and sell MT systems.

    Similarly, if you’d ask for statistical evidence that a bicycle’s performance is close to that of a motor car, you might hear that yes, but it “requires special skills, expertise and experience and needs to be tailored...“.

    Granted, it is not an appropriate example, but the blurred road on both sides of this column is sort of inspiring. Also, no matter how adequate a simile can be, chances are it will be interpreted as a “tirade“.

    ReplyDelete
    Replies
    1. Valerij

      It has become very easy to build low-quality MT systems with Moses, but it is still very difficult to build good professional-work-capable MT systems. The skills needed to do this are especially rare at the agency level.

      So yes, it will be much easier to find examples of unsuccessful PEMT experiences. To then conclude that PEMT does not work in general based on this kind of evidence would be a mistake in my opinion.

      I guess we are going around on this point so we can perhaps agree to disagree. I tend to see those elements that makes MT successful with ease (far fewer in number) and you see the larger number of failed MT attempts - which are usually doomed from the outset because the practitioners lack the skill, knowledge and expertise to make it work.

      A wise friend who ran a large company used to say, that what we see is a function of where we sit at the table.

      Hopefully we can all learn to see the world from each others viewpoints a little better.

      Delete
    2. "To then conclude that PEMT does not work in general based on this kind of evidence would be a mistake in my opinion."

      Kirti, can you refer me (and I believe that others would be interested, too) to an existing MT system that actually works in general? (I deliberately omit the "PE" part because any text can be edited to perfection, just the efforts differ.)

      I believe the evidence of poor MT results is known, but I think it would be useful to know the opposite evidence (in other format than some generic statistical conslusions that say very little about what texts were actually examined or that are based on input from lower-paid "self-described “professional” translators on oDesk").

      Thank you.

      Delete
    3. Tomas

      The private systems in use at Sajan, Omnilingua and ALT are examples of production systems that do work, however, these are private systems and not meant for general public use. They are also tuned to high volume production needs and and are NOT general purpose systems that can be used by any internet passer by.

      Having a general system that works for everybody is the equivalent of a human who could do any domain and any style at any time -- I think rare if ever possible.

      A general purpose system will be lower in quality by definition - MT is not magic and we will probably have to wait till the Star Trek era to find systems that can do anything and everything well. Today systems that are focused on clearly defined domains tend to do produce output that is useful for professional translation work.

      Delete
  10. Thank you, Kirti, for the mention. This is a link to my thesis http://www.tdx.cat/handle/10803/90247 and in the Bibliography section there are plenty of references to very significant work: O'Brien, Plitt, Tatsumi, just to name a few. In all this work, we see an increase in productivity and quality with MT but this is achieved always under certain circumstances related to, of course, the subject matter, but more importantly the quality of the output for that particular language combination in that particular domain, among others. Also, it is consistent that translators behave in very different ways when translating in general with MT or with TMs or on their own. We cannot make quick statements that MT always helps productivity or quality, but in certain circumstances it can do. This could be obvious but I think the more we help to understand how MT works and its impact in actual work, with empirical data rather than opinions, the better it will be for the translators. But I guess opinions can gather more followers :)

    ReplyDelete
  11. hello there

    I already asked this to you Kirti, by email some time ago ...

    I told you about the crappy translation into Panasonic digital camera manuals, unreadable even in the English version, and I asked (humorously) if Asia Online could have been the culprit ...
    J

    then I asked you some sample in my language pair (English to Italian) but unluckily you wasn't able to send me anything

    now, I think that all readers here could have a better insight in the matter, if they were able to check some samples in their language pair ..

    ReplyDelete
  12. Hi Claudio

    I did respond to your email -- perhaps you did not see the response. Asia Online has had no involvement with Panasonic manuals. I sent you some samples from Italian to English but I have no samples for the opposite direction.

    Thanks

    Kirti

    ReplyDelete