Pages

Friday, July 25, 2014

Understanding The Drivers of Success with the Business Use of Machine Translation

We have reached a phase where there is a relatively high level of acceptance of the idea that machine translation can deliver value in professional translation settings. But as we all know the idea and the reality can often be far apart. It would be more accurate to say this acceptance of the idea that MT can be valuable, is limited to a select few among large enterprise users and LSPs (the TAUS community) and has yet to reach the broad translator community who continue to point out fundamental deficiencies in the technology or share negative experiences with MT.   So while we see growth in the number of attempts to use  MT, as it has gotten mechanically easier to do, there is also more evidence that many MT initiatives fail in achieving sustainable efficiencies in terms of real translation production value.

In a typical TEP (Translate-Edit-Proof) business translation scenario, A "good" MT system will provide three things to be considered successful:

1) Faster completion of all future translation projects in the same domain
2) Lower cost/word than doing it without the MT system
3) Better consistency on terminology especially for higher volume projects where many translators need to be involved

All of this should happen with a final translation delivered to the customer that is indistinguishable in terms of quality from a traditional approach where MT is not used at all.


It is useful to take a look at what factors underlie success and failure in the business use setting, and thus I present my (somewhat biased) opinions on this as a long-time observer of this technology (largely from a vendor perspective). I think that to a great extent we can already conclude that MT is very useful to the casual internet user, and we see that millions use it on a regular basis to get the gist of multilingual content they run into while traveling across websites and social platforms. (e.g. I use it regularly in Facebook.)

What are the primary causes of failure with MT deployments in business translation settings?

Incompetence with the technology: The most common reason I see for failed deployments is the lack of understanding that the key users have about how the technology works. Do-it-yourself (DIY) tools that promise that all you need to do is upload some data and press play are plentiful, and often promise instant success. But the upload and pray approach does not often work to provide any real satisfaction and business advantage. Unfortunately the state of the technology is such that some expertise and some knowledge are required. The translators and post-editors who have to work with the output results of these lazy Moses efforts, are expected to clean-up and somehow fix this incompetence usually at lower wage rates. And thus resentment grows and many are speaking up frequently in blogs and professional forums about bad MT experiences. Those that have positive MT experiences rarely speak up in these forums since the work is not so different from regular TM-based translation work and MT is often regarded as just another background tool that helps to get a project done faster and more consistently. MT output that does not provide cost and turnaround advantages for translation work cannot be considered to be useful for any professional use. Thus, a minimum requirement for using MT in professional settings is that it should enhance the production process.

image

Lowering cost is the ONLY motivation: The most naïve agencies simply assume that using MT, however incompetently, is a way to reduce the cost of getting a translation project done, or more accurately a way to justify paying translators less. Thus the post-editors are often in a situation where they have to clean up low quality MT output for very low wages. Given that we live in a world where the customers who pay for professional translation are asking for more efficient translation production i.e. faster and cheaper, agencies are being forced to explore how to do this, but this exploration needs to happen from a larger vision of the business.  As Brian Solis points out, using technology without collaboration and vision is unlikely to succeed (emphasis mine).
"That's the irony about digital transformation, it doesn't work when in of itself technology is the solution. Technology has to be an enabler and that enabler needs to be aligned with a bigger mission. We already found that companies that lead digital transformation from a more human center actually bring people together in the organization faster and with greater results," Solis says. “When technology is heralded above all else, there becomes an even greater disconnect between employees (translators)  and the challenges that their business is trying to solve.”
What many LSPs fail to understand is that their customers are asking for ongoing efficiencies, and new production models to handle the new kinds of translation challenges they face in their businesses. They are not just asking for a lower rate for a single project. Agencies focused on the bigger picture are asking questions like how MT can enable them to achieve new things and what's different about their customers needs today versus yesterday. With the right MT strategy in place, technology becomes an enabler, not the answer and enables agencies to build strong long-term relationships with customers who could not get the same price/performance with another agency that does not understand how to leverage technology for these new translation challenges. Agencies must evolve and reimagine their internal process, structure and culture to match this evolution in customer behavior among their own employees and translators.

No engagement with key stakeholders: Many if not all the bad MT experiences I hear about have one thing in common. Very poor communication between the MT engine developers (LSP), the customer and the translators and editors. MT is as much about new collaboration models as it is about effective engine development, and collaboration cannot happen without open and transparent communication, especially during the initial learning phase when there is a great deal of uncertainty for all concerned. If this communication process is in place in the early projects, it enables everybody to rise together in efficiency, and gets easier and more streamlined and more accurately predictable with each successive MT project. The communication issue is quite fundamental and I have tried to address and explore this in a previous post


What are the key drivers of successful deployments of MT?

image

Expert MT Engine Development: The building of MT engines has gotten progressively easier in terms of raw mechanics, but the development of MT engines that provide long-term competitive advantage remains a matter of deep expertise and experience. If as an LSP, you instantly create an MT engine that any of your competitors could duplicate with little trouble, you have achieved very little. Developing MT systems that provide long-term production advantage and a real competitive advantage is difficult, and requires real expertise and experience. The odds of a developer who has built thousands of engines producing a competitive engine are much higher than someone who uploads some data and hopes for the best. Skillful MT engine development is an iterative process where problems are identified and resolved in very structured development cycles so that the engine can improve continuously with small amounts of corrective feedback. Knowing which levers to pull and adjust to solve different kinds of problems is critical to developing competition beating systems. Really good systems that are refined over time are very difficult to match and will continue to provide price/performance advantages over the long-term that competitors will find difficult to match.

image

Engaged Project Managers and Key Translators: The most valuable feedback to enhance MT system output will come from engaged PMs and translators who see broad error patterns and and can help develop corrective strategies for these errors. Executives should always strive to ensure that these key people are empowered and encourage them to provide feedback in the engine development process. For most PMs today, MT is new and an unknown and unpredictable element in the translation production process. Thus in initial projects, executives should allow PMs great leeway to develop critical skills necessary to understand and steer both the translators and the MT engine developers. These new skills are very key to success and can help build formidable barriers to competition. While very large amounts of high quality data can sometimes produce excellent MT systems, a scenario where you have a a good project manager steering the MT developers and coordinating with translators to ensure that key elements of an upcoming project are well understood, will almost always result in favorable results other things being equal, especially with challenging situations like very sparse data or when dealing with tough language combinations. 

Communication and collaboration are key to both short and long-term success. The worst MT experiences often tend to be with those LSPs (often the largest ones ) where communication is stilted, disjointed and focused on CYA scenarios rather than getting the job done right. Successful outcomes are highly likely when you combine informed executive sponsorship, expert MT engine development and have empowered PMs who communicate openly and frequently with key translators to ensure that the job characteristics are well understood and that outcomes have a high win-win potentiality. Even really good MT output can fail when the human factors are not in sync. Remember that some translators really don’t want to do this kind of work and forcing them to do it is in nobody’s interest.

Fair & Reasonable Compensation for Post-Editors: I have noted that a blog post I wrote on this issue almost 30 months ago still continues to be amongst the most popular posts I have written. This is an important issue that needs to be properly addressed with a basic guiding principal, pay should be related to the specific difficulty of the work and quality of the output. So low quality output should pay higher per word rates than very high quality output. This means that you have to properly understand how good or bad the output is in as specific and accurate terms as possible since people’s livelihoods are at stake. This accuracy can be gauged in terms of average expected throughput i.e. words per hour or words per day. You may have to experiment at first and be prepared to overpay rather than underpay. Make sure that translators are involved in the rate setting process and that the rate setting process is clearly communicated so that it is trusted rather than resisted. Translators should also ask for samples to determine when a job is worthwhile or not. The worst scenario is where an arbitrary low rate is set without regard for the output quality, and typically in these scenarios incompetent MT practitioners always tend to go too low on the rates, resulting in discontent all around. 

image

Real Collaboration & Trust Between Stakeholders: This may be the most critical requirement of all as I have seen really excellent MT systems fail when this was missing. Translation is a business that requires lots of interaction between humans with different goals and if these goals are really out of sync with each other it is not possible to achieve success from multiple perspectives. Thus we often see translators feel they are being exploited or agencies feeling they are being squeezed to offer lower rates because an enterprise customer has whipped together some second rate MT system together with lots of noisy data for them to “post-edit”. When the technology is used (actually misused) in this way it can only result in a state of in equilibrium that will try to correct itself or make a lot of noise trying to find balance. This I think is the reason why so many translators protest MT and post-editing work. There are simply too many cases of bad MT systems combined with low rates and thus I have tried to point out how a translator can make an assessment of a post-editing job that is worth doing from an economic perspective at least. 

Perhaps what we are witnessing at this stage of the technology adoption cycle is akin to growing pains, like the clumsy first steps of a baby or the shyster attempts of some agencies to exploit translators as some translators have characterized it. Both cases are true I feel. And so I repeat what I said before about building trusted networks as this seems to be an essential element for success.

The most successful translators and LSPs all seem to be able to build “high trust professional networks”, and I suspect that this will be the way forward i.e. collaboration between Enterprises, MT developers, LSPs and translators who trust each other. Actually quite simple but not so common in the professional translation industry.

There seems no way to discuss the use of MT in professional settings without raising the ire of at least a few translators as you can see from some of the comments below. So I thought it might be worth trying to lighten the general mood of these discussions with music. I chose this song carefully as some might even say the lyrics are quite possibly the result of machine translation or not so different from what MT produces. As far as I know it is just one example of the poetic mind of Bob Dylan. If you can explain the lyrics shown below you are a better interpreter and translator than I am. Musically this is what I would call a great performance and a good vibe. So here you have a rendition of Dylan's My Back Pages on the Empty Pages blog.




Crimson flames tied through my ears
Rollin’ high and mighty traps
Pounced with fire on flaming roads
Using ideas as my maps
“We’ll meet on edges, soon,” said I
Proud ’neath heated brow
Ah, but I was so much older then
I’m younger than that now

Half-wracked prejudice leaped forth
“Rip down all hate,” I screamed
Lies that life is black and white
Spoke from my skull. I dreamed
Romantic facts of musketeers
Foundationed deep, somehow
Ah, but I was so much older then
I’m younger than that now

Girls’ faces formed the forward path
From phony jealousy
To memorizing politics
Of ancient history
Flung down by corpse evangelists
Unthought of, though, somehow
Ah, but I was so much older then
I’m younger than that now

A self-ordained professor’s tongue
Too serious to fool
Spouted out that liberty
Is just equality in school
“Equality,” I spoke the word
As if a wedding vow
Ah, but I was so much older then
I’m younger than that now

In a soldier’s stance, I aimed my hand
At the mongrel dogs who teach
Fearing not that I’d become my enemy
In the instant that I preach
My pathway led by confusion boats
Mutiny from stern to bow
Ah, but I was so much older then
I’m younger than that now

Yes, my guard stood hard when abstract threats
Too noble to neglect
Deceived me into thinking
I had something to protect
Good and bad, I define these terms
Quite clear, no doubt, somehow
Ah, but I was so much older then
I’m younger than that now
 

14 comments:

  1. I don't know where the acceptance come from, probably from some lay people,
    because most professional translators think that it can only be used for entertainment purposes, and to have a vague idea
    what something means in ordinary life.

    ReplyDelete
    Replies
    1. Lilian,

      The experience of MT systems will vary by language and there has been much more success in languages like Spanish and Portuguese. Russian is much more difficult.

      Also, there is a great deal of variation amongst MT systems and care should be taken to not lump it all together. I have discussed this in more detail here: http://kv-emptypages.blogspot.com/2014/05/monolithic-mt-or-50-shades-of-grey.html

      Delete
  2. First of all, you shouldn't be talking of "wages" when speaking of freelance translators and editors. We are freelancers and not employees.

    Secondly, the acceptance is low because most systems are raw and produce rubbish. Most of them are still useless. But these are the cheaper systems sold to LSPs and big multinationals and these are the systems that freelancers and editors have to contend with. I still haven't seen a system that is a lot better than Google Translate. Post-editing this output offers no gain in productivity, Hence, no savings for the clients. I'm pretty sure the most advanced systems are so expensive (well, it takes years to build a decent one and expertise is rare...) to be uneconomical. You need masses and masses of documents to be translated to make them somewhat viable.

    Finally, it's pretty normal that a freelance translator would refuse to improve a system by training it with his/her knowledge so it can do his/her job better, making them redundant on the long term. Unfortunately for the MT vendors, they are not selling robots to build cars, but a system which is supposed to replace our brain... an impossible task. It's also an ethical stance. As long as MT will require our input, it will never take off. We are not stupid...

    ReplyDelete
    Replies
    1. "Wages" in the sense that I have used it here is a generic term for pay rate and I did not mean to imply any employer / employee relationship.

      Systems that are tuned for a specific purpose that produce much better output than Google do exist but do not have high visibility and unfortunately what you say about many LSP/enterprise MT systems producing very poor output is true. However, getting the system to produce much better quality is not necessarily a very expensive undertaking. It is much more a matter of skill and expertise than a huge investment.

      I agree that PEMT is not for everybody, and if you feel that it would make you redundant in the long term I can understand your reluctance to engage. But good MT systems can also create new kinds of projects and more work as I pointed out in my post last month. MT can only get better with informed linguistic input and I hope that some translators and linguists will see that it can create opportunity as well and not only take work away and thus help the technology evolve.

      Delete

  3. The problem is not in the MT technology but in its use

    Modern MT technologies may be professionally usable, but trying to use them as a black box - put language A in, get language B out, then fix the errors - is a dead end: for most texts, such translations aren't even anywhere close to the bottom edge of professional quality for most language pairs (with a possible exception of closely related languages), and the progress in this domain over the recent years is negligible. One exception is "preprogrammed" texts, e.g. responses to standard questionnaires, which are fairly constrained in their content; such texts, however, account for only a small percentage of the translation market.

    One professional situation where I see a potential use for MT is a translation of very large volumes of very uniform text by a trainable MT system that is regularly trained by very experienced translators and terminologists. The cost of such training per page of text will be substantially higher than translating the same text manually, and cost savings can only be realised after a great deal of training. I don't have any mathematically justified estimates, but my professional gut feeling is telling me the break-even point is somewhere above several million words of stylistically and lexically uniform text. Thus, for example, the entire body of documentation for Microsoft Windows is probably worth it. However, you really need top-notch translators to train the system, and finding those who will agree to do it is already a problem in itself.

    Furthermore, I wouldn't risk using MT in such safety-critical industries as aerospace, nuclear power, oil and gas, or pharmaceuticals, and these industries are exactly where most very large translation projects are concentrated. On the other hand, the quality of human translation in e.g. software localisation industry is at times appalling, even at the largest and most successful software companies, and so MT will not look so bad in comparison; nevertheless, such a "good enough" approach is already unprofessional.

    ReplyDelete
    Replies
    1. Yes, I absolutely agree the whole concept of MT is based on the wrong assumptions, in my opinion--that you use a as the input and get b--a perfect, accurate translation at the other end. It just does not happen like that in real translation because a word represents only an idea (or a certain meaning) when it is decoded as that meaning, which may depend on many factors, such as the context: culture, things not verbalized--just implied, and various other nuances

      Delete
    2. Anton, yes MT has to be used with care. It does not work everywhere but we have seen it work even with Life Sciences material when done right. You can watch the Hunnect webinar video at the Asia Online site if you are interested in the details.

      When properly done MT can provide break-even at a few hundred thousand words as many Asia Online customers have seen. But each system is unique and generalizations are usually not very useful.

      For the industries that you mention we too would recommend a much more rigorous proofing process.

      Delete
    3. Anton,
      If MT is under "proper control", there is no "risk for aerospace, nuclear power, oil and gas, or pharmaceuticals....". Do not blame a piano (MT), but those who do not properly tune it (customize all MT related resources. e.g. dictionaries, DNT lists, bilingual corpuses...) and play it (monitor and improve a raw MT output).

      Delete
  4. José Henrique LamensdorfJuly 26, 2014 at 2:59 PM

    An analogy

    Yesterday a colleague (?) was advocating for cost-free crowdsourced PEMT, to put an end to the translation business.

    I told her she had won a round air trip on the Elizabeth Arden circuit: Sao Paulo > Rome > Paris > London > New York > Sao Paulo (where she lives, based on her telephone area code)... for only USD 100!

    The only catch is that there is no human pilot. The jet liner is 100% computer driven. That computer has a database compiled from the "black boxes" from all planes flown in the past 10 years or so, sync-ed to all worldwide meteorology data collected in this period.

    During the flight, the computer will accept and execute all commands entered by all passengers via the computer terminals provided in front of each seat, in the very sequence they are entered.

    Now boarding at Gate 13.

    Will she pay the $100 and hop into that plane???


    IMO MT or PEMT is a matter for careful judgment on an individual case basis. It is neither a panacea for all cases, nor to be discarded altogether. Either "extreme" stance embodies a 50% chance of being wrong.

    ReplyDelete
  5. How about concentrating on Project Management Automation?

    Machine (pseudo) translation still needs to be reviewed by a human translation expert. All Machine (pseudo) translation is doing is confusing the end client, minimizing the earnings of the real translators and fattening the pockets of Project Managers and CEOs of MpT companies and large TBA (Translation Brokering Agencies).

    It seems to me, it would be easier to concentrate on creating software to automate Project Management and link end clients with the professionals who have to provide the final copy of text!

    I am sorry, but I do not see it as the drivers of success, as much as the drivers of profiteering at the expense of the expertise of those who love languages and communication.

    I would not encourage any young person to consider translation any more. It takes less time to become an engineer than to become a good translator, please remember that!

    ReplyDelete
  6. It will never work!

    This `telephone' has too many shortcomings to be seriously considered as a practical form of communication. The device is inherently of no value to us.
    - Western Union internal memo, 1878

    ReplyDelete
  7. Define success:

    Success at what? I'm being serious. As I read the article, that is the first thing that came to mind.

    ReplyDelete
    Replies
    1. In a typical TEP (Translate-Edit-Proof) scenario, A "good" MT system will provide three things to be considered successful:

      1) Faster completion of all future translation projects in the same domain
      2) Lower cost/word than doing it without the MT system
      3) Better Consistency on Terminology especially for higher volume projects where many translators need to be involved

      All of this should happen with a final translation delivered to the customer that is indistinguishable in terms of quality from a traditional approach where MT is not used at all.

      Delete
    2. I have added my response here to the blog post so that others don't have this confusion and issue.

      Delete