Why Machine Translation Matters for LSPs

In my last entry I pointed out some of the reasons why the information deluge on the web and the worldwide thirst for knowledge is driving the development of MT technology and only touched upon why this might matter to Language Service Providers (LSP) and the professional translation industry. I thought it might be useful to expand on this.

It is useful to summarize some of the key trends in the professional industry as they provide a useful backdrop for why the need for more automation will only continue to build momentum. I am told that SDL originally stood for Software and Documentation Localization. The world has changed and the increasing source data volume, falling prices and the increasing value of dynamic community-created content suggests that changes are needed.The list below explains this further.

-- Corporations are facing an increasing demand for translated content without significantly higher budgets. It is possible that the increase in source content volume could be anywhere from 5X to 100X historical volumes. The internet has been instrumental in increasing the demand for large amounts of multilingual content.

-- The sustainable price per word rate has steadily declined over the last 10 years and buyers are still demanding greater productivity.

-- Much of the industry is currently optimized around the TEP process and project management skills.

-- Translation Memory technology is now widely used by agencies and translators those who work with moderate to highly repetitive source content and there is growing momentum to centralize this.

-- Translation Management Systems (TMS) are increasingly being used to manage the growing workloads and improve project management efficiency.

-- The importance of communities and collaborative models are gaining momentum by the day. Crowdsourcing is seen as a legitimate and efficient way to do certain kinds of projects.

-- The value of speed and agility is increasing all the time as we enter the real time internet world. = We need it translated NOW!

-- The value of static content that has been the focus of professional translation continues to decline and many global corporations see that dynamic knowledge content and community content has much higher value to the global customer e.g. iPhone ships worldwide without a manual.

-- Long standing industry experts have pointed out that current approaches and processes are outmoded and that translation automation technology needs to evolve to keep up with the rapidly growing information needs of global customers.

The Consortium for Service Innovation whose members include Dell, Symantec, Cisco, Microsoft and others, provides some very compelling data on how the customer support reality is changing for a typical IT company that produces a software or hardware product for the global market.The graphic above shows that customers in the IT industry get 40X the support from self service knowledge bases and communities than they do from the documentation and the direct customer support organization.

The picture above shows how major corporations focus on the relatively low value, static content even though there is increasing evidence that customer loyalty is built elsewhere. This is slowly changing and it can already be seen by the increasing use of MT by market leaders like Microsoft, Intel and Cisco to widely distribute support information even though it is sometimes raw MT. The ROI in creating more self-service and community content is huge as can be seen from these diagrams.

As these forces gather it becomes increasingly important for an LSP to be able to help their customers respond to the much more dynamic world. How many LSPs have been involved with large knowledge base translation projects? Why not? Clearly automation is necessary and a different more collaborative process is required. Global enterprises are being driven to understand and use community content, and constantly changing support requirements to engage global customers. LSPs that have the skills and knowledge to help with this will be seen as valued partners.

Many say that SMT is in some sense a much more flexible and robust TM technology and will now be much more easily used and deployed with tools like Language Studio Pro. These domain focused MT systems improve over time and can thus become a strategic asset that enable an LSP to respond faster and at lower cost than otherwise possible. LSPs with good quality MT systems become valued partners for every global company interested in that domain that enable a much broader dialogue with global customers. However, it requires an investment to get to this point.

MT will also open up completely new markets for the professional translation industry as major Publishers begin to realize that they can make high value content usefully multilingual at much lower costs, and in a much faster timeframe. At every step, the leaders will be those companies that learn to use these tools well and also engage translators in this process as they are necessary and critical to producing high quality. There will be a need for new kinds of people, some with deep linguistic skills who identify translation error patterns and develop corrective strategies, and and there will also be a role for less skilled monolingual post-editors.

MT can also enable improved productivity and standardization on large localization projects. MT engines can leverage all future projects in the same domain or for the same customer. In some cases the availability of imperfect MT output can also trigger and create human translation demand e.g. patents. Most of all I think that LSPs that learn to use MT effectively become true globalization partners who are engaged in many international initiatives wherever new languages and new multilingual content can drive new market growth.

I predict that 2010 will be a year that we see some LSPs develop definite competitive advantage using their knowledge and expertise with MT as a key differentiator.

 "Inquiry and change are not separate; they are simultaneous. The moment we ask a question, we begin to create change.” – David Cooperrider, Appreciative Inquiry

There is an excellent audio-visual presentation  (requires registration) that describes the technology and provides some specific use scenarios for LSPs interested in exploring this further.

Why Machine Translation Matters

"I have not failed. I've just found 10,000 ways that won't work." - Thomas Edison

After more than fifty years of eMpTy promises and repeated failures, amazingly, interest in machine translation continues to grow. It is still something that almost everybody hopes will work someday. We just won’t give up. Why? How can an industry that fails to deliver for 50 years still be around? Clearly, MT is a difficult problem, but I think the main reason that we persist is that there is a huge thirst for information, data and knowledge that exists across language barriers. The growing volume of valuable information on the internet only makes this thirst more urgent.

Is automated translation finally ready to deliver on its promise? What are the issues with this technology and what will it take to make it work? I would like to provide my perspective on why it matters and why it is important that we continue in our quest to make it work better.

In the professional translation world there is much skepticism about MT and we see MT regularly being trashed in Translator and LSP blogs, forums and conversations at conferences. Many dismiss it entirely, as a foolish and pointless quest, based on what they see on Google and other free online translation portals. Very few understand or have ever seen the potential that carefully tuned and customized MT systems suggest.  There are a few who have begun to understand that MT is an imperative that will not go away and step tentatively forward. I am happy to see some wholeheartedly embrace it and try and learn how to use it skillfully to develop long term competitive advantage.

For some professionals there is a debate about whether Rule-based MT (RbMT) or Statistical Machine Translation (SMT) is better and of late it has become very fashionable to claim that the "right" approach is hybrid. Industry giants (Google, Microsoft, IBM) are all very focused on SMT with increasingly greater linguistic variations, and there is a healthy open source movement also underlying this (SMT) technology that is spawning innovative, new companies. My company, Asia Online, I think is one of the bright lights on the horizon.

My personal interest in MT is driven by a conviction that it can truly be an instrument to bring positive change in the world. It is possible that, using MT to ease access to critical knowledge could revolutionize and rapidly accelerate the development of much of the world’s poorest communities. I don’t think it is an exaggeration to say that “good” MT could help to improve the lives of millions in the coming years. And thus I feel that improving the quality of MT is a problem worthy of the attention of the best minds on the planet. I also think that getting the professional industry engaged with the technology is key to rapidly driving the quality of MT systems higher and perhaps to reach a tipping point where it enables all kinds of valuable information to rapidly become multilingual. My sense is that MT needs to earn the respect of professionals to really build a quality momentum and make the breakthroughs that so many us yearn for.

The Increasing Velocity of Information Creation

We live in a world where knowledge is power and information access, many say has become a human right. In 2006, the amount of digital information created, captured, and replicated was 1,288 x 1018 bits. In computer parlance, that's 161 exabytes or 161 billion gigabytes …

This is about 3 million times the information in all the books ever written!

Between 2006 and 2010, the information added annually to the digital universe will increase more than six fold from 161 exabytes to 988 exabytes. In 2007 it was already 281 exabytes. It is likely that the bulk of this new information will originate in just a few key languages of the digitally privileged knowledge driven economies. So are we heading into a global digital divide in the not so distant future?  The famous Berkeley study on How Much Information testifies to this huge momentum. A recent update to the study suggests that US households consumed approximately 3.6 zettabytes of information in 2008. Access to information is closely linked to prosperity and economic well being as shown below.

  
Peter Brantley at Berkeley in a personal blog quotes Zuckerman's wonderful essay:

“For the Internet to fulfill it’s most ambitious promises, we need to recognize translation as one of the core challenges to an open, shared and collectively governed internet. Many of us share a vision of the Internet as a place where the good ideas of any person, in any country, can influence thought and opinion around the world. This vision can only be realized if we accept the challenge of a polyglot internet and build tools and systems to bridge and translate between the hundreds of languages represented online."

Brantley goes on to say:

"Mass machine translation is not a translation of a work, per se, but it is rather, a liberation of the constraints of language in the discovery of knowledge."

Today, the world faces a new kind poverty. While, we in the West face a glut of information, much of the world faces information poverty. The cost for this can be high. “80% of the premature deaths in the developing world are due to lack of information” according to the University of Limerick President Prof. Don Barry. Much of the world’s knowledge is created and remains in a handful of languages, inaccessible to most who don’t speak these languages. Asia Online conducted a survey of local content available in SE Asian languages, and found that China and Japan each had 120X more content, and English speakers have perhaps 600X more content available to them than the billion people in the SEA region. Access to knowledge is one the keys to economic prosperity. Automated translation is one of those technologies that offers a way to reduce the digital divide and raise living standards across the world. As imperfect as it is, this technology may even be the key to real people-to-people contact across the globe.

The seminal essay The Polyglot Internet by Ethan Zuckerman has got to be the most eloquent justification for why translation technology and collaborative processes must and will improve.  It has become the inspiration and manifesto for the Open Translation Tools Summit.

While there is profound need to continue improving machine translation, we also need to focus on enabling and empowering human translators.Professional translation continues to be the gold standard for the translation of critical documents. But these methods are too expensive to be used by web surfers simply interested in understanding what peers in China or Colombia are discussing and participating in these discussions.
The polyglot internet demands that we explore the possibility and power of distributed human translation.
We are at the very early stages of the emergence of a new model for translation of online content – “peer production” models of translation.

Visionaries like Vint Cerf also points this out in a recent interview. Ray Kurzweil has spoken on the transformational potential that this technology could have on the world. Bill Gates has commented many times on the potential of MT to help unlock  knowledge, both for emerging countries and those who do not speak English. The Asia Online project is focused on breaking the language barriers for knowledge content using a combination of automated translation and crowdsourcing. Much of the English Wikipedia is intended to be translated into several Asian languages that are content starved using hybrid SMT and crowdsourcing.  Meedan is yet another example of how SMT and a community can work together to translate interesting content quickly at high quality levels to share information. There are many more.

While stories of MT mishaps and mistranslations abound, (we all know how easy it is to make MT look bad), it is becoming increasingly apparent to many, that it is important to learn how to use and extend the capabilities of this technology successfully. While MT is unlikely to replace human beings in any application where quality is really important, there are a growing number of cases that show that MT is suitable for:

· Highly repetitive content where productivity gains with MT can dramatically exceed what is possible with just using TM alone

· Content that would just not get translated otherwise

· Content that cannot afford human translation

· High value content that is changing every hour and every day

· Knowledge content that facilitates and enhances the global spread of critical knowledge

· Content that is created to enhance and accelerate communication with global customers who prefer a self-service model

· Content that does not need to be perfect but just approximately understandable

The forces that drive the interest in this technology continue to build momentum. Disruption is coming and much of the momentum is from outside the professional industry. I believe there is an opportunity for the professional translation industry to lead, and to develop and demonstrate best practice models that others will follow and emulate. Some may even learn to build competitive advantage from their use and superior understanding of how to leverage MT in professional projects.

I invite those interested in a productive professional dialogue to join the Automated Language Translation group in LinkedIn to come and explore how to learn to use this technology to professional advantage. I think we will continue to see more companies learn to use MT technology and I look forward to changing the archaic translation model that rules today for large and massive scale translation projects.

So here you have finally a real MT focused blog entry.

Rising Asia and its Implications

Somebody whose opinion I really value, just told me that my blog is about "men fighting" after seeing my first three posts. While I really do care about responsible free speech and will always aggressively defend it; enough about censorship and moderator abuses and let's move on.

I find the notion of "A Rising Asia" interesting,  and I think it presents a major opportunity for the localization and translation industry in the years to come. I was first introduced to this concept many years ago by Nick Kristof and Sheryl WuDunn in their excellent book, Thunder from the East.  (They were also among the first to point out the promise of China/Rising Asia, several years before the "experts" did.)  I really liked the book because it also quickly provided an economic history of the world and is a very easy read.

I thought it would be good to update and expand upon an article I wrote for GALA a little while ago. I have been surprised how little real awareness there is even in the localization industry where leaders often equate Asia with China & Japan (CJK).  This was really brought home at the #LTBKK conference, when I saw what a revelation Biraj Rath's excellent presentation on the Indian localization market opportunity was, to many industry experts. LISA, to their credit announced an India Forum shortly after the conference.

So I thought it might be useful to provide a basic primer on the broader Asian market opportunity. For some in the L10N industry this might all be obvious but here goes anyway. I am interested because:

- There are a lot of people living in Asia (maybe 95% of the next billion Internet users)

- The Internet has very low penetration thus far. Multilingual content will very likely play a key role in driving increasing  penetration and commercial opportunity.

- They will need a lot of information quickly (huge opportunity for automated translation technology)

- Largest concentration of young people in the world (also the Middle East and Brazil) 

 

Asia is extremely diverse, economically and culturally, but yet there are some strong common elements. It is also much less connected than Europe. Today, we are aware of the current economic momentum that India and China have, historically they both also had a deep and lasting cultural influence on much of Asia. An awareness of this history is very useful in developing effective business strategies for different countries. The internet is only just beginning to take root in much of Asia (18% vs. 73% for North America), however, it is expected that almost half of all Internet users will be Asian by 2013. Already, China has more people online than the US. Asia could be a major opportunity for companies that learn to tap into this new emerging online population. But this will require an understanding of the diversity and characteristics of the various segments and will also need new approaches in communication and marketing. Asian economies continue to rise in importance and growth, as both a supplier and consumer. Today China and India are the largest mobile phone markets in the world.

Some interesting and perhaps less known facts about Asia that provide a useful contrast to Europe are shown below. They also give one a sense for the different type of opportunities available and the differing reality of Asia.

-GDP per Capita in Asia (~$15,000) is less than half of the EU average and there is a much wider standard distribution and a large population living in poverty throughout the continent.
-While India and China are among the fastest growing economies in the world, the GDP per Capita is $2,800 for India and $6,000 for China and they should still be considered developing economies.
-The top GDP/Capita countries (2008) in Asia are: Singapore ($52K), HK, Japan, Taiwan, South Korea ($23K), Malaysia, Kazakhstan and Thailand ($8.5K).
-India has 22 official languages that are as distinct and different as the 23 EU languages, and also include at least 6 different scripts. English is only spoken by about 7% of the people in India. However, it is possible to get deep penetration into the Indian market with 5 key languages.
-There is very little local language content for Asian languages on the web in general. Based on a survey done by Asia Online in 2007, less than 15% of the total content on the web is in Asian languages. Almost 90% of the Asian language content is in Chinese and Japanese. There is huge need for more local language content all over SE Asia.
-Mandarin is beginning to edge out English as the preferred 2nd language in Asia
-China is now the fastest growing patent office in the world. The WIPO and others state that China is clearly an emerging scientific and technological power.
-The share of Asian country based patent filings is now in excess of 50% of all patents filed across the world.
-India has more gifted and talented students in high school than the total school student population in the US.
-China has more students in Science and Technology college degree programs than India and the US combined.
-McKinsey has identified a “Rising Asia” as a stable long term trend that will fundamentally change consumption patterns. Gartner suggests using IT to reach the market. They suggest that global companies use IT to ‘lighten’ their Asian business model to address the specific cultural, geographic reach, and supply chain considerations.
-The wealthy Asians are concentrated in major cities like Shanghai, Beijing, Hong Kong, Singapore, Kuala Lumpur, Mumbai, Delhi, Seoul, Manila and Bangkok.
-China is now the fastest growing market for Bentley and BMW.
-More cars are now sold in China than in America.
-Even countries like Laos, Nepal, Pakistan, Sri Lanka, Myanmar, and Cambodia which have very low GDP/Capita are interesting markets for cell phones and basic commodities.
-An understanding of Buddhism, Hinduism and Confucianism cultural perspectives can dramatically enhance your communications strategy into most parts of Asia.
-The fastest growing FaceBook markets in 2H2009 are Taiwan, Indonesia, Philippines and Thailand.
-Google is not dominant in key Asian markets, in Korea they have less than 2% search market share and they are a distant second in China and Japan. Maybe even completely out of China soon. Local companies dominate because of better understanding of local content, language and customer preferences. This suggests that standard US approaches may not work as well in many Asian markets.
-Chinese social networking startups have produced many innovations that have led to them becoming profitable much faster than US equivalents like MySpace and Facebook.  We are now seeing Asian innovation gradually making its way to the west.
-Most of Asia has been relatively unscathed by the global financial and real estate market collapse.
-India is increasingly considered a "soft power". Influential culturally way beyond it's direct sphere of influence.

-The venture capital markets in India and China are rapidly developing with help from "returning" entrepreneurs and hostile US immigration policies.

But simple strategies like simply making your web content available in the local language may not work. Asian cultures may look superficially similar and even western on the surface, but can have deep cultural differences. The localization market is estimated to be $1.5B in 2010 and could grow dramatically. My sense is that those numbers miss much of the impact of recent growth as the Facebook trends show, mobile computing and successful bottom of the pyramid marketing strategies.


All of these factors point to fundamental shifts in the global economy and indicate that many of these trends will accelerate further. Asia is a significant opportunity for informed globalization managers -- and probably key for long-term leadership for many global enterprises.

Global companies need to develop broad and unique country-specific strategies to be able to prosper and thrive in this rapidly changing world. Localization and translation will be key elements of any successful globalization plan and should present significant opportunities to vendors that prepare for this change.

It's wise to remember that the Chinese ideogram for "change" can also mean "opportunity."