Linking Translation Quality to Business Purpose

The question of translation quality has been a hot mess for the professional translation industry for some time now. It is regularly a subject of discussion, a core theme in every industry conference and yet is also continuously the greatest source of confusion or disagreement amongst translation professionals. This is even further complicated when considering how to determine what an acceptable quality measurement is for many machine translation projects.  One major source of confusion is caused by the conflation of process quality standards with linguistic quality standards.They are different and should not be mixed and matched. The SMT developer community has added to the confusion by introducing automated translation quality measurements like BLEU, METEOR and TERp. I have attempted in this blog to try and clarify some of the SMT stuff at least. But to be able to develop useful MT systems it is necessary to get some rough working definitions in place so that work can be planned, done and fairly compensated. 

Quality Definitions that do NOT seem to work
For the most part, certification in ISO 9001, EN15038, Microsoft QA and LISA QA 3.1 does not provide a buyer with a clear sense of the actual translation quality that will be delivered, even though it does suggest that the certified LSP has process discipline and a quality sensibility. Also, automated measurements like BLEU have very little value to translation professionals as general  linguistic quality measurements and should be used with care anyway. This is why buyers always ask for test/sample translations. They need to have some understanding of what they can expect before they undertake a larger project.

The most widely accepted quality definition that I am aware of in the TEP world is that quality is achieved when the buyer or the customer is satisfied.  Making content useful for a target customer is perhaps a way to approach MT projects as well. Since MT is unlikely to produce the quality produced by a typical human translation process it does not make sense to always use the same quality assessment procedures unless the goal is in fact to get to a human quality level. Wherever high quality is required more human post-editing support will be required, but there are many business use scenarios where lower linguistic quality can adequately accomplish business objectives.  However, in every case we discuss below, the objective is to produce better quality than the free translation services like Google, MSN and Babelfish as I assume that competent professional involvement will produce systems that outperform these free alternatives.


I thought that it would be useful to present three different scenarios where MT could be used, and then see how the business objectives impact the translation quality requirements.

 

eDiscovery & Litigation Support

Business Objective: Identify Key Documents and ensure Searchability of Multilingual Content. Speed & Fast Turnaround Are Critical.
Translation Quality Required: Raw MT produced by a lightly customized engine that has paid particular attention to accurate translation of key terminology and search terms is adequate. Once key documents are identified they go through a more standard human translation process.
Typical Business Use Scenarios: Very large content, very short time frame for turnaround and rapid identification of key documents e.g. Litigation Support and other eDiscovery applications. Also valuable in Military and National Security applications that involve information triage of foreign language blogosphere or websites to “find a smoking gun”.
LSP Role: Translation professionals can play a significant role in developing critical terminology and working on the MT engine to get it to acceptable quality levels for the business application. In patent litigation some TM preparation and training may also be required.

Knowledge Base Support Content

Business Objective: Provide Current and Comprehensive Technical Support and Knowledge Base content to global customers in target languages. Rapid Availability and Breadth of coverage are critical and should be aligned and equivalent to English content. The success and ability of the customer to perform self-service and solve technical problems is critical.
Translation Quality Required: The most active and most critical content is translated by humans. For the bulk of the KB content, raw MT produced by a carefully customized engine that uses localization content, active KB article translations and key terminology as training material is adequate. The content can be linguistically flawed and imperfect but still useful as long as the basic meaning is accurate. Initial content is put into production and made available quickly and can be further refined and improved gradually over time. Users are warned when content is raw MT.In 2010 possibly 1 billion+ people will use this type of translated content.
Typical Business Use Scenarios: Very large content, very short time frame for turnaround and continuing updates on a very regular basis. Major IT firms like Microsoft, Intel, HP have a huge need for this to reduce support costs, improve support experience of global customers and facilitate support  processes for key resellers in global markets. Any industrial engineering / manufacturing company that has a global customer base and wants to improve the technical support experience could be a viable user and many will do so as the success of initial projects is publicized.
LSP Role: Translation professionals can play a significant role in translating most active KB articles, critical content (financial transactions and security related), developing critical terminology and tuning the MT engine to get it to acceptable quality levels for customer acceptance testing. It is possible that they have an ongoing role in improving the MT engine to periodically re-translate the whole corpus as a higher quality level.

High Volume Localization Content

Business Objective: Improve production and efficiency of critical dealer channel service and maintenance documentation for global customers in several target languages. Cost and Quality are as important as Speed to ensure brand identity. OEMs who rely on globally distributed service and maintenance network need to provide regular updates on maintenance documentation to their dealer channel with new models of major products.
Translation Quality Required: Brand consistent human quality is required. Large high quality TMs and glossaries are available to train MT systems. MT engine is carefully developed and tuned to maximize the raw quality and ensure high productivity post-editing effort. MT + Post-editing _ proofing process is used to quickly bring MT output to human quality levels.  Same quality checks are applied as would be to a fully human process and the buyers will have the same expectations.
Typical Business Use Scenarios: Large content (500 to 3000 page manuals), rapid turnaround expected but not real time, and periodic updates as models change. Major Automotive manufacturers, Electronic Equipment OEMs have an ongoing need for this to ensure that their service and dealer channels can properly service new models.
LSP Role: Translation professionals need to play a significant role in building the MT engine and ensuring that it produces high quality output that requires a minimum amount of clean-up work (post-editing).  LSPs need to monitor and validate that they are getting  increasingly higher levels of productivity. This should make each successive version or model easier and more cost effective to produce. It is reasonable to expect new manuals to be done faster and more efficiently.

Thus we have a relatively new concept and approach to quality as the use of MT expands the scope of translation projects. Some refer to this less than human quality  as “usable” and some call it “good enough”. There are many who find this expanding use of MT very threatening, but the greatest use of MT so far is still in areas that just could not be done any other way. As skill levels amongst translation professionals increase and the MT quality improves, it is likely that we will see MT being used in many high volume localization projects, driven primarily by cost and production efficiencies. The industry has yet to really develop clear quality definitions so that buyers and service providers can build effective business models around these new possibilities. However, given the content deluge that we increasingly face, this is coming, and this is a significant opportunity for those service providers who take the time to develop the skills to undertake these and other high-volume high-value kinds of projects.