The Continuing Evolution of Enterprise MT

I recently hosted a round table panel discussion organized by Multilingual with three leading enterprise MT players, to discuss and share thoughts on the state of Enterprise MT today. The panelists are all firmly established in the enterprise MT arena, and a key objective of the session was to contrast the focus and differing requirements of enterprise MT from the generic consumer portal MT that is so easily available today.  

MT technology has become pervasive, and today more than a trillion words are translated every single day by the many free MT portals on the web. However, the MT needs of the enterprise are different and much more specialized. Generic MT technology is not always adequate for the many new uses that a global enterprise might have for high volume translation. In this panel discussion, we talk to three leaders working in the Enterprise MT space about:

• Customization and adaptation requirements for the enterprise
• Issues in measuring and understanding relative MT output quality
• The outlook for continuing improvements in core MT technology
• Use cases for MT beyond localization

The MultiLingual Summer Series: Meaningful Conversations with Thought Leaders have other interesting sessions coming up in August.

The panelists present in the session included:

 

Chris Wendt graduated as Diplom-Informatiker from the University of Hamburg, Germany, and subsequently spent a decade on software internationalization for a multitude of Microsoft products including Windows, Internet Explorer, MSN, and Bing – bringing these products to market with equal functionality worldwide. Since 2020 he is leading the group of language services in the Azure Cognitive Services family, bringing natural language processing capabilities to its customers.

 

Alon Lavie leads and manages Unbabel’s US AI lab based in Pittsburgh, and provides strategic leadership for the AI R&D teams company-wide. For almost 20 years (1996-2015) Alon was a Research Professor at the Language Technologies Institute at Carnegie Mellon University, where he continues to serve as an adjunct professor.

 

 

 Joern Wuebker, Director of Research at Lilt. the AI-powered enterprise translation software and services company. Prior to joining Lilt, Joern earned a Ph.D. in computer science from RWTH Aachen, where he focused on machine translation research.

The complete session is available as a video on the Multilingual site at the links shown above, however, it requires a viewer to sit through the full session. While there are some solutions that allow video content to be indexed and annotated, and thus make the content discovery easier, these are not in wide use yet. Thus, I thought it would be useful to provide subject-related links to the different sections of the full session in this post to allow a viewer to selectively focus on the subject matter of greatest interest. I have also summarized some of the important points in my own words and added new comments where I thought it might add greater clarity.

What are the characteristics and requirements of enterprise MT? How does it differ from generic MT?

• Strong focus on data security and privacy of customer data
• The adaptability and customization possibilities available to tailor the MT system to unique customer requirements
• Service level requirements to meet corporate IT requirements: Scalability, Batch vs. Interactive
• Matching and adjusting MT quality to different enterprise use case requirements: correct translation in context and proper handling of relevant content types especially when MT is used without downstream human adjustment
• Handling of varied content types (structured documents) and formats in different business workflows
• Maintaining data privacy in dynamic workflows and data interactions
• The growing need for interactivity and dynamic adaptation to meet organizational requirements as rapidly and efficiently as possible

What do we mean by adaptation and customization to meet unique organizational needs? 

• Offline adaptation can often be ineffective referring to TAUS study according to Lilt who favors dynamic and interactive human-in-the-loop adaptation which seems especially well suited for localization use cases but could be less useful for very high volume use cases like eDiscovery and eCommerce
• The TAUS COVID MT study should not be used to draw larger conclusions
• Enterprise content is constantly changing and so MT engines need to evolve and cannot be static for best outcomes when using the technology
•  Adaptation should be driven by content variations and evolution
• Microsoft has seen that on average their customers get 10+ BLEU improvements from their customization efforts
• Handling of unique terminology, style, and branding correctly is increasingly seen as a critical need for the enterprise

How do we handle the problem that there never seems to be enough data to properly customize or adapt an MT engine to unique customer requirements?

• Lilt claims they don't need large data volumes to adapt an engine, but rather they need very specific and relevant in-context data that is current and reflects the immediate focus to enable effective adaptation
• Adaptation can start with just a few key terms and evolve with TM and/or human feedback that can be gathered over time
• Context and high relevance is now increasingly being seen as more important than data volume
• Make the customization process demand-driven. Focus your correction and improvement processes on the content that matters the most
• Data manufacturing is of dubious value but there are some synthetic data strategies that make some sense e.g. back translations which can be "surprisingly effective".

How do we address the challenges around evaluating MT quality given the increasing awareness and recognition that BLEU scores are not as useful with NMT?

• Understand the source data and choose the right metrics to determine accuracy. At Lilt, they use "next word accuracy" as a key measure to determine quality evolution progress
• The purpose of  MT deployment needs to be properly understood first? Does the MT solve the problem? Does the machine translation improve user experience even though it is not linguistically perfect?
• BLEU is a better laboratory evaluation rather than a real-world evaluation and still has value to developers
• A human evaluation has to be involved to get a better picture of the MT quality reality but these are expensive and slow 
• Unbabel uses MQM based error evaluation techniques to develop improved quality predictors 
• Unbabel has developed a neural-based evaluation approach that is showing great promise with an initiative called COMET. Details will be coming forth soon.

How does Lilt measure MT output quality improvements given that their engine is changing dynamically?

• Lilt measures every sentence and learns from every single sentence
• They can tell from the improvements in "next word accuracy" and see how it evolved from an initial engine
• GPT-3 next word prediction is not a threat to our approach as it is monolingual and requires a huge model that would make it impractical

How do we handle MT use cases beyond localization where potentially billions of words are being translated and a human-in-the-loop deployment is not practical?

• Raw MT can be used to make much more content available to global customers even if the value of this is unknown. The enterprise can make more content available to identify what content has the greatest value in different markets
• User-generated content (UGC) is hard to customize around and generic engines may work just as well
• Communication scenarios are much more important to the business mission usually than UGC, even when an enterprise has very short shelf-life content and thus quality measurement happens in realtime e.g. chatbots, live support
• Communication content strategies vary for inbound (UGC) and outbound (customer support). Outbound communications can involve human-in-the-loop (HITL) if the content turnaround times allow this or if the content needs it.
• Realtime quality estimation capabilities become a much more critical enabling element for other use cases especially with social media and user forum communication where it is desirable to use raw MT.

What is the difference between linguistic steering and post-editing?

• Focusing on high-density language patterns rather than the full corpus to ensure that critical word patterns are properly learned and handled. PEMT is a full corpus evaluation strategy
• Unbabel develops dedicated test suites to ensure that the quality is acceptable for the linguistic material that really matters
• Raw MT can create the danger of making a catastrophic mistake and experts need to find ways to identify and handle these kinds of errors
• Lilt feels that they could handle the translation of a 25M word corpus through their interactive and dynamic evolution and predictions will get better and better. Lilt can push the boundaries of HITL to handle a very large corpus.

What is the impact of "ground-breaking NLP research (BERT/GPT-3)" on the world of MT?

• Text-generation technologies are very cool but they actually do NOT really understand the text
• Catastrophic failure will be common and easy to achieve :-)
• Large LM approaches can help with document-level context but they are mostly monolingual 
• Also, these approaches are not as effective as dedicated NMT engines on handling the translation task
• These new breakthroughs can be used to leverage other downstream NLP tasks
• The primary issue is that they are mostly monolingual at the moment, but they could be valuable as they become more multilingual
• COMET is built on top of some of these initiatives
• Multilingual training experiments have shown that there can be improvements for low resource languages even if they are unrelated

Audience question: Given Cybernetics 2.0, how crucial is human feedback for NMT?

• Human feedback will always be required 
• Language is a living thing and will continue to evolve
• Consider COVID 19, how many MT engines knew about this even 6 months ago?

Do you see continuously improving NMT displacing ever greater numbers of human translators?

• Machines are already translating 1000X what humans are doing  so this is already true in terms of volume but humans are the primary communicators 
• The roles of translators are evolving as technology improves but there are a finite number of human translators and HT will always be needed
• "Human Translation" is not monolithic, and there are many kinds of translators, thus, we should be careful to not lump all translators as equivalent, as there is a wide range in terms of competence and expertise
• MT cannot replace competent, subject matter expert translators who understand the communication intent, the semantic core and comprehend the communication impact of a translation
• Competent HT will always be the final measure for quality 
• As long as there are humans there will be a need for human translation.

This last question evoked some spirited social media response and I saw this comment being retweeted many times on Twitter.

The full session can be viewed below and I will add a link of the KUDO version which will also provide interpretation in Chinese, Italian, Russian, and Spanish when it becomes available.

Understanding Data Security with Microsoft Translator

In this time of the pandemic, many experts have pointed out that enterprises that have a broad and comprehensive digital presence are more likely to survive and thrive in these challenging times. The pandemic has showcased the value of digital operating models and is likely to force many companies to speed up their digital innovation and transformation initiatives. The digital transformation challenge for a global enterprise is even greater, as the need to share content and expand the enterprise's digital presence is massively multilingual, thus putting it beyond the reach of most localization departments who have a much narrower and much more limited focus. 

Thus, today we are seeing that truly global enterprises and agencies have a growing need to make large volumes of flowing content multilingual, to enable communication, problem resolution, collaboration, and knowledge sharing possible, within and without the organization. Most often this needs to be as close to real-time as possible. The greater the enterprise commitment to digital transformation, the greater the need, and urgency. Sophisticated, state-of-the-art machine translation enables multilingual communication and content sharing to happen at scale across many languages in real-time, and is thus becoming an increasingly important core component of enterprise information technology infrastructure. Enterprise tailored MT is now increasingly a must-have for the digitally agile global enterprise.

However, MT is extremely complex and is best handled by focused, well funded, and committed experts who build unique competence over many years of experience. Many in the business translation world dabble with open source tools, and build mostly sub-optimal systems that do not reach the capabilities of generic public systems, and thus create friction and resistance from translators who are well aware of this shortcoming. MT system development still remains a challenge for even the biggest and brightest, and thus, in my opinion, is best left to committed experts.

Given the confidential, privileged and mission-critical nature of the content that is increasingly passing through MT systems today, the issue of data security and privacy is becoming a major factor in the selection of MT systems by enterprises concerned with being digitally agile, but who also wish to ensure that their confidential data is not used by MT technology providers to refine, train, and further improve their MT technology. 

While some believe that the only way to accomplish true security is by building your own on-premise MT systems, this task as I have often said, is best left to large companies with well-funded and long-term committed experts. Do-it-yourself (DIY) technology with open source options makes little sense if you don't really know, understand, and follow what you are doing with technology this complex.

It is my feeling that MT is also a technology that truly belongs in the cloud for enterprise use, and also usually makes more sense on mobile devices for consumer use. While in some rare cases, on-premise MT systems do make sense for truly massive scale users like national security government agencies (CIA, NSA) who can appropriate the resources to do it competently, for most commercial enterprise MT provides the greatest ROI when it is delivered and implemented in the cloud by an expert and focused team who do not have to re-invent the wheel. Customization on a robust and reliable expert MT foundation appears to be the optimal approach. MT is also a technology that is constantly evolving as new tools, algorithms, new data, and processes come to light to enable ongoing incremental improvements, and this too suggests that MT is better suited to cloud deployment. Neural MT requires relatively large computing resources, deep expertise, and significant data resources and management capabilities to be viable. All these factors point to MT best being a cloud-based deployment, as it essentially remains a work-in-progress, but I am aware that there are still many who disagree on this, and that the cloud versus on-premise issue is one where it is best to agree to disagree.

I recently sat down with Chris Wendt, Group Program Manager and others in his team responsible for Microsoft Translator services, including Bing Translator and Skype Translator. They also connect Microsoft’s research activities with its practical use in services and applications. My intent in our conversation was to specifically investigate, better understand, and clarify the MT data security issues and the many adaptation capabilities that they offer to enterprise customers, as I am aware that the actual facts are often misrepresented, misunderstood, or unclear to many potential users and customers.

Microsoft is a pioneer in the use of MT to serve the technical support information needs of a global customer base, and was the first to make massive support knowledge bases available in MT'd local language for their largest international markets. They were also very early users of Statistical MT (SMT) at scale (tens of millions of words translated for millions of users) and were building actively used systems around the same time that Language Weaver was commercializing SMT. Many of us are aware that the Microsoft Translator services are used both by large enterprises and many LSP agencies in the language services industry because of the relative ease of use, straightforward adaptation capabilities, and relatively low cost.  Among the public MT portals, Microsoft is second only to Google in terms of MT traffic, and their consumer platforms solutions on the web and mobile platforms are probably used by millions of users across the world on a daily basis.

It is important to differentiate between Microsoft’s consumer products and their commercial products when considering the data security policies that are in place when using their machine translation capabilities, as they are quite different.

Consumer Products: 

The consumer products are Bing, the Edge browser, and the Microsoft Translator app for the phone. These products run under the consumer terms of use, which make it possible for Microsoft to use the processed data for quality improvement purposes. Microsoft keeps a very small portion of the data, non-consecutive sentences, and without any information about the customer who submitted the translation. There is really nothing to learn from performing a translation. The value only comes when the data is annotated and then used as Test or Training data. The annotation is expensive, so there are only a few thousand sentences used per language every year, at most.

Some people read the consumer terms of use and assume the same applies to commercial enterprise products.

That is not the case.

Enterprise Products:

The Translator API is provided via an Azure subscription, which runs under the Azure terms of use. The Azure terms of use do not allow Microsoft to see any of the data being processed. Azure services generally run as a GDPR processor, and Translator ensures compliance by not ever writing translated content to persistent storage.

The typical process flow for a submitted translation is as follows:

Decrypt > translate > encrypt > send back > and > forget.

The Translator API only allows encrypted access, to ensure data is safe in transit. When using the global endpoint, the request will be processed in the nearest available data-center. The customer can also control the specific processing location by choosing a geography-specific endpoint from ten locations which are described here. 

Microsoft Translator is certified for compliance with the GDPR processor and confidentiality rules. It is also compliant with all of the following:

CSA STAR: The Cloud Security Alliance (CSA) defines best practices to help ensure a more secure cloud computing environment, and to helping potential cloud customers make informed decisions when transitioning their IT operations to the cloud. The CSA published a suite of tools to assess cloud IT operations: the CSA Governance, Risk Management, and Compliance (GRC) Stack. It was designed to help cloud customers assess how cloud service providers follow industry best practices and standards and comply with regulations. Translator has received CSA STAR Attestation. 

FedRAMP: The US Federal Risk and Authorization Management Program (FedRAMP) attests that Microsoft Translator adheres to the security requirements needed for use by US government agencies. The US Office of Management and Budget requires all executive federal agencies to use FedRAMP to validate the security of cloud services. Translator is rated as FedRAMP High in both the Azure public cloud and the dedicated Azure Government cloud. 

GDPR: The General Data Protection Regulation (GDPR) is a European Union regulation regarding data protection and privacy for individuals within the European Union and the European Economic Area. Translator is GDPR compliant as a data processor.

HIPAA: The Translator service complies with the US Health Insurance Portability and Accountability Act (HIPAA) Health Information Technology for Economic and the Clinical Health (HITECH) Act, which governs how cloud services can handle personal health information. This ensures that health services can provide translations to clients knowing that personal data is kept private. Translator is included in Microsoft’s HIPAA Business Associate Agreement (BAA). Health care organizations can enter into the BAA with Microsoft to detail each party’s role in regard to security and privacy provisions under HIPAA and HITECH.

HITRUST: The Health Information Trust Alliance (HITRUST) created and maintains the Common Security Framework (CSF), a certifiable framework to help healthcare organizations and their providers demonstrate their security and compliance in a consistent and streamlined manner. Translator is HITRUST CSF certified.

PCI: Payment Credit Industry (PCI) is the global certification standard for organizations that store, process or transmit credit card data. Translator is certified as compliant under PCI DSS version 3.2 at Service Provider Level 1. 

SOC: The American Institute of Certified Public Accountants (AICPA) developed the Service Organization Controls (SOC) framework, a standard for controls that safeguard the confidentiality and privacy of information stored and processed in the cloud, primarily in regard to financial statements. Translator is SOC type 1, 2, and 3 compliant. 

US Department of Defense (DoD) Provisional Authorization: US DoD Provisional Authorization enables US federal government customers to deploy highly sensitive data on in-scope Microsoft government cloud services. Translator is rated at Impact Level 4 (IL4) in the government cloud. Impact Level 4 covers Controlled Unclassified Information and other mission-critical data. It may include data designated as For Official Use Only, Law Enforcement Sensitive, or Sensitive Security Information.

ISO: Translator is ISO certified with five certifications applicable to the service. The International Organization for Standardization (ISO) is an independent nongovernmental organization and the world’s largest developer of voluntary international standards. Translator’s ISO certifications demonstrate its commitment to providing a consistent and secure service. Translator’s ISO certifications are:

• ISO 27001 Information Security Management Standards
• ISO 9001:2015 Quality Management Systems Standards
• 27018:2014 Code of Practice for Protecting Personal Data in the Cloud
• 20000-1:2011: Information Technology Service Management
• ISO 27017:2015: Code of Practice for Information Security Controls

 

The Translator service is subject to annual audits on all of its certifications to ensure the service continues to be compliant.

These standards force Microsoft to review every change to the live site with two employees, and to enforce minimal access to the runtime environment, as well as having processes in place to protect against external attacks on the data center hardware and software. The standards that Microsoft Translator is certified for, or compliant with, include specific ones for the financial industry and health care providers.

Different from the content submitted for translation, the documents the customer uses to train a custom system are stored on a Microsoft server. Microsoft doesn’t see the data and can’t use it for any purpose other than building the custom system. The customer can delete the custom system as well as the training data at any time, and there won’t be any residue of the training data on any Microsoft system after deletion, or after account expiration.

Translation in Microsoft’s other commercial products like Office, Dynamics, Teams, Yammer, SharePoint, and others follow the same data security rules described above.

Chris also mentioned that, "German customers have been very hesitant to recognize that trustworthy translation in the cloud is possible, for most of the time I have been working on Translator, and I am glad to see now that even the Germans are now warming up to the concept of using translation in the cloud." He pointed me to a VW case study where I found the following quote, and also rationale on the benefits of a cloud-centric translation service to a global enterprise that seeks to enable and enhance multilingual communication, collaboration and knowledge sharing. A deciding factor for the team responsible at VW [in selecting Microsoft] was that none of the data – translation memories, documents to be translated, and trained models – was to leave the European Union (EU) for data protection reasons.

“Ultimately, we expect the Azure environment to provide the same data security as our internal translation portal has offered thus far,”

Tibor Farkas, Head of IT Cloud at Volkswagen

Chris closed with a compelling statement, pointing to the biggest data security problem that exists in business translation: incompetent implementation. Cloud services properly implemented can be as secure as any connected on-premise solution, and in my opinion the greatest risk is often introduced by untrustworthy or careless translators who interact with MT systems, or incompetent IT staff that maintain an MT portal as the Translate.com fiasco showed. . 

"Your readers may want to consider whether their own computing facilities are equally well secured against data grabbing and whether their language service provider is equally well audited and secured. It matters which cloud service you are using, and how the cloud service protects your data."

While I have not focused much on the speech-to-text issue in this post, we should understand that Microsoft also offers SOTA (state-of-the-art) speech-to-text capabilities and that the Skype and Phone app experience also gives them a leg up on speech-related applications that go across languages.

I also gathered some interesting information on the Microsoft Translator customization and adaptation capabilities and experience. I will write a separate post on that subject once I gather a little more information on the matter.

Data Preparation Best Practices for Neural MT

In any machine learning task, the quality and volume of training data available is a critical determinant of the system that is developed. The importance of data is real for both Statistical MT and Neural MT, which are both data-driven, and produces output that is deeply influenced by the data used to train them. Some believe that Statistical MT systems have a higher tolerance for noisy data. Thus it is assumed that more data volume is better even if it is "noisy," but in my experience, all data-driven MT systems are better when you have quality data. Research shows that Neural MT is more sensitive to noise than Statistical MT. Still, as SMT has been around for 15+ years now, many of the SMT data preparation practices in use historically often continue and are carried over to NMT model building today.

This problem has raised interest in the field of parallel data filtering to identify and correct the most problematic issues for NMT, e.g., segments where source and target are the same, and misaligned sentences. This presentation by eBay provides an overview of the importance of parallel data filtering and its best practices. It adds to the useful points made by Doctor-sahib in this post. Data cleaning and preparation have always been necessary for developing superior MT engines, and most of us agree that it is even more critical now with neural network-based models.

This guest post is by Raymond Doctor, who is an old and wise acquaintance of mine who has spent over a decade at the Centre for Development of Advanced Computing (C-DAC) in Pune, India. He is a pioneer in digital Indic language work and was involved in several Indic language NLP based initiatives conducting research on Indic language Parsers, Segmentation, Tokenization, Stemming, Lemmatization, NER, Chunking, Machine Translation, and Opinion Mining.

The success of these MT experiments is yet more proof that the best MT systems come from those who have a deep understanding of both the underlying linguistics, as well as the MT system development methodology.

He and I also share two Indian languages in common (Hindi and Gujarati). Over the years, he has shown me many examples of output from MT systems he has developed in his research that were the best I had seen for these two languages going into and out of English. The success of his MT experiments is yet more proof that the best MT systems come from those who have a deep understanding of both the underlying linguistics, as well as the MT system development methodology. 

Overview of the SMT data alignment processes

"True inaccuracy and errors in data are at least relatively straightforward to address, because they are generally all logical in nature. Bias, on the other hand, involves changing how humans look at data, and we all know how hard it is to change human behavior."

- Michiko Wolcott

Some other wisdom about data from Michiko:

Truth #1: Data are stupid and lazy.

Data are not intelligent. Even artificial intelligence must be taught before it learns to learn on its own (even that is debatable). Data have no ability on their own. It is often said that insights must be teased out of data.

Truth #2: Data are rarely an objective representation of reality (on their own).

I want to clarify this statement: it does not say that data is rarely accurate or error-free. Accuracy and correctness are dimensions of quality of what is in the data themselves.

The text below is written by the guest author.

                                                  **************

Over the years, I have been studying the various recommendations given to prepare training data before submitting it to an NMT learning engine. I feel these recommended practices mainly emerged as best practices at the time of SMT, and have been carried over to NMT with less beneficial results.

I have identified six major pitfalls that data analysts make when preparing training data for NMT models. These data cleaning and preparation practices originated as best practices with SMT, where they were of benefit. Many data practices that made sense with SMT are still being followed today, and it is my opinion that these should be avoided and are likely to result in better outcomes.

While I have listed a few practices that I feel should be avoided, many other SMT-based data prepping practices ensure that the training data is likely to produce a sub-optimal NMT system. But the factors I have listed below are the most common practices which have resulted in lower output quality than would be possible by ignoring these practices. I disregarded the advice given regarding punctuations, deduping, removing truncations, MWEs, and found the quality of NMT output considerably improves in my research with Indic language MT systems.

As far as possible, examples have been provided from a Gujarati <> English NMT system I have developed. But the same can apply to any other parallel corpus.

1. PUNCTUATION

Quite a few sites tell you to remove punctuations before submitting the data for learning. It is my observation that this is not optimal practice.

Punctuations are markers that allow for understanding the meaning. In a majority of languages word order does not necessarily show interrogation

Tu viens? =You are coming?

Removing the interrogation marker creates confusion and dupes [see my remark below]

See what happens when a comma is removed:

Anne Marie, va manger mon enfant=Anne Marie. Come have your lunch

Anne Marie va manger mon enfant=Anne Marie is going to eat my child

 

The mayor says, the commissioner is a fool.

The mayor, says the commissioner is a fool.

I feel that in preparing a corpus the punctuation markers should be retained.

2. TRUNCATIONS AND SHORT SENTENCES

Quite a few sites advise you to remove short sentences. Doing this, in my opinion, is a serious error. Short sentences are crucial for translating headlines, one of the stumbling blocks of NMT. Some have no verbs and are pure nominal structures.

Curfew declared: Noun + Verb

Sweep of Covid19 over the continent: Nominal Phrase

 

Google does not handle nominal structures well, and here is an example:

Sweep of Covid over India= ભારત ઉપર કોવિડનો સ્વીપ

I have found that retaining such structures strengthens and improves the quality of NMT output.

3. MULTIWORD EXPRESSIONS

Multiword expressions (MWEs) are expressions that are made up of at least two words, and which can be syntactically and/or semantically idiosyncratic in nature. Moreover, they act as a single unit at some level of linguistic analysis.

Like short sentences, MWEs are often ignored and removed from the training corpus. These MWEs are very often fixed patterns found in a given language. These can be short expressions, titles, or phrasal constructs, just to name a few of the possibilities. MWEs cannot be literally translated and need to be glossed accurately. My experience has been that the higher the volume of MWEs provided, the better the quality of learning. A few MWEs in Gujarati are provided below:

agreement in absence =અભાવાન્વય

agreement in presence =ભવાન્વય

agriculture parity =કૃષિમૂલ્ય સમાનતા

aid and advice =સહાય અને સલાહ

aider and abettor =સહાયક અને મદદગાર

aim fire =નિશાન લગાવી ગોળી ચલાવવી

4. DUPLICATES

A large number of sites providing recommendations on NMT training data preparation tell you to remove duplicates, both in the Source and Target texts. This action in popular parlance is termed as deduping. The argument being that deduping the corpus makes for greater accuracy. However, it is common to have an English sentence that can map to two or more strings in the target language. This variation can be because of synonyms used in the target languages, or because of a flexible word order that is especially common in Indic languages. De-duping the data results in weakening the quality of MT output. The only case where deduping needs to be done is when we have two identical strings, both in the Source and Target language. Higher quality NMT engines incorporate these slight variations on a single segment to enable the MT engines to produce multiple variants.

Change of verbal expression and word order:

How are the trade talks between China and the US moving forward now. =ચીન તથા અમેરિકા વચ્ચે વેપાર વ્યવહાર વિષયક વાતચીત હવે કેવી આગળ વધે છે.

How are the trade talks between China and the US moving forward now. =ચીન તથા અમેરિકા વચ્ચે હવે વેપાર વિષયક વાતચીત કેવી આગળ વધે છે.

Synonyms:

Experts believe. =એક્સપર્ટ્સ માને છે.

Experts believe. =જાણકારોનું માનવું છે.

Experts believe. =નિષ્ણાતોનું માનવું છે.

Deduping the data in such cases results in reducing the quality of output.

The only case where deduping needs to be done is where we have two identical strings, both in the Source and Target language. In other words, an exact duplicate. High-end NMT engines do not practice deduping since this deprives the MT system from being able to provide variants, which can be seen by clicking on full or part of the gloss.

5. VERBAL PATTERNS

The inability to handle these are the Achilles heel of a majority of NMT engines, including Google, insofar as English to Indic languages are concerned. Attention to this area is ignored because it is felt that the corpus will handle all verbal patterns in both the source and target language. Even the best of corpora does not handle this.

Providing a set of sentences with the Verbal Pattern of both the source and target languages goes a long way.

Gujarati admits around 40+ verbal patterns and NMT fails on quite a few:

They ought to have been listening to the PM's speech =તેઓએ વડા પ્રધાનનું ભાષણ સાંભળ્યું હોવું જોઈએ

Shown below is a sample of Gujarati verbal patterns with “to eat “ as a paradigm

You are eating =તમે ખાઓ છો
You are not eating =તમે ખાતા નથી
You ate =તમે ખાધું
You can eat =તમે ખાઈ શકો છો
You cannot eat =તમે નહીં ખાઈ શકો
You could not eat =તમે ખાઈ શક્યા નહીં
You did not eat =તમે ખાધું નહીં
You do not eat =તમે ખાતા નથી
You eat =તમે ખાધું
You had been eating =તમે ખાતા હતા
You had eaten =તમે ખાધું હતું
You have eaten =તમે ખાધું છે
You may be eating =તમે ખાતા હોઈ શકો છો
You may eat =તમે ખાઈ શકો છો
You might eat =તમે કદાચ ખાશો
You might not eat =તમે કદાચ ખાશો નહીં
You must eat =તમારે ખાવું જ જોઇએ
You must not eat =તમારે ખાવું ન જોઈએ
You ought not to eat =તમારે ખાવું ન જોઈએ
You ought to eat =તમારે ખાવું જોઈએ
You shall eat =તમે ખાશો

Similarly, the use of a habitual marker used when glossed into French by a high-quality NMT system

6. POLE AND VECTOR VERBS

This construct is very common in Indic languages and often leads to mistranslation.

Thus,  Gujarati uses જવું કરવું as an adjunct to the main verb. The combination of the pole and the vector verb such as જવું creates a new meaning.

 મરી જવું is not translated as die go, but is simply die

Gujarati admits around 15-20 such verbs, as do Hindi and other Indic languages, and once again, a corpus needs to be fed this type of data in the shape of sentences to produce better output.

 In the case of English it is the prepositional phrases that often create issues:

Pick up, pick someone up, pick up the tab

Conclusion

We noticed that when training data that ignores some of the frequent data preparation recommendations are sent in for training, the quality of MT output markedly improves. However, there is a caveat. If the threshold of the training data is lower than 100,000 segments, following or not following the above recommendations make little or no difference. Superior NMT systems require a sizeable corpus, and generally, we see that at least a million+ segments are needed.

Here is a small set of sentences from various domains is provided below as proof of the quality of output using these techniques

Now sieve this mixture.=હવે આ મિશ્રણને ગરણીથી ગાળી લો.

It is violence and violence is sin.=હિંસા કહેવાય અને હિંસા પાપ છે.

The youth were frustrated and angry.=યુવાનો નિરાશ અને ક્રોધિત હતા.

Give a double advantage.=ચાંલ્લો કરીને ખીર પણ ખવડાવી.

The similarity between Modi and Mamata=મોદી અને મમતા વચ્ચેનું સામ્ય

I'm a big fan of Bumrah.=હું બુમરાહનો મોટો પ્રશંસક છું.

38 people were killed.=તેમાં 38 લોકોના મોત થયા હતા.

The stranger came and asked.=અજાણ્યા યુવકે આવીને પૂછ્યું.

Jet now has 1,300 pilots.=હવે જેટની પાસે 1,300 પાયલટ છે.

====================================================================

 

 

Raymond Doctor,  has spent over a decade at the Centre for Development of Advanced Computing (C-DAC) in Pune, India. He is a pioneer in digital Indic language work and was involved in several Indic language NLP based initiatives and conducted research in furthering Indic language Parsers, Segmentation, Tokenization, Stemming, Lemmatization, NER, Chunking, Machine Translation, and Opinion Mining. 

Evaluating Machine Translation Systems

This post is the first in a series of upcoming posts focusing on the issue of quality evaluation of multiple MT systems. MT system selection has become a more important issue in recent times as users and buyers realize that potentially multiple MT systems can be viable for their needs, but would like to develop better, more informed selection procedures.

I have also just ended my tenure at SDL, and this departure will also allow my commentary and opinion in this blog to be more independent and objective, from this point onwards. I look forward to looking more closely at all the most innovative MT solutions in the market today and providing more coverage on them.  

As NMT technology matures it has become increasingly apparent to many buyers that traditional metrics like BLEU that are used to compare/rank different MT systems and vendors are now often inadequate for this purpose, even though these metrics are still useful to engineers who are focused on building a single MT system.  It is now much more widely understood that best practice involves human evaluations used together with automated metrics. This combined scoring approach is a more useful input in conducting comparative evaluations of MT systems.  To the best of my knowledge, there are very few in the professional translation world who do this well, and it is very much an evolving practice and learning that is happening now. Thus, I invite any readers who might be willing to share their insights into conducting consistent and accurate human evaluations to contact me about doing this here.

Most of the focus in the localization world's use of MT remains on MTPE efficiencies (edit distance, translator productivity), often without consideration of how the volume and useable quality might change and impact the overall process and strategy. While this focus has value, it misses the broader potential of MT and "leaves money on the table" as they say.

We should understand the questions that we are most frequently asking is: 

• What MT system would work best for our business purposes?
• Is there really enough of a difference between systems to use anything but the lowest cost vendor?
• Is there a better way to select MT systems than just looking at generic BLEU scores?

I have covered these questions to some extent in prior posts and I would recommend this post and this post to get some background on the challenges in understanding the MT quality big picture.

The COVID-19 pandemic is encouraging MT-use in a positive way. Many more brands now realize that speed, digital agility, and a greater digital presence matter in keeping customers and brands engaged. As NMT continues to improve, much of the "bulk translation market" will move to a production model where most of the work will be done by MT.  Translators who are specialists and true subject matter experts are unlikely to be affected by the technology in a negative way, but NMT is poised to penetrate standard/bulk localization work much more deeply, driving costs down as it does so.

This is a guest post and an unedited independent opinion from an LSP (Language Service Provider) and it is useful in providing us an example of the most common translation industry perspective on the subject of multiple MT system evaluations. It is interesting to note that the NMT advances over SMT are still not quite understood by some, even though the bulk of the research efforts and most new deployments have shifted to NMT. 

Most LSPs continue to stress that human translation is "better" than MT which most of us on the technology side would not argue against, but this view loses something when we see that the real need today is to "translate" millions of words a day. This view also glosses over the fact that all translation tasks are not the same. Even in 2020 most LSPs continue to overlook that MT solves new kinds of translation problems that involve speed and volume and that new skills are needed to really leverage MT in these new directions. There is also a tendency to position the choice as binary MT vs Human Translation, even though much of the evidence is pointing to new man + machine models that provide an improved production approach. The translation needs of the future are quite different from the past and I hope that more service providers in the industry start to recognize this. 

I also think it is unwise for LSPs to start building their own MT systems, especially with NMT. The complexity, cost and expertise required are prohibitive for most. MT systems development should be left to real experts who do this on a regular and continuing basis. The potential for LSPs adding value is in other areas, and I hope to cover this in the coming posts.

Source: MasterWord

                                                                                                                                                               * =======*

It’s not a secret that machine translation (MT) has taken the world by storm. Almost everyone now has had some experience with MT, mostly in the form of a translation app such as Google Translate being popular. But MT comes in a variety of formats and is heavily utilized by businesses and institutions all over the world.

With that in mind, which MT system is best? Since MT comes in many colors, figuratively speaking, which one should you ought to rely on if you decide to build your own MT system? We’ll also talk more about translation quality and whether or not MT is suitable for specialized translations such as medical translation; a critical field now for any active translation company in light of the current coronavirus pandemic that has the whole world at a standstill.

What is Machine Translation?

Machine Translation, or MT, is software that is capable of translating text from a source language to a translated text of the target language. Over the years, there have been multiple variations of MT, but there are three definitive types; Rules-based Machine Translation (RBMT), Statistical Machine Translation (SMT), and Neural Machine Translation (NMT). Here’s a quick rundown of their characteristics, including their pros and cons between each other;

1. RBMT

Rules-Based Machine Translation is one of the earliest forms of MT. Its algorithm is language-based, meaning for it to know how to translate one source language to the other, it must rely on input data in the form of a lexicon, grammar rules, and other linguistic fundamentals. The problem with RBMT systems is scaling it efficiently as it becomes more complicated as more language rules are added. Also, RBMT is never ideal for obscure languages with minuscule data. However, with the development of advanced MT systems over the years, RMBT has largely been superseded, in which you'll know more about its successor next.

2.  SMT

Statistical Machine Translation, compared to RBMT, is designed to translate languages from statistical algorithms. SMT works by being fed with data in the form of bilingual text corpora, SMT is programmed to identify patterns in the data and form its translations from it. Patterns in this context mean how many times a certain word/phrase appears consistently in a certain context. This probability learning model allows SMT systems to render relatively appropriate translations compared. It’s pretty much like ‘If this is how was it done, then this is how it should be done’. 

SMT also must be fed with plenty of data just like RBMT, but MT developers of which includes translation app developers prefer SMT due to its ease of setting up due to numerous open-source SMT systems available, cost-effectiveness due to free quality parallel text corpora that are available online, higher translation accuracy than RMBT, and its ease of scalability as the system grows bigger.

But just like RBMT, SMT can’t function well if it’s fed with insufficient and poorly structured parallel text corpora. That being said, it’s not that ideal to translate obscure languages.

3. NMT

Neural Machine Translation is the latest development in MT. Think of it as an upgraded version of SMT in which its abilities are now supplemented with artificial intelligence (AI), specifically deep learning. Not only is it capable of coming through data faster, but it can also produce better outputs through constant trial and error. SMT does it the same way as well but the only difference, albeit a definitive one, is that it’s able to do it much faster and more accurately. Google Translate recently made the switch in 2016 to NMT from its old SMT system.

Its deep learning capability is such a real game-changer that it’s able to accomplish what RBMT and SMT; translating obscure regional languages. That’s why Google Translate can cover over 100 languages such as Somalian and Gaelic. But its outputs are questionable, to say the least as it needs some time to learn a language that has little reliable data lying around for it to use. However, the development of NMT just goes to show how far MT overall has evolved over the years.

What Makes A Good Machine Translation (MT) System?

There have been many MT systems over the years and many still in development. The ones that happened to survive the test of time are select variants of RBMT and most variants of SMT. NMT has quickly gained popularity and will slowly replace SMT as the years go by. What’s generally expected out of a good custom-built MT system is reliability and quality of outputs, pretty much like any other product or service out there.

If you’re looking for a reliable metric, then BLEU (Bilingual Evaluation Understudy) is one of the most widely used MT evaluation metrics. BLEU ranks MT systems between 0 being the worst and 1 being the best. It rates how close the translated text is to a human. The more human-like and natural-sounding the translation is, the better the score.

That being said, every MT developer creates their system according to not only the developer’s but also a client’s specifications and linguistic needs. So not one of them is alike. But there are MT platforms that are widely used by multiple clients due to their flexibility of being adapted to the client’s needs and ease of use. But even with a variety of MT systems being developed over the years, one thing remains the same; MT systems have to learn from a lot of quality data and must be given the time to learn.

They say that machines are inherently dumb and that they’re only as good as job or data are given to them. For MT, that notion still rings true up to this day and will most likely keep ringing for decades to come. However, quality data isn’t only what makes a good MT system.

There are platforms in which MT is integrated with other processes for it to render quality or at the very least, passable translations. Indeed, MT itself is a process onto its own, but its outputs, even with deep learning capabilities, is still not up to par with that of a professional translator. MT has to be integrated with other processes, namely computer-assisted translation (CAT) tools.

There are many CAT tools but two of the most essential are a glossary tool and translation memory. A glossary is simply a database of terminologies and approved translations. It’s a very simple feature but very important as it saves up a lot of time for the translator as they don’t need to constantly look back and forth which translation is the perfect choice for the source text at hand.

A translation memory is also like a glossary, but stores phrases and sentences. It also saves the translator valuable time as many translations recycle the same language such as user manuals, marketing collateral, and etc. A translation memory also helps by providing consistent language at a given domain and language pair.

I Now Pronounce You Man and Machine

However, even with all the bells and whistles, developers can equip an MT system with, is MT alone enough? Can MT alone produce accurate and quality translations that are demanded by the clients of language services today? MT is part of the solution but doesn’t comprise the complete picture. It sounds counterintuitive, but MT is best paired with a professional translator as a means of optimizing the translation process.

This unlikely union broke the predictions of many that saw MT giving professional translators a run for their money and driving translation companies out of business. Professional translators work with CAT tools as it helps them be more churn out more words than ever before and helps them be more consistent. Why the need for speed? Domo’s latest report states that “2.5 quintillion bytes of data are created every single day”—that’s a lot of data and most of it is not in English which creates the rising demand for translation services.

Also, by having a translator work together with an MT system, the translator is doing the MT system a favor as well by constantly feeding back revisions for the MT to learn from and render better outputs and suggestions. All in all, it’s a highly productive and beneficial two-way street between a translator and an MT system.

Of course, this ‘relationship’ will be all for moot if the MT system wasn’t developed to a satisfactory standard. That being said, developers have to take into account both translation clients and translators themselves.

They have to ensure that not only will the MT system procure quality translations for clients but can also adapt to the needs of the translators using them. Being convenient to use and having a friendly UX design is one thing, but being able to incorporate the inputs of a translator and accurately replicating it in similar contexts is also another thing.

What Do Professional Translation Services Have Over MT?

Specifically, what can a translation company that hires professional translators to do better than artificial intelligence (AI)? Apart from translation quality and consistency, a professional translator has one advantage; they’re human. It may sound cliche but a human can understand nuances and no MT or AI are light years away from replicating.

Unable to Understand Emotional, Cultural, and Social Nuances

As of now, there is no MT yet that is capable of accurately understanding jokes, slang, creative expressions, and so on. The abilities of MT shine brightly with formulaic sentences and predictable language conventions. But if confronted with linguistic habits that are natural in everyday conversations, MT falls apart. This problem is made more pronounced at a global scale since every culture and society has its own way of speaking all the way down to highly distinct street lingo.

Unable to Process Linguistic Nuances

Parent languages are divided by their regional vernaculars and dialects. When someone’s trying to translate English to Spanish, it’s actually just generic Spanish with no local ‘flavoring’. But if you’re aiming for translations that resonate true to how Spanish people or how Mexican people speak, then a professional translator with native-speaking ability is who you need. No MT system now is able to comprehend, let alone translate linguistic nuances reliably.

Unable to Keep Up With Linguistic Trends

Languages change every day with new words being constantly added and removed to the lexicon of world languages. Humor, slang, and creative expressions are a testament to that notion. Even social media has given rise to new creative expressions in ways human society has never experienced before with meme culture as one of the most notable examples. Even if NMT was somehow capable of keeping up, it would still need time for the data to accumulate for it to start translating. By that time, new slang would have already popped out.

Unable to Render Specialized and Highly Contextual Translations

What we mean by specialized here is text with highly nuanced terminology such as the literary field and also texts belonging to critical fields such as the legal, scientific, medical sector. Authors inherently embed their works with highly nuanced expressions and linguistic ‘anomalies’, so much so that there is no identifiable pattern for any MT that can work with since each author has their own voice.

For the legal, and the medical sector, have their own language conventions that although seem formulaic on the surface, the inherently specialized terminologies and the risk factor involved in these fields means no margin of error can be given to MT. There are MT systems used in these sectors but are always paired with a professional legal translator and professional medical translator.

Developing Your Own MT System

Even with the quality issues and other imperfections associated with MT, the demand for machine translation services. According to a report published in Market Watch, “The Global Machine Translation Market was valued at USD 550.46 million in 2019 and is expected to reach USD 1042.46 million by 2025, at a CAGR of 11.23% over the forecast period 2020 - 2025.”.

However, many are looking to develop their own company MT instead of ‘borrowing’ one from an external provider and for good reason. If a translation company is rendering plenty of niche translations in a given year, then configuring their own MT system is the most cost-effective investment as there will be no need to pay for licensing fees to external MT providers.

Many industries have their language conventions and jargon, in regards to internal communication mostly. For example, legalese is perfectly comprehensible to lawyers but downright alien-sounding to those with little legal knowledge. That being said, even businesses and organizations have their own language conventions that veer off from the industry norm. In that case, they would then have to build their very own MT systems, especially if they’re focusing on specific target foreign markets and audiences. 

So out of the 3 listed earlier, which one should you choose? It’s most likely SMT due to its popularity and how much support it gets. There are who have gone for a Hybrid MT by combining SMT and RBMT but that’s probably too intimidating for first-timers. If you want to make the big leap right from the start, then, by all means, go NMT if it meets your company’s objectives. 

Mind you that investing and training any MT system does come at a price and will take time. It’ll take time for glossaries and translation memories to develop, provided that the data used to feed the system is of standard. For a translation company, that usually isn’t a problem as in tandem with open-source parallel text corpora are the translation company’s own document archives.

Can You Choose MT Over a Translation Company?

Back then, instant language translation belonged to the category of futuristic science fiction gadgets. In fact, it still is today albeit we’ve heightened our standards. What we dream of now is instant voice interpretation. Specifically, being able to conduct a seamless multilingual conversation with anyone without the awkward pauses. But let’s get back to reality now. It’s hard not to be impressed with the abilities of MT today since we can easily witness it from our smartphones.

Even so, there are plenty of flaws associated with MT as discussed earlier that’s actually hindering it from developing serious widespread adoption. Be that as it may, MT as it now nevertheless has its own perks. Although one shouldn’t rely too much on MT at certain thresholds, doesn’t mean that you shouldn’t use it at all at specific situations. Here are some reasons why.

Cost

There are plenty of translation apps out there such as Google Translate as you might know already. All of them are free with the exception of premium access subscription payments to unlock more features. There are plenty of free translation plugins as well for website developers. Keep in mind that we’re talking about generic translators here and not the specialized MT systems from external providers that have licensing fees.

Speed and Convenience

At specific situations, some are just looking to have translation at the very moment they want it. Whether you’re a language student or a traveling businessperson, MT is your answer. It’s free and they can get results the moment they click the translate button. Even if it’s not 100% accurate, it at least gives them an implied meaning behind the translation.

For Generic, Repetitive, and Well-Resourced Languages

*Consider this pointer at your own risk*. One can certainly find MT if they have non-contextual and predictable text at hand such as simple and formulaic phrases. What you decide to do with it is all on you whether you use it only as a reference or actually employ it in a professional setting. That being said, the most quality translations you can get are from well-resourced such as Spanish, German, French, etc. If you tried translating, even a simple phrase from English to Chinese, you’ll unlikely get a similarly accurate translation since English and Chinese have vastly different language rules and an unrelated linguistic history.

A Note on Translation Quality in the Context of the Coronavirus Pandemic

Despite the vast improvements to MT, quality is still a significant issue and as you’re aware, human translators are there to guarantee that. However, in no situation is quality ever more necessary than in global communication in crisis as made evident by the current coronavirus pandemic, specifically in the form of medical translation. Medical translation is a highly specialized niche in translation and critical one too wherein the slightest mistranslation would lead to potentially unfortunate and even fatal consequences.

Medical translation must be provided by specialized medical translators who have complete mastery over their language pair (Ex. English to Spanish, Spanish to English) and extensive familiarity with medical terminology, medical practices, and code of ethics. They must undergo additional lengthy training before they can be classified as certified medical translators. That being said, are MT systems out of the picture?

There are MT systems that translate medical documents and medical research, but it must be under constant supervision from a certified medical translator. Connecting it to today’s crisis, there hasn’t been a recent time in history where a speedy translation of medical research has been more important than ever. Medical scientists all over the world are working together to understand the COVID-19 virus for them to come up with viable treatments and eventually, a vaccine. With that in mind, medical translation is the only bridge that’s making this level of coordination between medical scientists around the world possible.

Final Takeaway

Will there be a future where MT would be so advanced and almost human-like that professional translators would be an endangered species? If you were to judge by the pace of development of MT in such a short period, it would not be that unreasonable to believe in a future like that. However, let’s not put too much thought into it as it doesn’t pay attention enough to what is demanded from translation in the first place.

It’s apparent now that MT is good at servicing the translation speed and optimization needs, but as for quality, much of it belongs to the hands, or should I say the mind of a professional translator. That union would likely last for the next few decades. But let’s not hold ourselves to that prediction. Perhaps a game-changing MT feature is just a few years away or if our prediction holds true decades. But still, that’s considering our standards on translations, particularly on quality and human-ness, haven’t changed.

Author Bio:

Laurence Ian Sumando is a freelance writer penning pieces on business, marketing, languages, and culture.