The MT languages that will matter most over the next 50 years

 Translated recently announced that ModernMT now supports 200 languages, setting a new benchmark in the industry. No other commercial MT service currently supports such an extensive range of languages. By expanding its language coverage to a potential reach of 6.5 billion people, Translated enhances the ability of enterprises to create stronger connections with their users and customers worldwide, fostering better communication and understanding of a larger global customer base.

In the modern era, we are rapidly moving to a world where a global enterprise needs to expand the scope and nature of its communication and information sharing with the modern digital customer. There is a relationship between content strategy, e-commerce, and MT, as much of this new content that enhances the customer experience is constantly changing, and there is great value in making it multilingual to enable engagement with a broader global customer base.

The modern digital-first customer demands and expects much more relevant information from every organization they interact with. The lack of needed information can easily trigger a potential customer to walk away from a brand and company that may otherwise be a very good fit in terms of matching customer needs to available product offerings. We know that today:

• The modern buyer and customer journey has many digital touchpoints.
• Global-savvy companies are increasingly moving to a business model focused on customer needs, attempting to serve as much information as needed to improve the global customer experience.
• Companies are translating everything that might be useful to a customer, not just what is mandated by local commercial regulations.

So, what are some of the things that are required of a globally focused business to be successful with an ever-growing global customer base?

The list of actions recommended by globalization consultants on best practices for providing relevant information to the modern digital-first customer includes all of the following actions:

• Personalize communication and content to their interests and needs
• Provide easy access to information through self-service portals and knowledge bases
• Utilize chatbots and AI technology to provide instant and accurate answers to common questions
• Collect customer feedback to continuously improve and adjust information and communication strategies
• Offer various communication channels for different preferences, such as email, phone, social media, and messaging apps.

A  global enterprise can immediately establish a comprehensive digital presence in international markets if they solve the translation challenge and make relevant content multilingual at scale. The notion of digital-first applies both to the global enterprise and the global customer. Digital-first also means that it is much easier for a company to start engaging with a global customer base. A digital-first globalization strategy allows the enterprise to expand rapidly and be customer-centric across the world.

Never has it been more critical for a company to translate its content into many different languages to quickly establish a relationship with global customers. The sheer volume and broad scope of the task require that the translation challenge needs to be handled in a way that is efficient, streamlined, and scalable.

To achieve a comprehensive digital presence, a global enterprise needs to focus on solving the translation challenge and making all relevant content multilingual rather than trying to minimize the translated content volume. 

The modern era requirement in the digital age is to "translate everything".

Achieving this goal necessitates utilizing an adaptable machine translation technology that consistently learns and enhances itself, allowing corporations to interact with individuals in developing economies worldwide.

Source: The World in 2050 Study by PwC

The Changing Customer Requirements

The primary motivation for translating enterprise content is to enable and drive international revenue. Translation of all relevant content is necessary for building an international business. While the translation tasks related to the product packaging and basic instructions for the use of a product are mandated, there is now a growing need for translating more dynamic content that is a combination of both unstructured internal corporate content and external non-corporate content such as user impressions, reviews, and feedback from social media and influencers on the customer experience related to the product offerings of a company.

The digital landscape and audience of the 21st century present a challenging environment for modern global enterprises.

Most internet users do not search for products, but rather they seek answers to questions. If an enterprise can provide useful answers, potential customers may develop trust and possibly become advocates and product champions.

If the enterprise can help a customer understand, and educate them on the general subject domain, not just specific product-related subject matter, potential customers may begin to trust your corporate content, and if you can provide a good customer experience after they buy your product, they may even advocate using your products.

Generally, no potential customer is searching and hoping to find a website that is merely a corporate ad, filled with self-congratulating content on how great the company thinks it is and how wonderful it thinks its products are.

Unfortunately, this kind of crap content is still common on many corporate websites which are filled with marketing-speak. Social media technologies have facilitated an ongoing, real-time dialogue that has reversed the traditional direction of conversations between brands and their customers.

Consumers are now leading the conversation and brands need to listen.

Trusted, authentic user-generated content (UGC) has a significant influence on purchasing decisions, especially in online marketplaces.

New buyers trust the shared authentic experience of other real customers more than any slick pitch created by the corporation.

Customers want to know what does not work well, as much as what does, BEFORE they buy.

Marketing-speak refers to the use of clichés, buzzwords, and empty superlatives in marketing content. It is typically self-congratulating in tone and is often an abundance of empty assertions of being “the best” without meaningful context and trustworthy references.

Marketing-speak or corporate-speak is often found in press releases, brochures, white papers, and sales letters.

Changing Macroeconomic Trends

The world is also changing both in terms of geographical shifts in international trade opportunities and relative economic power. The relative importance of emerging and developing economies is growing, and any global enterprise wishing to be relevant 10 years from now needs to understand this shift.

The global economy is gradually moving away from a G7-dominated perspective. Examining these patterns clarifies why this language expansion project is crucial at this moment.  

This comparison between the G7 and E7  economies done by Pricewaterhouse Coopers (PwC) for “The World in 2050” study, provides a capsule view of these trends. PwC estimates six of the seven largest economies in the world are projected to be emerging economies in 2050 led by China (1st), India (2nd), and Indonesia (4th).

Over the coming decades, emerging economies will drive global growth. Vietnam, India, and Bangladesh could be three of the fastest-growing larger economies over this period. This growth momentum directly relates to the languages that are growing in importance. By 2050, PwC projects that the G7’s share of world GDP will fall to only around 20%, while the E7 will increase their share to almost 50% of global GDP at Purchasing Power Parity (PPPs). This means that the top 10 Indic languages which include Bengali become strategic growth opportunities.

The chart above shows estimates of three of the fastest-growing economies with expected improvements in global rankings. In contrast, three of the fastest-falling countries are Australia (from 19^th to 28^th), Italy (12^th to 21^st), and Spain (15^th to 26^th).

Some other highlights from the PwC “The World in 2050” research include:

• The top 15 fastest-growing economies over the next 30 years will all be developing and emerging market economies according to PwC projections
• Europe’s share of the world economy at PPPs could fall from around 15% to 9% by 2050
• Brazil and Mexico could be larger than Japan and Germany by 2050
• India could increase its share of world GDP at PPPs by 8% to 15% by 2050
• China’s share of world GDP at PPPs could increase to around 20% by 2050

Rising incomes in emerging markets will open up great opportunities for businesses with sufficiently flexible and patient strategies for these fast-evolving markets. As the purchasing power of an increasing portion of the population grows in these regions, so will the consumption as we see a rising middle-class emerge.

Other research also points to the rise of Asian economies, especially in South Asia and South East Asia as Chinese GDP growth also starts to slow down as the demographic impact of the One Child policy kicks in over the next two decades.

Goldman Sachs was one of the first to point out the emerging rise of the BRIC economies 15 years ago. This forecast has been more accurate for the Asian economies but in their latest research, they expect that growth will be more evenly distributed even though Asian economies will still dominate.

China, Vietnam, Uganda, Indonesia, and India are projected to be among the fastest-growing economies by 2030 according to the Harvard Growth Lab projections. Their research also factors in the ability of the country to develop complex production capabilities and finds that countries that have diversified their production into more complex sectors, like Vietnam and China, are those who will experience the fastest growth in the coming decade.

The Harvard Growth Lab has identified three growth poles using their Economic Complexity Index (ECI) which they believe is a much better predictor of economic growth prospects. 

They state that several Asian economies already hold the necessary economic complexity needed to drive the fastest growth over the coming decade, led by China, Cambodia, Vietnam, Indonesia, Malaysia, and India. In East Africa, several economies are expected to experience rapid growth, though this is driven more by population growth than gains in economic complexity, and this includes Uganda, Tanzania, and Mozambique. They also saw several Eastern European countries including Georgia, Lithuania, Belarus, Armenia, Latvia, and Romania ranking high on a per capita basis because of improvements in their ECI.

According to the IMF’s recent World Economic Outlook on Africa, five of the world’s fastest-growing economies are Angola, Ethiopia, Nigeria, Kenya, and South Africa. However, many experts say that for most of Africa, the business opportunity for global enterprises is further out into the future. Perhaps in the 10-to-20-year time frame as a properly supported (improved infrastructure, education, health services) demographic dividend starts to kick in. However, there are some exceptions as shown in the Growth Lab chart above.

This growth momentum data correlates with macroeconomic business potential, but a global business needs to determine a market opportunity match by including several other factors beyond the possibility that there is a large and growing potential customer base. Apart from product fit and basic organizational infrastructure issues needed to serve global customers across the world, several other macroeconomic factors also need to be considered. These include all of the following:

After this analysis has been done the best-fitting products and services can be presented to the new market. Market viability tests can often be done initially by creating a digital presence and window front to assess interest and better define implementation issues. 

It is at this point that the relevant content for the buyer and customer journeys and translation issues come into focus. This is what ModernMT and the service and process infrastructure at Translated are designed to address.

The ModernMT Language Expansion

One of the motivating ideas behind Translated's language expansion efforts is to help our enterprise customers reach more of the world's population. By expanding language coverage to potentially reach 6.5 billion native speakers, Translated enables companies to forge stronger connections with global users and customers by building the core translation infrastructure needed to share, communicate, and listen to new customer groups.

This initial launch and introduction of these new languages is the beginning of an evolutionary process. Translated has ensured that the current quality of MT produced by these new languages is at least equal to or better than systems produced by Big Tech companies, and has added high-quality training data resources when available to immediately improve the performance of “low-resource” languages.

The expectation and plan behind this launch are to enable these language systems to start improving immediately using the highly adaptive ModernMT technology which allows this to happen.

History shows that because of the volume of production work, greater data availability, and ongoing activity around the high-resource languages, those MT systems can reach levels of accuracy where discussions of human-equivalent performance are possible. Thus, we begin this journey with a large set of new languages. In addition to this gradual improvement effort, ongoing fundamental research will continue to increase the rate at which these systems can and will improve.

Digital leadership in emerging markets will require that enterprises translate tens of millions of words a month, to enable them to listen, communicate, understand, and share relevant information with these customers.

For the first time, 30 new languages are supported in the market, leapfrogging directly to the more powerful adaptive MT technology. Among the new languages now supported by ModernMT, are Bengali, Punjabi, and Javanese, which together with all the other newly added languages are spoken by over 2 billion people worldwide. Many of these languages have high commercial potential, enabling companies to connect and engage with some of the fastest-growing economic regions in the world.

In the customer-centric world of the future, it is also important that important tools in the localization technology stack can easily interact and connect to superior continuous learning tools like ModernMT. To further enable this we have also added API connectivity to Blackbird.io which is an Integration Platform as a Service (IPaaS). The inter-application connectivity reach will continue to improve. This will allow ModernMT to ingest data from and export translated data back to a growing set of TMS, CMS, Marketing Automation, CDP, Analytics, QE, and Storage solutions needed in modern CX-related automation deployments.

Unfortunately, many TMS systems of yesteryear still have no ability to interact with fast-evolving adaptive MT systems and trap enterprise data to create tech debt that undermines global success. Buyers need to be wary of such systems and move to open-source approaches that allow the greatest flexibility and agility.

Marco Tombetti was interviewed by Multilingual about this announcement, where he explains how data scarcity, and enabling the new languages to function with adaptive MT architecture were the two main challenges that had to be overcome to achieve this.

He also said, “We envision that this effort is merely the first step, and while 200 languages may appear substantial, it is not an extraordinary figure. We are at the beginning, and we plan to refine adaptive MT support for these languages in the coming months, as well as for numerous others.”

For a full listing of the languages supported by ModernMT: https://www.modernmt.com/api/#languages

Understanding Adaptive Machine Translation

Machine translation has been around for over 70 years and has made steady progress in tackling what many consider to be one of the most difficult challenges in computing and artificial intelligence. We have seen the approach to this challenge change and evolve, and MT has become much more widely used, especially since the advent of neural MT.

The deep learning neural net methods used in Neural MT have led to significant improvements in output quality, especially in terms of improved fluency, and have encouraged much wider use of machine translation in global business-driving applications.

While the momentum of Neural MT is well understood and recognized as a major advance in state-of-the-art (SOTA) machine translation, it is surprising that Adaptive MT has not had a greater impact.

This is especially true in the enterprise and professional translation market, where Adaptive MT can address specific and unique business needs much more effectively than alternatives. This paper explains why, even in the age of large language models, it remains a critical technology for global enterprises and professional users.

To better understand the value of Adaptive MT systems, it is useful to present a contrast to the typical generic static systems that most are familiar with.

The Typical Generic Static MT Engine

The MT engines that are generic and relatively static are intended for use by a large number of people. These engines do not alter how a phrase is translated until the next major update. Creating a generic baseline engine requires a considerable amount of effort in terms of cost, complexity, and data, which is why it is not done frequently. The diagram above displays the usual process involved in developing and producing a static engine.

A key characteristic of these static engines is that they do not evolve quickly because they require large new data sources to drive improvements, data that is not readily available, and thus generic static engines are typically updated no more than once a year.

On any given day, the major generic MT engine portals (Google, Microsoft, Baidu) allow hundreds of millions of people to translate material of interest. We have already reached the point where 99% or more of the translation done on the planet is done by computers, thanks to these generic engines.

In professional or business settings, the demands for using Machine Translation (MT) are quite particular. Generally, generic MT engines need to be tweaked and fine-tuned to cater to company- or project-specific language usage and terminology. This process of adjusting the MT engine to suit corporate requirements is known as customization or adaptation.

For example, if we consider the needs of IKEA, Pfizer, Airbnb, and Samsung, it is clear that they all have very different needs in terms of subject domain focus, style, and critical terminology and would be better served by enterprise-optimized MT than by a generic, one-size-fits-all MT solution.

Customization or adaptation of MT models to the correct terminology is necessary for successful outcomes with MT use in most enterprise or professional use settings.

The typical MT customization process using static engines is described below. The customization effort and process is a scaled-down version of the generic engine development process. Typically, it requires the collection and incorporation of enterprise translation memory relevant to the use case into the generic model via a scaled-down "training process."

This effort results in limited or coarse optimization if sufficient training data resources are available. The optimization is considered coarse because the training data available to perform the optimization is typically minuscule compared to the base data used in the generic engine. There is little value in training an engine with limited data as there would be no difference in performance from the generic baseline.

Thus, many attempts to use MT in professional settings face data scarcity problems. Limited data availability limits and reduces the potential impact of adaptation. To further complicate matters, it is usually necessary to build separate engines for each different use case, e.g., customer support, marketing, and legal would all be optimized separately.

Since many global enterprises have multiple product lines and businesses that cross multiple domains (TVs, semiconductors, PCs, home appliances) this will often result in a large number of MT engines needed to cover global business needs. As a result, it is often necessary to manage and maintain many MT engines. This management burden is often not understood at the outset when localization teams embark on their MT journey. This complexity also creates a lot of room for error and misalignment as data alignment can easily get out of sync over time.

Over time, many enterprise MT initiatives can be characterized by several problems that are common to users of these static MT systems. These problems are summarized below in order of frequency and importance:

1. Ongoing scarcity of training data: Static models require a lot of data to drive improvements. There is little value in retraining a model until new or corrective data volumes reach critical levels.
2. Tedious MTPE experience: Post-editors must repeatedly correct the same errors because these MT engines do not regularly improve, often leading to worker dissatisfaction.
3. MT model management overhead and complexity: There are too many models to manage and maintain, which can lead to misalignment errors.
4. Communication issues: Typically, between the MT development team and localization team members and translators, who have very different views of the overall process.
5. Context insensitivity: Sentence- and document-level context is typically missing from these custom models.

From a technical standpoint, static MT systems often have a significant disparity between the encoding (training) and decoding (inference) stages of model deployment, resulting in a notable disconnect. Adaptive MT, on the other hand, aims to bridge the gap between the two phases of model creation (training) and deployment (inference), thus providing more effective support to expert users such as translators and linguists.

The Adaptive MT Experience

The static MT approach makes sense for large ad-supported portals where the majority (99%+) of the millions of users will use the MT systems without attempting modification or customization.

In contrast, the adaptive MT approach makes more sense for those enterprise and professional translators who almost always attempt to modify the behavior of the generic model to meet the specific and unique needs of a business use case.

ModernMT is an adaptive MT technology solution designed from the ground up to enable and encourage immediate and continuous adaptation to changing business needs. It is designed to support and enhance the professional translator's work process and increase translation leverage and productivity. This is the fundamental difference between an adaptive MT solution like ModernMT and static generic MT systems.

“Simplicity is the ultimate sophistication”

Leonardo da Vinci

While the ModernMT adaptive MT engine also has a basic generic engine underlying its capabilities, it is designed to work instantly with any available translation memory resources and to learn instantly from corrective linguistic feedback.

This is done without any user intervention or action to "train" the system. The user simply points to any available TM and it is used if it is relevant to the translation task at hand. Thus, while many struggle to use MT in an environment where use case requirements are constantly changing, this adaptive MT system uses memories, corrective feedback, and overall context gathered from both the memories and the overall document.

As the use of MT grows in the enterprise, the benefits of an adaptive MT infrastructure continue to accrue, as the management and maintenance of the many production MT systems require nothing more than the organization of TM assets and the provision of continuous corrective feedback to drive continuous improvements in system performance.

Thus, content creators and linguistically informed users can be the primary drivers of the ongoing system evolution. Because the underlying continuous improvement process is always active in the background, there is no need for any technology process management by these users. Translation issues that may arise in widespread use, can be quickly identified and corrected by linguists without the need for support from MT technology experts.

New use cases of large-scale deployments can be rapidly deployed by targeting human translation efforts on the most relevant and statistically present content. Adaptive MT technology allows for evolutionary approaches that ensure continuous improvement.

Independent market research points to some key factors that are often overlooked by those attempting to deploy MT in professional and enterprise environments. Surveys conducted by Common Sense Advisory and Nimdzi show that most LSPs/Enterprises struggle to deploy MT in production for three key reasons:

1. Inability to produce MT output at the required quality levels. Most often due to a lack of training data needed for meaningful improvement.
2. Inability to properly estimate the effort and cost of deploying MT in production.
3. The ever-changing needs and requirements of different projects with static MT that cannot adapt easily to new requirements create a mismatch of skills, data, and competencies.

Given these difficulties, it is worth considering the key requirements for a production-ready MT system. Why do so many still fail with MT?

One reason for failure is that many LSPs and localization managers have used automated metrics to select the "best" MT system for their production needs without having any understanding of how MT engines improve and evolve. Automated MT quality metrics such as BLEU, Edit Distance, hLepor, and COMET are used to select the "best" MT systems for production work.

The scores can be helpful for MT system developers in enhancing and refining their systems. However, globalization managers who rely solely on this method to choose the "best" system may miss some noticeable limitations in selecting the most suitable MT system.

Ideally, the "best" MT system would be determined by a team of competent translators who would run directly relevant content through the MT system after establishing a structured and repeatable evaluation process. This is slow, expensive, and difficult, even if only a small sample of 250 sentences is evaluated.

Thus, automated measurements (metrics) that attempt to score translation adequacy, fluency, precision, and recall must often be used. They attempt to do what is best done by competent bilingual humans. These scoring methodologies are always an approximation of what a competent human assessment would determine, and can often be incorrect or misleading, especially with static Test Sets.

This approach of ranking different MT systems by scores based on opaque and possibly irrelevant reference test sets has several problems. These problems include:

• These scores do not represent production performance.
• These scores are typically obtained on static MT systems and do not capture a system's ability to improve.
• The results are an OLD snapshot of a constantly changing scene. If you change the angle or focus, the results would change.
• Small differences in scores are often meaningless, and most users would be hard-pressed to explain what these small numerical differences might mean.
• The score is an approximate measure of system performance at a historical point in time and is generally not a reliable predictor of future performance.
• These scores are unable to capture the dynamic evolution typical of an adaptive MT system.
• Generic, static systems often score higher on these rankings initially but this does not reflect that they are much more difficult to tune and adapt to unique, company-specific requirements.

When choosing MT systems for production use, relying solely on score-based rankings can lead to suboptimal or even incorrect choices. This approach is often used because NMT system performance is difficult to understand and can be shrouded in mystery. However, automated metrics are not always reliable and should not be the only factor considered in making purchase decisions. Using scores to justify choices may be a "lazy buyer" strategy that fails to fully account for the complexity involved in selecting the best MT system for a given purpose.

But the failure of so many LSPs with MT technology suggests that this approach may not be the best way forward to achieve production-ready and production-grade MT technology. So what criteria are more relevant in the context of identifying production-grade MT technology? The following criteria are much more likely to lead to technology choices that make long-term sense. For example:

• The speed with which an MT system can be tuned and adapted to unique corporate content. Systems that require complex training efforts by technology specialists will slow the globalization team’s responsiveness.
• The ease with which the system can be adapted to unique corporate needs The need to have expensive consulting resources or dedicated MT technology staff on hand and ready to go greatly reduces the agility and responsiveness of the globalization team.
• An automated and robust MT model improvement process as corrective feedback and improved data resources are brought to bear.
• The complexity of MT system management increases exponentially when multiple vendors are used as they may have different maintenance and optimization procedures. This suggests that it is better to focus on one or two partners and build expertise through deep engagement.
• The ability of a system to enable startup work even if little or no data is available.
• A straightforward process to correct any problematic or egregious translation errors. Many large static systems need large volumes of correction data to override such errors.
• The availability of expert resources to manage specialized enterprise use cases and trained human resources (linguists) to help prime and prepare MT systems for large-scale deployment.

It is now common knowledge that machine learning-based AI systems are only as good as the data they use. One of the keys to long-term success with MT is to build a virtuous data collection system that refines MT performance and ensures continuous improvement. The existence of such a system would encourage more widespread adoption and enable the enterprise to become multilingual at scale. This would allow the enterprise to break down the barrier of language as a barrier to global business success.

One should not assume that all adaptive MT systems follow the same technological approach. In fact, real-time, in-context adaptation can be achieved through various architectural methods. Upon closer examination of other adaptive MT solutions, it is apparent that dynamic adaptation can be accomplished using different technological strategies. However, as more buyers come to realize that the responsiveness of the MT system holds greater importance than a static COMET score on a random test set, the evaluation strategies will evolve. It will be more beneficial to assess which systems can adapt most effortlessly with minimal effort.

The ModernMT approach to adaptation is to bring the encoding and decoding phases of model deployment much closer together, allowing dynamic and active human-in-the-loop corrective feedback, that is not so different from the in-context corrections and prompt modifications we are seeing with large language models.

It is possible that in the future, as Large Language Models (LLMs) become more cost-effective, scalable, secure, and controllable, they could be utilized to enhance SOTA adaptive MT models. This could improve both core translation quality and output fluency, either as stand-alone solutions or more likely as hybrid models that work with MT purpose-focused models that are yet to come.

Although LLMs have demonstrated their effectiveness in certain high-resource languages, their performance in lower-resource languages is notably poor according to initial evaluations. This outcome is not surprising, given that LLMs are not specifically tailored for translation tasks. The challenge lies in the fact that LLMs rely on extensive data caches for each language, and the significant data volumes required for improvement are often difficult to locate. Therefore, resolving this issue will not be a quick or simple process.

In contrast, ModernMT just announced support for 200 languages that can all immediately benefit from the continuous improvement infrastructure that underlies the technology, and begin the steady quality improvement process that is described in this article.

It has become evident that real-time systems that can enhance performance and swiftly respond to informed feedback from experts are highly favored to tackle the task of large-scale automated language translation.

Once customers have experienced the advantages of dynamic adaptive systems, they are unlikely to revert to the complexities, inconveniences, and slowly improving output quality of batch-trained static MT systems.