An Overview of ModernMT V7

 Serious MT technology development requires ongoing efforts and research to continually improve the performance of systems and to address important emerging requirements as the use of MT expands. Researchers have been working on MT for over 70 years and success requires a sustained and continuing effort.

These efforts approach the goal of producing as close as possible to human-quality MT output in multiple ways, and these improvement strategies can be summarized in the following ways:

1. Acquire better and higher volumes of relevant training data. Any AI initiative is highly dependent on the quality and volume of the training data that is used to teach the machine to properly perform the task.
2. Evaluate new algorithms that may be more effective in extracting improved performance from available training data. We have seen the data-driven MT technology evolve from Statistical MT (SMT) to various forms of Neural MT (NMT) using different forms of deep learning. The Transformer algorithm which also powers LLMs like GPT-4 is the state-of-the-art in NMT today.
3. Use more powerful computing resources to dig deeper into the data to extract more learning. As the demand for translation grows with the massive increases in content and ever-expanding volumes of user-created content (UGC) it becomes increasingly important for MT to handle massive scale. Today there are global enterprises that are translating billions of words a month into a growing portfolio of languages and thus scalability and scale are now key requirements for enterprise MT solutions. Some researchers use more computing during the training phase of the MT model development process as there can be quality advantages gained at inference from doing this extra-intensive training.
4. Build more responsive and integrated human-machine collaboration processes to ensure that expert human feedback is rapidly incorporated into the core data used to tune and improve these MT engines. While the benefits gained from more and better data, improved algorithms, and more computing resources are useful, the integration of expert human feedback into the MT model's continuous learning is a distinctive advantage that allows an MT model to significantly outperform models where only data, algorithms, and compute are used.
5. Add special features that address the unique needs of large groups of users, or use cases that are being deployed. As the use of MT continues to build momentum with the enterprise many specialized requirements also emerge e.g. enforcement of specific terminology for brand integrity, profanity filters to avoid egregious MT errors, and improvement of document-specific content awareness.

All these different approaches have the goal of producing improved MT output quality and it will require progress along all of these different fronts to achieve the best results.

The ModernMT development team pursues ongoing improvements along all these fronts on an ongoing basis, and ModernMT V7 is the result of several measured improvements on many of these dimensions to provide improved performance.

As machine translation (MT) continues to evolve and expand beyond the traditional use case areas such as e-commerce, global collaboration, and customer care, those interested in the expanding future of localization are now also looking to use generative artificial intelligence (AI) and, in particular, large language models (LLMs) such as OpenAI’s GPT

Unlike typical Neural MT, LLMs prioritize fluency over accuracy. But while LLMs show promising results in improving the fluency of translations, they can also produce confabulations (hallucinations), i.e. output that is inaccurate or unrelated to the input data and thus require careful monitoring and oversight to ensure accuracy.

With the latest release of ModernMT (V7), Translated has introduced a novel technique to increase the accuracy of neural MT models, called “Trust Attention,” which can also be used to address reliability within generative AI models.

The design and implementation of Trust Attention was inspired by how the human brain prioritizes trusted sources in the learning process, linking the origin of data to its impact on translation quality.

ModernMT V7 preferentially uses the most trusted data (identified by users) and thus the highest quality and most valuable training data has the greatest influence on how a model performs. This is in stark contrast to most MT models which have no discernment of data quality and thus tend to perform using only statistical density as the primary driver of model performance.

The Trust Attention capability prioritizes its learning based on data value and importance like how humans sift through multiple sources of information to identify the most trustworthy and reliable ones. Data extracted from translations performed and reviewed by professional translators is always preferred over other data, especially unverified translation memory content acquired from web crawling, which is typically used by most MT systems today.

The development team at ModernMT considers Trust Attention to be as significant an innovation as Dynamic Adaptive MT engines. It is the kind of feature that can dramatically improve MT system performance for different use cases when properly used.

According to an evaluation by professional translators, done to validate the beneficial impact, Trust Attention alone improves MT quality by up to 42%, and by an average of 16.5% in cases across the top 50 languages. Interestingly, even many high-resource languages, such as Italian and Spanish, showed significant improvements (in the 30% range) in human evaluations.

ModernMT V7 New Features: Up to 60% Better MT Quality

ModernMT V7 is the evolution of Translated’s renowned adaptive MT system, recognized as a leader in the Machine Translation Software Vendor Assessment for enterprises by IDC Marketscape 2022, and as “the most advanced implementation of responsive MT for enterprise use” in CSA Research’s 2023 Vendor Briefing.

In addition to Trust Attention, ModernMT V7 includes several other new features that further enhance the reliability and dependability of MT output. Here are the most impactful:

• Advanced Terminology Control: Along with its ability to learn the client’s terminology from past translations, ModernMT now provides companies with self-managed glossary control to ensure brand and context-specific terminology consistency. This ability to enforce terminology has not been needed in the past because the dynamic adaptive MT technology acquires terminology very effectively even without this feature.
• DataClean AI: V7 relies on a new sanitization algorithm that identifies and removes poor-quality data to refine the training material and reduce the likelihood of hallucinations. The close examination of errors over many years has provided clues on the root causes of strange output from MT engines. This learning and related benefits also transfer to LLM-based MT engines should they become more viable in the future.
• Expanded Context: ModernMT can now leverage up to 100,000 words of document content —Four times more than GPT-4 - to preserve style and terminology preferences, providing unparalleled document-specific accuracy in MT suggestions and providing controls to solve persistent problems such as gender bias and inconsistent terminology.
• Profanity Filter: V7 masks words in translation suggestions that could be regarded as inappropriate in the target language, minimizing the possibility of cultural offenses.

The combined effect of all the improvements and innovations described above has a significant impact on the overall performance and capabilities of ModernMT.

The MT quality is now considered to be 45% to 60% better than the previous version according to systematic human evaluations.

These improvements have greatly reduced the Time to Edit (TTE) for MT suggestions. At the end of July, the aggregate TTE measured across tens of thousands of samples showed a 20% reduction, reaching a record low of 1.74 seconds. This milestone indicates an acceleration towards singularity in translation, a trend further supported by preliminary TTE data collected continuously since the 1.74 seconds record was established.

The Hallmark of the Symbiosis Between Translators and MT

ModernMT V7 is available in 200 languages and covers all the fastest-growing economies likely to emerge over the next 20 years. Its hallmark is the ability of the MT model to learn from corrections in real time, enabling a powerful collaboration between the expertise of professional translators and the speed and capacity of MT.

Thanks to this unique approach, combined with Translated’s vast community of professional translators and leading AI-enabled localization solutions (Gartner 2022), Airbnb was able to ditch the translate button and simply make multilingual content pervasive and comprehensive across the platform and become one of the top 3 global brands (Global by Design 2023).

Success stories like that of Airbnb and others, along with market research that shows the ever-growing demand for more multilingual content, have led Translated to estimate that once MT reaches what is commonly referred to as “parity with human translation” (singularity in translation), we can expect a 100-fold increase in MT requests alongside a 10-fold growth in demand for professional translations.

We are entering a new era in which significantly larger volumes of content will be translated automatically. In this scenario, professional translators play an increasingly important role, not only in guiding the MT through the adaptive process but also in ensuring that the key messages are appropriately conveyed. By engaging the best translators with the best adaptive MT, companies can now take on projects that simply weren’t feasible before.

Moving Towards LLMs for Translation

Recently, Translated conducted a large-scale study to compare the performance of the most advanced MT systems with LLMs in terms of enterprise readiness. The findings showed real potential for LLMs, particularly in terms of more fluent translation quality, and also revealed areas where improvements are needed. Based on this research, Translated believes elements of both MT systems and LLMs will be critical as we move forward, and plans to provide in-depth insights into using LLMs in translation in the coming weeks and months.

Comments by John Tinsley of Translated SRL on LLM-based Translation in November 2023:

❗ LLMs - the new default for machine translation ❗

I've seen a lot of commentary along these lines over the past few months. I've also seen a lot of well-articulated commentary, not strictly opposing this line, but with added nuance and context (a challenge on the internet!)

I wanted to offer my two cents, from being at the forefront of these developments through actually building the software, and from having many conversations with clients.

In summary, today, LLMs are not fit for purpose as a drop-in replacement for MT for enterprises.

More broadly, any general-purpose GPT application will find it super challenging to outperform a purpose-built enterprise solution that considers an entire workflow in a holistic way (note, the purpose-built solution could be GPT-based itself, but with a much narrower scope).

🧠 As a concrete example, at Translated, we've built a version of ModernMT that uses GPT-4 as a drop-in replacement for our Transformer model (while retaining the framework in ModernMT that allows us to do real-time adaptation). We've also built, and continue to test, a version of ModernMT with other open source LLMs fine-tuned for translation.

While we find that they perform well in terms of quality on some content types and some languages, it's far from unanimous across the board. And that's just quality. Other critical enterprise factors such as speed, cost, and importantly, information security, are just not there yet. Similarly, language coverage for LLMs is a challenge as there are large discrepancies in performance, particularly for content generation.

I appreciate there's a lot of downward pressure today to use AI across workflows, particularly in localization teams for translation and content creation. Let me hop on my soapbox to give you some information that might help with those conversations...

📣 If you're using MT, you're already using very advanced AI! 📣

You probably already know that the T in GPT stands for Transformer. But did you know that the Transformer was invented at Google in 2017...specifically for machine translation!? So what we're seeing today is a repurposing of that technology for a different application (generative AI) other than translation.

There will come a day, possibly soon, when it's better across the board to use LLMs for translation. When that happens, it will become the standard and people will stop talking about it. Just like when Neural MT came on the scene ~6 years ago.

When it happens, Translated will have already deployed it in ModernMT and worked out the best way for you to adapt it to your business. We already have a lot of ideas. We already have a lot of data from the testing I mentioned earlier. And in the meantime, we still have what I believe to be the most complete enterprise translation solution available.

ModernMT: A Closer Look At An Emerging Enterprise MT Powerhouse

 As one observes the continuing evolution of MT use in the professional translation industry, we see that we have reached a point where we have some useful insights about producing successful outcomes in our use of MT. From my perspective as a long-term observer and expert analyst of enterprise MT use, some of these include:

• Adaptation and customization of a generic MT engine done with expertise generally produces a better outcome than simply using a generic public MT system. 
• Working with enhanced baseline engines built by experts is likely to produce better outcomes than dabbling with Open Source options with limited expertise. While it has gotten easier to produce MT systems with open-source platforms, real expertise requires long-term exposure and repeated experimentation. 
• The algorithms underlying Neural MT have become largely commoditized and there is little advantage gained by jumping from one NMT platform to another.
• More data is ONLY better if it is clean, relevant, and applicable to the enterprise use case in focus. It can be said today that (training) data often matters more than the algorithms used, but data quality and organization is a critical factor for creating successful outcomes.
• A large majority of translators still view MT with great skepticism and see it as marginally useful, mostly because of repeated exposure to incompetently deployed MT systems that are used to reduce translator compensation. Getting active and enthusiastic translator buy-in continues to be a challenge for most MT developers and getting this approval is a clear indicator of superior expertise.
• Attempts to compare different MT systems are largely unsuccessful or misleading, as they are typically based on irrelevant test data or draw conclusions based on very small samples.
• A large number of enterprise use cases are limited by scarce training data resources and thus adaptation and customization attempts have limited success.

I have been skeptical of the validity of many of the comparisons we see of MT systems produced by LSPs and "independent" evaluators nowadays, because of the questionable evaluation methodologies used. The evaluators often produce nice graphics but just as often produce misleading results that need further investigation. However, these comparative evaluations of different MT systems can still be useful to get a rough idea of the performance of generic systems of these MT vendors. Over the last few years ModernMT has been consistently showing up amongst the top-performing MT systems in many different evaluations, and thus I decided to sit down with the ModernMT team to better understand their technology and product philosophy and understand what might be driving this consistent performance advantage. The level of transparency and forthcoming nature of the responses from the ModernMT team was refreshing in contrast to other conversations I have had with other MT developers.

The MT journey here began over 10 years ago with Moses and Statistical MT, but unlike most other long-term MT initiatives I know of, this effort was very translator-centric right from its inception. The system was used heavily by translators who worked for Translated and the MT systems were continually adapted and modified to meet the needs of production translators. This is a central design intention and it is important to not gloss over this, as this is the ONLY MT initiative I know of where Translator Acceptance is used as the primary criterion on an ongoing basis, in determining whether MT should be used for production work or not. The operations managers will simply not use MT if it does not add value to the production process and causes translator discontent.  Over many years the ongoing collaboration with translators at ModernMT has triggered MT system and process development changes to reach the current status quo, where the MT value-add and efficiency is clear to all the stakeholders. The long-term collaboration between translators and MT developers, and resulting system and process modifications are a key reason why ModernMT does so well in both generic MT system comparisons, and especially in adapted/customized MT comparisons.

Thus, translators who actively use the ModernMT platform do so most often through MateCat, an open-source CAT tool that ties together MyMemory (a large free-access shared TM repository with around 50 billion words in it) together with ModernMT or other MT platforms. MT is presented to translators as an alternative to TM on a routine basis, and corrections are dynamically and systematically used to drive continuous improvements in the ModernMT engines. Trados and other CAT tools are also able to seamlessly connect to the ModernMT back-end but these systems may see less immediate improvements in the MT output quality. However, this has not stopped ~25,000 downloads of the ModernMT plugin for Trados on the SDL Appstore. Translators who do production work for Translated are often given a choice of using Google instead of ModernMT but most have learned that ModernMT output improves rapidly from corrective feedback and that collaborative input is also easier, and thus tend to prefer it as shown in the surveys below. Over the years the ModernMT product evolution has been driven by changes to identify and reduce post-editing effort rather than optimizing BLEU scores as most others have done. 

In contrast to most MTPE experiences, the individual translator experience here is characterized by the following:

• A close and symbiotic relationship between a relevant translation memory and MT, even at the translator UX level
• An MT system that is constantly updated and can potentially improve with every single interaction and unit of corrective feedback
• Immediate project startup possibilities as no batch MT training process is necessary
• Translator control over all steering data used in a project means very straightforward control over terminology and term consistency, mirroring the latest TMs and linguistic preferences
• Corrective feedback given to the MT system is dynamic and continuous and can have an immediate impact on the next sentence produced by the MT system
• One of very few MT systems available today that can provide a context-sensitive translation 
• Measurable and palpable reduction in post-editing effort and translator UX compared to other MT platforms
• Continuing free access to the CAT tool needed to integrate MT with TM, and interact proactively with MT with the option to use other highly regarded CAT tools if needed. 

 

Memory here refers to user input data TM and glossaries to tune the generic system to the needs of the current translation task

Instance-Based Adaptation

ModernMT describes itself as an "Instance-Based Adaptive MT" platform. This means that it can start adapting and tuning the MT output to the customer subject domain immediately, without a batch customization phase. There is no long-running (hours/days/weeks) data preparation and pre-training process needed upfront. There is also no need to wait and gather a sufficient volume of corrective feedback to update and improve the MT engine on an ongoing basis. It is learning all the time. 

Rapid adaptation to customer-unique language and terminology is perhaps the single most critical requirement for a global enterprise, and thus this is an optimal design for enterprises that works optimally with their specialized and unique content. This is also true for LSPs too, for that matter. ModernMT can adapt the MT system with as little as a single sentence, though the results are better if more data is provided. The team told me that 100K words (10-12,000 sentences)  would generally produce consistently good results that are superior to any generic engine. The long-term impact of this close collaboration with translators who provide ongoing corrections, feedback on critical requirements to improve efficiency and process workflow, and careful acquisition of the right kind of data, results in the kind of relative performance rankings that ModernMT now regularly sees as a matter of course. One might even go so far as to say that they have built a sustainable competitive advantage. 

I have always felt that a properly designed Man-Machine collaboration would very likely outperform an MT design approach that relies entirely on algorithms and/or data alone. We can see this is true from the comparative results of the large public MT portals who probably have 100X or more of the resources and budget that ModernMT does. The understanding of the translation task and resulting directives that ongoing translator feedback brings to the table is an ingredient that most current MT systems lack. Gary Marcus and other AI experts have been vocal in pointing out that machine learning and data alone is not the best way forward and more human steering and symbolic knowledge is needed for better outcomes.  

   

Special Features

ModernMT is a context-aware machine translation product that learns from user corrections. There has recently been growing interest in the MT research community to bring a greater degree of contextual awareness to MT systems and ModernMT has also been investigating implementing capabilities around doing this. The current production version has an implementation of this already, and this feature continues to evolve in speed, efficiency, and capability.

The ModernMT Context Analyzer analyzes an entire document text to be translated in milliseconds before producing a translation. This analysis seeks out and identifies the distinctive terminology and intrinsic style of the document. This information is then used to automatically select the most suitable private translation memories loaded by the user for that particular document. This results in the engine selecting the translation memory inventory that best reflects the right terminology and writing style. It is precisely this inventory that the MT engine leverages to customize the output in real-time, for each and every sentence of the document.

As translators at Translated working with ModernMT regularly have the ability to compare the MT output with that of Google Translate, the developers monitor translator preferences on an ongoing basis. This ensures that translators are always working with the MT output that they find most useful and that developers understand when their own engines need to be improved or enhanced. The following charts are based on feedback from translators during production work and show a very definite preference for the rapidly improving ModernMT engine output. This preference is seen in internal translator assessments working in production mode rather than just a selective test set, and this has also been confirmed by independent third-party assessments with both automated scores and human evaluations. They all consistently show that ModernMT customizations regularly outperform most others in independent comparative evaluations. The forces driving this superior performance are the result of design philosophy and long-term man-machine collaboration that cannot be easily replicated by others.

 

Recent comparative assessments done by independent third parties also confirm this preference using different evaluation methods that include both human and automated metrics as shown below. It is not unreasonable to presume that this performance advantage will remain intact for at least the short term.

Data Privacy

In response to a question on data privacy, Davide Caroselli, VP of Product, ModernMT responded: "Any content sent to ModernMT, whether a “TMX” memory or an MTPE correction from a professional translator, is saved in the user’s private data area. In fact, only you will be able to access your resources and make ModernMT adjust to them; in no way will another user be able to utilize that same inventory for his/her system, nor will ModernMT itself be able to use those contents, other than to exclusively offer your personalized translation service.

In addition, ModernMT uses state-of-the-art encryption technologies to provide its cloud services. Our data centers, employee processes and office operations are ISO 27001:2013 certified." 

On-Premise Capabilities

While the bulk of the current ModernMT customer base works with the secure cloud deployment, the team at ModernMT has also defined a range of on-premise deployment capabilities for those enterprises that need the security, control, and assured data privacy needs that characterizes some National Security, Financial, Legal, and Healthcare/Pharma industry requirements. The open-source foundations of much of the ModernMT infrastructure should make it particularly interesting to US Government Intelligence and Law-Enforcement agencies seeking large-scale multilingual data processing capabilities for eDiscovery and Social Media Surveillance applications.  

Given that ModernMT is a continuous learning MT platform that learns with each correction, dynamically, there is a requirement for more GPU infrastructure than some other on-premise solutions in the market. However, there is a strong focus on computational efficiency to minimize the IT footprint needed to deploy it on-premise, and based on information provided to me, their capabilities are quite similar to competitive alternatives both in terms of hardware requirements and software pricing. Hardware costs are linked to throughput expectations with more hardware required for high throughput requirements. As with most machine learning-intensive capabilities, only enterprises with competent IT teams could undertake this as an internal deployment, and most LSPs and localization departments will see a lower total cost of ownership with the cloud deployment. 

Enterprise Readiness  

As ModernMT has evolved from the localization world it is already optimized for MT use cases where there is a significant need for a machine-first human optimized approach. More and more we see this model as being a preferred approach for the exploding volumes of localization content.  The Localization Use Case is possibly the most challenging MT use case out there, as it requires very high-quality initial output that translators are willing to work with where it can be proven that the MT enhances productivity and efficiency. Localization use cases demand the highest quality MT output from the outset compared to eDiscovery,  social media surveillance, eCommerce,  customer service & support use cases which are all more tolerant of lower MT output quality on much larger volumes of data. Very few MT developers have had success with the high-quality and rapid responsiveness needs of the localization use case and many have tried and failed. This is why LSP adoption of MT is so low. ModernMT's success with the challenging localization use case, however, positions them very well for other MT use cases as their growing success with these other use cases proves.

The ASTW case study illustrates the success of ModernMT in Intellectual Property (Patents) and Life Science focused translations, where the ease of customization for complex terminology and morphology, the ability to learn continuously and quickly from corrective feedback, and superior MTPE experience compared to other MT solutions has quickly made it a preferred solution. 

"ModernMT is currently our favorite MT engine, especially in patent translations and in the Life Science sector, because it proves reliable, efficient, qualitatively better than its competitors, easily customizable and advantageous in terms of cost."

Domenico Lombardini, CEO ASTW

We see that eCommerce giants understand the positive impact of translating huge volumes of catalog and user-generated CX content has on driving international revenue growth with the examples of eBay, Amazon, and Alibaba. ModernMT is now the MT engine driving the multilingual expansion of Airbnb web content and is translating many billions of words a month for them. User-generated content influences future customers, and there is great value in translating this content to drive and grow international business. Interestingly ModernMT began this initiative with almost no translation memory and had to perform specialized heuristic analysis on Airbnb content to build the training material.    

ModernMT has reached this point with very little investment in sales and marketing infrastructure. As this builds out and expands I will be surprised if ModernMT does not continue to expand and grow its enterprise presence, as enterprise buyers begin to understand that a tightly integrated man-machine collaborative platform that is continuously learning, is key to creating successful MT outcomes. I am aware that many other high-profile enterprise conversations are underway, and I expect that most enterprise buyers who evaluate the ModernMT platform will very likely find it is a preferred, cost-efficient way to implement large-scale MT solutions in a way that dramatically raises the likelihood of success. 

Future Directions

Davide also mentioned to me that his team is very connected to the AI community in Italy, and have been experimenting with GPT-3 and BERT, and will continue to do so until clear value-added applications that support and enhance their MT product emerge. ModernMT has a close relationship with Pi Campus and thus has regular interaction with luminaries in the AI community e.g.  Lukasz Kaiser who will be speaking about improvements in the Transformer architecture later this month.

The team also showed me demos of complex video content that had ModernMT-based automated dubbing from English to Italian injected into it. Apparently, Italy is one of the largest dubbing markets in the world. Who knew? Since my wife speaks Italian, I showed her some National Geographic content on geology, filled with complex terminology and scientific subject matter that she was shocked to find out had been done completely without human modification. The Translated team is exploring Speech Translation and I expect that they will be quality leaders here too.

ModernMT will continue to expand its connectivity to other translation and content management infrastructure to make it easier to get translation-worthy data in and out of their environment. They also continue to explore ways to make the ModernMT continuous training infrastructure more computationally efficient so that it can be more easily deployed on smaller footprint hardware. 

I expect we will see more and more of ModernMT on the enterprise MT stage from now on, as buyers realize that this is a significantly improved next-generation MT solution that is more likely to produce successful outcomes in digital transformation-related enterprise use scenarios. The ModernMT approach reduces the uncertainty that is so common with most MT-related initiatives and does it so seamlessly that most would not realize how sophisticated the underlying technology is until they attempt to replicate the functionality.

On a completely different note, I participated some months ago in responding to a question posed by  Luca Di Biase, the Imminent Research Director. He posed this same question to many luminaries in the translation industry, and also to me. The question has already triggered several discussions on Twitter.

“Is language a technology or a culture?”  

My response was as follows, but I think you may find the many other responses more interesting and complete if you go to this link or look at some of the other Twitter comments.

It is neither. Language is a means of communication and an information-sharing protocol that employs sounds, symbols, and gestures. Language can sometimes use technology to enable amplification, extend the reach of messages, and accelerate information and knowledge sharing. Language can create a culture when shared with(in) a group and used with well-understood protocols and norms. Intercultural communication can also mean cross species, e.g., when communicating with dogs and horses.

Translated's Research Center has just released the Imminent publication which has a distinctive style coupled with interesting content, that I think most in the language industry would find compelling and worth a close look.