tag:blogger.com,1999:blog-6748877443699290050.post8643026364242341786..comments2024-03-29T00:21:17.976-07:00Comments on eMpTy Pages: The Ongoing Neural Machine Translation MomentumKirti Vasheehttp://www.blogger.com/profile/16795076802721564830noreply@blogger.comBlogger9125tag:blogger.com,1999:blog-6748877443699290050.post-75838607107644723452017-07-30T04:05:20.354-07:002017-07-30T04:05:20.354-07:00We've seen "hybrid", we've seen ...We've seen "hybrid", we've seen "workaround", now I'm going to bring in the term "pragmatic" in reference to my use of NMT.<br />Being a translator-cum-coder who - in addition to licensing access to our Dutch-English/English-Dutch neural MT engines to translators - earns a living as an occasional LSP or an insourced project manager, I'm always looking for the most cost-effective way to complete a project by means of automated translation. Like Systran (and I have some fond memories of Peter Toma) whose commitment to NMT is borne out by the energy they devote to the OpenNMT project, I have enough evidence in the language pair of which I have specialist knowledge to come to the conclusion that NMT is probably the best way forward. "Forward" is the operative word here as we are certainly not there yet.<br />I also use a variety of approaches to MT to meet my clients' needs, and have been known to run a complete job through RBMT & NMT and pick the best bits from each. In practice, translation jobs can be cross-domain. I am currently working out a strategy to handle a vast project that is multi-domain in terms of content. Much of it involves documents containing legal and administrative jargon, but there are also documents containing technical specifications, while another couple of documents contain lists of protected flora and fauna. I'm very happy with the results of my test runs and will be using NMT to process the legal and adminisatrative documents. I know from experience that our RBMT + PBSMT set-up will currently cope better with the technical documents and there will certainly be no time to do specialisation of our baseline NMT engine. As for the lists of protected flora and fauna, they are likely to go to a human translator! All the translated documents will find their way into a translation memory program where they will be reviewed by a professional translator. <br />"End-to-end" neural machine translation of every kind of document is a noble research goal but in daily translation practice NMT is just one of the tools in the LSP's armoury. Our translation server logs show me that one specialist translator accessed our Neural MT Dutch-English server via our memoQ plugin around 6 hours a day last week, although I know that our NMT model has not been trained for her specific subject area. Whether she accepted, rejected or modified the MT proposals I have no idea, but she definitely kept coming back for more so something was useful. The best translation memory programs allow the display of results from numerous sources, and another translator has specifically asked me to display RBMT and NMT results alongside each other so that he can choose the best proposal (or reject them both).<br />Customers want translations of the agreed quality in the required time frame at the specified cost. Whether these translations are hand-crafted by a hundred Cistercian monks toiling in their dimly lit cells or shot down fiber optic by an array of GPU's, they don't give a simian's posterior. RBMT, SMT, NMT and HT are all welcome at my party.<br />Terence Lewishttp://www.mydutchpal.comnoreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-66304180263651378602017-07-27T23:38:02.155-07:002017-07-27T23:38:02.155-07:00Kirti, as it relates to this discussion, I offer m...Kirti, as it relates to this discussion, I offer my updated comment from another other article to relate to this article.<br /><br />Our customers are translators who use SMT themselves. This is a feat that many in the MT community said was impossible. In their world, the only test that matters is how much effort MT increases or reduces their work today... not in some uncertain future.<br /><br />BLEU is a "closeness" score much like fuzzy matches. Fuzzy reports how closely the source segment matches a source segment in the TM. We use the BLEU score post-facto to report how closely the MT suggestion matches the translator's finished work. In the graphs above, the BLEU scores jump roughly 10% from SMT to NMT. Yet, would a translator be satisfied with source fuzzy closeness scores in these ranges?<br /><br />We embedded SMT into a desktop application without a requirement for big data. The translator converts his own TMs to make an SMT engine that serves only him. In this use-case, the translator regularly experiences BLEU scores in the high 70's to high 80's between the MT suggestion and their final work. Furthermore, the percent of ed-0 segments jumps to 25%, 30% and sometimes 45%. We've never seen the percent fall below 25% with a translator's personal TMs for any language pair. It's the same fundamental SMT technology, totally different user experience.<br /><br />For us, MT -- of any kind -- is not about the quality of the translation. The translator is responsible for the quality. For us, MT is about enhancing the individual translator's experience. A reduced workload is one way to measure their experience. The MT output's quality is another aspect of that experience, and there are many other aspects to consider.<br /><br />NMT algorithms will mature. Hardware support is also improving as Intel embeds GPU co-processors into the CPU, much like they did with floating-point co-processors in the early-1990's. When NMT technology is viable in a desktop application use case, it is certainly possible (probable?) that it could push the closeness scores even higher than we experience with today's desktop SMT. As of today, however, a change to NMT would degrade our customers' experience and that's not acceptable.tahoarhttps://www.blogger.com/profile/06893656133001786619noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-90232076825494026562017-07-27T22:57:22.178-07:002017-07-27T22:57:22.178-07:00Andy, I agree that there are advantages. That'...Andy, I agree that there are advantages. That's why I asked my question. I expect Manuel's answer could demonstrate one of them. We tested NMT vs SMT cooperation with Terence Lewis starting from virtually identical NL-EN corpora and the exact same test set of 2300 segments. Slate (SMT) generated 219 edit-distance zero (9% ed-0) segments while his NMT engine generated 25 (1% ed-0). To our customers, the ed-0 segments are important because they represent the engine's capacity (potential) to reduce the translator's work to a cognitive exercise with zero mechanical effort. There are other real and consequential benefits of SMT-over-NMT in this bleeding-edge environment. I think Chris Wendt gave an excellent description of their current relationship in a podcast a few months ago. Paraphrasing, he said that big-data SMT has reached its quality limits with potential for improvement. NMT picks up at about where SMT is today and holds the promise of continued improvement in the future.tahoarhttps://www.blogger.com/profile/06893656133001786619noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-8562591532899576352017-07-27T11:38:59.044-07:002017-07-27T11:38:59.044-07:00Andy,
Firstly, I have never claimed that this blo...Andy,<br /><br />Firstly, I have never claimed that this blog is anything but a forum for opinions (both mine and others), so it is possible that we may use words that don’t belong in scholarly papers. So it goes. I fully understand that I am wrong sometimes, and generally do admit it when I see that is so e.g. when NYT pointed out that Schuster (Google) was not happy with the exuberant marketing claims about GNMT.<br /><br />I am not sure how you determined that our statements imply there is scorn when SMT is compared with NMT. It is clear to all of us that we are at a transition stage where many SMT systems do indeed outperform NMT. But we are also aware that there are a growing number of cases where given exactly the same data resources, NMT is CLEARLY outperforming many mature and carefully refined SMT systems. At this point in time both are equally valid, (as is RBMT) and perhaps, in anything less than total customization, maybe SMT is superior for domain adapted systems. We don’t have enough data to say with any great certainty. Systran uses them all (RBMT, SMT, (RBMT+SMT hybrid) and NMT) to meet client requirements but are heavily focused on moving completely to NMT since they have enough evidence to convince themselves that NMT is the best way forward. <br /><br />The term “workaround” is in quotes because it is exactly the word that is often used by vendors. I think that we may also differ on what “hybrid” means. To me it would have to be at the training level to truly count as hybrid. Otherwise, it is what I would consider a workaround. My issue with “hybrid” is its use for marketing purposes, when in fact we are talking about a workaround.<br /><br />It is also my sense that the major MT developers who are “all in” are using pure NMT strategies (which make more sense in a deep learning environment) rather than using the techniques that worked in SMT but make less sense with NMT models. It is possible that this will take longer, but I would expect that it is also likely to produce more robust solutions.<br /><br />Finally, it is also my sense that you are defending something that I am not attacking. You may wish to look at some of my older posts on NMT to see that my view is not so different from yours.<br />Kirti Vasheehttps://www.blogger.com/profile/16795076802721564830noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-72508552135720857972017-07-27T05:59:49.481-07:002017-07-27T05:59:49.481-07:00This article makes some good points, but loses cre...This article makes some good points, but loses credibility by being unduly scornful of the advantages of SMT compared to NMT. Putting scare quotes around "evidence" where SMT outperforms NMT, as well as around "hybrid NMT" is unnecessarily pejorative. There is plenty of evidence that SMT will continue to beat NMT, especially in resource-poor language pairs, and criticising (all) "hybrid" models of NMT/SMT as "trigger[ing] blips on my bullshit radar" betrays a lack of scholarliness, and ignores the many cases which have shown hybrid models to work better than NMT (or SMT). I'm surprised at you, Manuel, I have to say ...<br />Andy.Andy Wayhttps://www.linkedin.com/in/andy-way-0a8a0238/noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-72771633163751549762017-07-27T02:40:30.112-07:002017-07-27T02:40:30.112-07:00Manuel, simple question. Your dev set is 2,000 seg...Manuel, simple question. Your dev set is 2,000 segment pairs. You calculated TER and WER scores for each segment and report their cumulative scores for both SMT and NMT. Can you please report the simple count of how many dev pairs scored TER=0/WER=0? That is, these are predictive models. So, how many times in 2,000 segments did the respective engines exactly predict the expected result?tahoarhttps://www.blogger.com/profile/06893656133001786619noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-52125239560660756202017-07-26T03:29:19.297-07:002017-07-26T03:29:19.297-07:00Kirti, I'm afraid I haven't made myself cl...Kirti, I'm afraid I haven't made myself clear.<br />To make "the right kind of data available" a specific know-how is needed that is not provided to translation students, and often to language students either.<br />At the recent TAUS Industry Summit, Jaap van der Meer was impressed with what he named "the Bodo dilemma" summing up the frustration at not finding talents willing to work with the fantastic localization technology suites available today. Those talents should obsviously be "young," while no one in the industry is still doing much to educate and train young talents, so certain skills can be found only in experienced people who acquired them on their own, possibly out of curiosity (or despair, possibly) and are not interesting for the labor market. And we both know why.<br />Basic TMs are what most translation industry players still ask for, especially LSPs. This means that they are making the industry doomed.<br />I don't think that translation is going to vanish, while most translation-related professions will.<br />Please, don't get caught in the "premium market hoax." There is no "premium market;" there could be highly-remunerative customers, in few vertical segments (this is the key concept,) but they will increasingly be fascinated and won by MT.<br />This will be NMT, of course, but definitely not now. Not even in 2018, maybe in five years from now or more. And we both now that nowadays this is a very long time. May Keynes forgive me for this.<br />Marketing is not the right side to view the future from, that's why I don't think NMT is or will be the killer application any time soon. And the Big Ones will make a clean sweep of most MT companies.Luigi Muziihttps://www.blogger.com/profile/11617962606487603486noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-62479232566548861922017-07-25T10:56:38.120-07:002017-07-25T10:56:38.120-07:00Luigi, I am not sure how you concluded that I am i...Luigi, I am not sure how you concluded that I am in agreement with Andrew's post about the language data market. I am more than skeptical about "a language data market" because the notion of value around TM data is so vague and ambiguous as to make it an impossibility. With data, One man's meat is another man's poison.<br /><br />However, I am in complete agreement with Norvig on the importance of better data analysis, management and manufacturing tools. Especially language tools that go beyond TM and TMS, and assist in making the right kind of data available. Data that supports a growing range of machine learning initiatives. Basic TM is an increasingly low value proposition.This broader scope data has been completely ignored by the translation industry and this may result in "the industry" becoming much less relevant in the market of business driven translation work in future. <br /><br />For those who have not realized yet, free, generic MT is already a killer application that does more than 99% of the translation done on the planet. IMO NMT will likely take it to 99.9%<br /><br />This does not mean that HT disappears -- as BLS statistics show, the rise of widely used MT is also accompanied by a doubling of the employment in translation. However, I think there will definitely be a bi-modal market - a premium market where real SME and deep translation competence matters and a bulk volume market where price is the key driver. Kirti Vasheehttps://www.blogger.com/profile/16795076802721564830noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-7572410561532237952017-07-25T03:09:05.654-07:002017-07-25T03:09:05.654-07:00The problem with language data is that most lingui...The problem with language data is that most linguists (especially translators and even terminologists) are still not trained to deal with it, to handle it, to process it, to understand it. This is a serious issue that is braking the so-called translation industry and undermining all translation-related professions.<br /><br />I'm afraid I can't agree with you on Andrew Joscelyne's vision of the unescapable success of a language data market. Even Microsoft has recently abandoned the idea of a data market for its Azure platform. For a data market to succeed, its prospect players should be aware of the importance of data and capable of estimating its value. Unfortunately, as for too many corporations with people, LSPs (and their customers too) look at (their) data as an asset, maybe a valuable asset, but they don't treat it as such. And a major fact for consolidation in the industry is the acquisition of a company also for its data, even though I'd rather say for its resources as a whole (including the customer and the vendor base.)<br /><br />All this should partly explain why "the state of the aggregate language data within most LSPs is pretty bad, maybe even atrocious." Of course, this also depends on the insufficient grasping of technology of too many LSPs. In fact, the exploitation of most translation technologies, in most cases, is delegated to vendors (i.e. freelancers.) The conceptual backwardness of many TMSs is a reflex of this attitude. There is only one company, in my own experience, that pays attention to project data to run the kind of estimates, measurements and analysis needed today.<br /><br />Also, I've found a few more pitfalls in the TAUS article: The requisites for a "conceptual shift from Big Data to language data" are missing, and would possibly be late now, even because language data has never been "big" in the Big Data sense. Also, to help this shift the many language data repositories offering versions of the same data sets should consolidate, but this requires time and investments that I really can't see now. Another point is the assumption that data is not a commodity. Well, a data market place would make it exactly that.<br /><br />Finally, what I find really interesting in Manuel Herranz's reasoning is the idea that NMT is expected to be the killer application. Let me remain skeptical about this.Luigi Muziihttps://www.blogger.com/profile/11617962606487603486noreply@blogger.com