tag:blogger.com,1999:blog-6748877443699290050.post8966944659587365376..comments2024-03-27T23:43:31.674-07:00Comments on eMpTy Pages: Understanding Machine Translation Quality: A ReviewKirti Vasheehttp://www.blogger.com/profile/16795076802721564830noreply@blogger.comBlogger10125tag:blogger.com,1999:blog-6748877443699290050.post-13185483402073279012021-10-22T11:19:17.307-07:002021-10-22T11:19:17.307-07:00Very useful recap, many thanks!Very useful recap, many thanks!Blanca Pinero-Canovashttps://www.linkedin.com/in/blanca-pinero-canovas-422b22/noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-91919756172456474932021-10-22T11:18:02.404-07:002021-10-22T11:18:02.404-07:00Systematic and repeatable human assessment of MT o...Systematic and repeatable human assessment of MT output is a critical driver of progress with MT and tools like yours will bring much needed structure and experimental repeatability to this process. I look forward to learning more about your product.Kirti Vasheehttps://www.blogger.com/profile/16795076802721564830noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-7228181514441996132021-10-22T11:17:03.106-07:002021-10-22T11:17:03.106-07:00Thanks for properly positioning human insight into...Thanks for properly positioning human insight into MT quality, Kirti! From ContentQuo’s work with some of the best enterprise&LSP Machine Translation teams in the world, we see how relentlessly they invest into continuous, multi-faceted, carefully instrumented processes for human assessment of MT output in order to constantly be ahead of the game when tracking the evolution of their MT engine mix. Even for fast-feedback scenarios like eCommerce where raw MT quality converts almost directly into cash, human evaluation seems essential to understand & preserve the quality drivers of high conversion. Not to mention the majority of scenarios, where fast meaningful business ROI is nigh impossible to obtain!..Kirill Solovievhttps://www.linkedin.com/in/kirillsolovievnoreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-30122334214974352252021-10-22T11:11:04.462-07:002021-10-22T11:11:04.462-07:00Interpreting what a score means is difficult, and ...Interpreting what a score means is difficult, and it is even harder to understand what small improvements in a score may actually mean.<br /><br />Having a "representative" reference sample is an issue with all these metrics. Quite possibly BLEU remains popular because we understand what a score might mean and enough of us have learned to do this "properly" after doing it a thousand times.<br /><br />This is not true for hLepor, Comet, ChrF -- few have the experience to properly interpret them. It may take years for widespread understanding to happen so in the meantime BLEU is still better.Kirti Vasheehttps://www.blogger.com/profile/16795076802721564830noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-92144106533100380052021-10-22T11:09:55.442-07:002021-10-22T11:09:55.442-07:00Kirti, IMVHO, the problems with BLEU are the scale...Kirti, IMVHO, the problems with BLEU are the scale, which does not allow for understandable increments, and the reference sample, but I can see that, despite the efforts, the other metrics are not that better.<br />In my previous comment, I forgot to mention a point I have long tried to catch the attention of researchers on: Assessing the source. No quality rate can be really understood if it does not take the quality of the source content into consideration. No machine can perform adequately with poor content, while an experienced HT could: The difference is all in the ability to guess. ;-)Luigi Muziihttps://www.linkedin.com/in/luigimuzii/noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-91025731652388266182021-10-22T11:08:05.663-07:002021-10-22T11:08:05.663-07:00Amazingly BLEU is still the most commonly used mea...Amazingly BLEU is still the most commonly used measure even at MT Summit in August 2021. Hardly obsolete. The reason I think that this is so is that it is well understood and it is not really that much worse than hLepor, Comet and others. There is very little to be gained for all the hassle/effort incurred in using the others.<br /><br />The other things you mention are very interesting and probably deserve a closer examination that I hope to do in futureKirti Vasheehttps://www.blogger.com/profile/16795076802721564830noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-56218895236872695992021-10-22T11:06:19.659-07:002021-10-22T11:06:19.659-07:00Well, Kirti, as you know, price is a fundamental p...Well, Kirti, as you know, price is a fundamental parameter, so a price/performance ratio is crucial. Also, the response to trimming the so-called learning slope. I may also add the resilience to variances in training data and, last but not least, the predictability of results, although I'm not sure how this last parameter can be measured.<br />Anyway, I'm fairly surprised that still no thought is given to subjectivity in assembling the assessment data, especially with BLEU, which I now considered fairly obsolete and inadequate.<br />All said, I'm quite amused by the amount of attention given to human assessment, but given the churning rate of pointless discussion on PEMT and quality at large in this industry I must say I'm not surprised.Luigi Muziihttps://www.linkedin.com/in/luigimuzii/noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-51434126419913945382021-10-22T11:05:03.782-07:002021-10-22T11:05:03.782-07:00Thank you Luigi for your comment
What other varia...Thank you Luigi for your comment<br /><br />What other variables should be added to get a more complete equation?Kirti Vasheehttps://www.blogger.com/profile/16795076802721564830noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-79505618543751158912021-10-22T11:03:16.181-07:002021-10-22T11:03:16.181-07:00However unfortunate for championing a commercial p...However unfortunate for championing a commercial platform that's by no means better than competitors, at least this is a very honest piece on MT "quality" assessment. Many variables are still out of the equation, though.Luigi Muziihttps://www.linkedin.com/in/luigimuzii/noreply@blogger.comtag:blogger.com,1999:blog-6748877443699290050.post-4924299139092445162021-10-22T11:00:07.828-07:002021-10-22T11:00:07.828-07:00Such a good article. Thank you!Such a good article. Thank you!Sergio Pelinohttps://www.linkedin.com/in/sergiopelino/noreply@blogger.com