It has been sometime since I first wrote a blog
post about MT humor primarily because I really have not been able to find anything worth the
mention, until now, and except for some really lame examples about how MT mistranslates (sic) I have not seen much to laugh heartily at. It seems a group of
people on the web have discovered the humorous possibilities of MT in translating song
lyrics which might be difficult even for good human translators. (It really seems
strange to be saying “human translator”.)
I should point out that in all these recent cases one does have to work at degrading the translation quality by running the same text through a whole sequence of preferably not closely related languages.
It has often surprised me that there are some in the MT industry who use “back translation” as a way to check MT quality, as from my vantage point it is an exercise that can only result in proving the obvious. MT back translation by definition should result in deterioration since to a very great extent MT will almost always be something less than a perfect translation. This point seems to evade many who advocate this method of evaluation, so let me clarify with some mathematics as math is one of the few conceptual frameworks available to man where proof is absolute or pretty damned certain at least.
If one has a perfect MT system then the Source and Target segments should be very close if not exactly the same. So mathematically we could state this as:
But in real life where humans play on the internet
and you have DIY MT systems being used to determine what MT can produce, the
results are less likely to equal 1 which is perfect as shown in the example
above.
So lets say you and I do a somewhat serious evaluation of the output of various MT systems (each language direction should be considered a separate system) and find that the following table is true for our samples by running 5,000 sentences through various MT conversions and scoring each MT translation (conversion) as a percentage “correct” in terms of linguistic accuracy and precision.
So if we took 1,000 new sentences and translate them with MT we should expect that the percentage shown above would be “correct” (whatever that means). But if we now chain the results by making the output of one, the input of the other, we will find that results are different and and get continually smaller e.g.
So with MT we should expect that every back test
will result in a lower or degraded results as we are multiplying the effect of two different
systems. Since computers don’t really speak the language one cannot assume that
they have equal knowledge going each way and if you provide a bad source from system A to system B you
should expect a bad target as computers like some people, are very
literal.
So now if we take our example and run it through multiple iterations we should see a very definite degradation of the output as we can see below.
So if you are trying to make MT look silly you have to run it through multiple iterations to get silly results. It would help further if you chose language combinations like EN to Japanese to Hindi to Arabic as this would cause more rapid degradation to the original English source. Try it and share your results in the comments.
So here we have a very nicely done example and you should realize it takes great skill for the lead vocalist to mouth the MT words as if they were real lyrics and still maintain melodic and rhythmic integrity so be generous in your appreciation of their efforts.
This video shows very effectively how using multiple languages very quickly can degrade the original source as you can see when they go to 64 languages. Somehow words get lost and really strange.
And here is one from a vlogger who really enjoys the effect of multiple rounds of MT on a songs lyrics. She is a good singer and is able to maintain the basic melody without breaking into a smile so I found it quite enjoyable and I would not be surprised that some might believe that these were indeed the lyrics of the song. She has a whole collection of recordings and has what I consider are high production values for this kind of stuff.
And she produces wonderful results on this Disney classic "When you paint the colors of your air can" which used to be a favorite of my daughter. I actually think the song from the Little Mermaid is much funnier and was done by just running it only through four iterations in Google Translate, but since I could not embed it here directly I have given the link.
Here is another person who has decided that 14 iterations is enough to get to generally funny with this or any pop song. I'm not sure how funny this really is since I don't know the original song.
So it appears that we are going to see a whole class of songs that are re-interpreted by Google Translate and it is possible to get millions of views as MKR has, and probably even make a living doing this. So here you see one more job created by MT.
So anyway if somebody suggests doing a back test with MT you should know the cards are clearly stacked against the MT monster and the results are pretty close to meaningless. A human assessment of a targeted sample set of sentences is a much better way to understand your MT engine.
Hope you all had a good Thanksgiving vacation and are not feeling compelled to shop too fervently now.
In this time of strife and distrust in Ferguson it is good to see spontaneous goodwill and instant musical camaraderie between these amateur musicians.
My previous posts on MT humor for those who care are:
I should point out that in all these recent cases one does have to work at degrading the translation quality by running the same text through a whole sequence of preferably not closely related languages.
It has often surprised me that there are some in the MT industry who use “back translation” as a way to check MT quality, as from my vantage point it is an exercise that can only result in proving the obvious. MT back translation by definition should result in deterioration since to a very great extent MT will almost always be something less than a perfect translation. This point seems to evade many who advocate this method of evaluation, so let me clarify with some mathematics as math is one of the few conceptual frameworks available to man where proof is absolute or pretty damned certain at least.
If one has a perfect MT system then the Source and Target segments should be very close if not exactly the same. So mathematically we could state this as:
Source (1) x Target (1) = 1
since in this case we
know our MT system is perfect ;-)
So lets say you and I do a somewhat serious evaluation of the output of various MT systems (each language direction should be considered a separate system) and find that the following table is true for our samples by running 5,000 sentences through various MT conversions and scoring each MT translation (conversion) as a percentage “correct” in terms of linguistic accuracy and precision.
| Language Combination | Percentage Correct |
| English to Spanish | 0.8 or 80% |
| Spanish to English | 0.85 or 85% |
| English to German | 0.7 or 70% |
| German to English | 0.75 or 75% |
So if we took 1,000 new sentences and translate them with MT we should expect that the percentage shown above would be “correct” (whatever that means). But if we now chain the results by making the output of one, the input of the other, we will find that results are different and and get continually smaller e.g.
EN > ES > EN = .8 x .85
= 0.68 or 68% correct
EN > DE > EN = .7 x .75
= 0.525 or 52.5% correct
So now if we take our example and run it through multiple iterations we should see a very definite degradation of the output as we can see below.
EN > ES > EN(from MT) > DE > EN = .8
x .85 x .7 x .75 = 0.357 or 35.7%
So if you are trying to make MT look silly you have to run it through multiple iterations to get silly results. It would help further if you chose language combinations like EN to Japanese to Hindi to Arabic as this would cause more rapid degradation to the original English source. Try it and share your results in the comments.
So here we have a very nicely done example and you should realize it takes great skill for the lead vocalist to mouth the MT words as if they were real lyrics and still maintain melodic and rhythmic integrity so be generous in your appreciation of their efforts.
This video shows very effectively how using multiple languages very quickly can degrade the original source as you can see when they go to 64 languages. Somehow words get lost and really strange.
And here is one from a vlogger who really enjoys the effect of multiple rounds of MT on a songs lyrics. She is a good singer and is able to maintain the basic melody without breaking into a smile so I found it quite enjoyable and I would not be surprised that some might believe that these were indeed the lyrics of the song. She has a whole collection of recordings and has what I consider are high production values for this kind of stuff.
And she produces wonderful results on this Disney classic "When you paint the colors of your air can" which used to be a favorite of my daughter. I actually think the song from the Little Mermaid is much funnier and was done by just running it only through four iterations in Google Translate, but since I could not embed it here directly I have given the link.
Here is another person who has decided that 14 iterations is enough to get to generally funny with this or any pop song. I'm not sure how funny this really is since I don't know the original song.
So it appears that we are going to see a whole class of songs that are re-interpreted by Google Translate and it is possible to get millions of views as MKR has, and probably even make a living doing this. So here you see one more job created by MT.
So anyway if somebody suggests doing a back test with MT you should know the cards are clearly stacked against the MT monster and the results are pretty close to meaningless. A human assessment of a targeted sample set of sentences is a much better way to understand your MT engine.
Hope you all had a good Thanksgiving vacation and are not feeling compelled to shop too fervently now.
In this time of strife and distrust in Ferguson it is good to see spontaneous goodwill and instant musical camaraderie between these amateur musicians.
My previous posts on MT humor for those who care are:
I took your rather heart-warming statement of faith: "math is one of the few conceptual frameworks available to man where proof is absolute or pretty damned certain at least." and put it through translationparty.com.
ReplyDeleteThe result after several iterations in several languages was "And never used is math man responsible, or at least, there is a concept of particular evidence."
Not exactly humour, but not informative about the original message either.
Victor, I have been told that my use of lengthy, run-on sentences is not very friendly to MT. I usually just write these posts in a single sitting and may only give it a single edit review. But translationparty.com is a shortcut to create MT silliness. However, I bet if you chose the right sequence of languages you would get much funnier results.
DeleteAha, so "lengthy run-on sentences" are not very friendly to MT. In my translation work I almost always get lengthy run-on sentences, with the added detail that in my source language (German) and my usual domains (law/contracts/building/real estate) the word order invariably does a little waltz along the way. Perhaps that is one reason why I am so incurably sceptical about the worth of MT, and especially "post-editing".
ReplyDeleteYes I understand that MT has many shortcomings and that we could generate a very long list of things that human translators can do that MT cannot. But this does not mean that MT cannot provide value and utility for a lot of business translation scenarios quite effectively. For a different perspective on this you could take a quick look at the following: What do job trends in the translation industry mean to you? | LinkedIn ow.ly/Fag4k or go directly to the post here ow.ly/FagEn
DeleteThanks for the link. My spontaneous reaction is that while this guy is speculating, translators are getting on with their work - because there is plenty of translation work that needs solid conventional translating skills, and the MT+post-editing model just would not cut it with the type of work I get. I certainly use technology in my work (translation memory and terminology management with DVX3, web research of various kinds, occasional recourse to Google Translate as an "extra dictionary"), but there is nothing digital that could produce even a rough first draft (even some highly educated Germans can't really work out what some of my source texts are talking about).
ReplyDeleteHaving said that, I note with interest that Diego Bartolome recently attracted a couple of hundred applicants for post-editing work just by posting a couple of tweets. I would be interested to hear whether these people are really suitable applicants, what their career aspirations are and what prospects Diego can offer them. My normal prejudice would assume that they are clutching at straws on their slippery path to the bottom of the pile (to mix a few metaphors), but perhaps there is more to it than that.
It is clear to most that MT is not suitable for many types of specialized translation tasks. However, much of the focus of business translation is material that has a limited shelf-life and is often very repetitive stuff about products that do lend themselves to skilled MT use. I think your observations about "MT" are really about generic MT and Do-It-Yourself MT are probably correct. Romance languages are generally easier to do than DE and JP and some other languages.
DeleteI would not assume that everybody who is involved with post-editing has lost self-respect and is rapidly sliding down a slippery slope. There are some out there who understand how to do post-editing work in a way that makes economic sense to them.
Hmmm, you write "There are some out there who understand how to do post-editing work in a way that makes economic sense to them."
ReplyDeleteStrange, it is always people "out there" who can happily post-edit. No face, no name, and certainly no post-editors who can speak for themselves and describe their own business experience.
The only arguments I have seen so far to support post-editing as a career have been hearsay and speculation. Never any personal confessions.
Victor,
DeleteHave you considered the possibility that they may wish to keep a low profile because of the censure of other "real" translators or ridiculed for accepting lower rates?
I think there are very few who ONLY do post-editing and nothing else, but the group that sometimes does post-editing is growing all the time, and I think we may soon reach a point where some will come forward and talk about it at conferences or industry events and help others to understand when it is worth serious consideration.
I think better understood PEMT compensation and more transparent practices around this kind of work will also eventually make it less negative. And in some cases we may also see that it is not so different from working with TM.
But it is also clear that for some or many translators, PEMT is work that they would prefer to avoid whatever the facts at hand.
Keep sharing more.
ReplyDelete