Pages

Monday, November 20, 2017

Post Editing - What does it REALLY mean?

While many people may consider that all post-editing is the same, there are definitely variations that are worth a closer look. This is a guest post by Mats Dannewitz Linder that digs into three very specific PEMT scenarios that a translator might view quite differently. Mats has a more translator-specific perspective and as the author of the Trados Studio manual, I think provides a greater sensitivity to the issues that do matter to translators. 

From my perspective as a technology guy, this post is quite enlightening as it provides real substance and insight on why there have been communication difficulties between MT developers and translator editors. PEMT can be quite a range of different editor experiences as Mats describes here, and if we now factor in the change that Adaptive MT can have, we now have even more variations on the final PEMT user experience.  

I think a case can be made for both major cases of PEMT that I see from my vantage post, the batch chunk mode and the interactive TU inside the CAT mode.  Batch approaches can make it easier to do multiple corrections in a single search and replace action, but interactive CAT interfaces may be preferred by many editors who have very developed skills in a preferred CAT tool. Adaptive MT, I think, is a blend of both and thus I continue to feel that it is especially well suited for any PEMT scenario as described in this post. The kind of linguistic work done for very large data sets is quite different and focuses on correcting high-frequency word patterns in bulk data, described in this post: The Evolution in Corpus Analysis Tools. This is not PEMT as we describe here, but it is linguistic work that would be considered high value for eCommerce, customer support and service content and any kind of customer review data that has become the mainstay of MT implementations today. 

For those in the US, I wish you a Happy Thanksgiving holiday this week, and I hope that you enjoy your family time. I have pointed out previously, however, that for the indigenous people of the Americas, Thanksgiving is hardly a reason to celebrate.“Thanksgiving” has become a time of mourning for many Native People, hopefully, this changes, but it can only change when at least a few recognize the historical reality and strive to alter it in small and sincere ways.

The emphasis and images below are all my doing so please do not blame Mats for them.

==========


I have read – and also listened to – many articles and presentations and even dissertations on post-editing of machine translation (PEMT), and strangely, very few of them have made a clear distinction between the editing of a complete, pre-translated document and the editing of machine-translated segments during interactive translation in a CAT tool. In fact, in many of them, it seems as if the authors are primarily thinking of the latter. Furthermore, most descriptions or definitions of “post-editing” do not even seem to take into account any such distinction. All the more reason, then, to welcome the following definition in ISO 17100, Translation services – Requirements for translation services:

      post-edit

      edit and correct machine translation output

Note: This definition means that the post-editor will edit output automatically generated by a machine translation engine. It does not refer to a situation where a translator sees and uses a suggestion from a machine translation engine within a CAT (computer-aided translation) tool.

And yet… in ISO 18587, Translation services – Post-editing of machine translation output – Requirements, we are once again back in the uncertain state: the above note has been removed, and there are no clues as to whether the standard makes any difference between the two ways of producing the target text to be edited.


This may be reasonable in view of the fact that the requirements on the “post-editor” arguably are the same in both cases. Still, that does not mean that the situation and conditions for the translator are the same, nor that the client – in most cases a translation agency, or language service provider (LSP) – see them as the same. In fact, when I ask translation agencies whether they see the work done during interactive translation using MT as being post-editing, they tell me that it’s not.

But why should this matter, you may ask. And it really may not, as witness the point of view taken by the authors of ISO 18587 – that is, it may not matter to the quality of the work performed or the results achieved. But it matters a great deal to the translator doing the work. Basically, there are three possible job scenarios:
  1. Scenario A:- The job consists of editing (“post-editing”) a complete document which has been machine-translated; the source document is attached. The editor (usually an experienced translator) can reasonably assess the quality of the translation and based on that make an offer. The assessment includes the time s/he believes it will take, including any necessary adaptation of the source and target texts for handling in a CAT tool.
  2. Scenario B:- The job is very much like a normal translation in a CAT tool except that in addition to, or instead of, an accompanying TM the translator is assigned an MT engine by the client (normally a translation agency). Usually, a pre-analysis showing the possible MT (and TM) matches is also provided. The translator is furthermore told that the compensation will be based on a post-analysis of the edited file and depend on how much use has been made of the MT (and, as the case may be, the TM) suggestions. Still, it is not possible for the translator either to assess the time required or the final payment. Also, s/he does not know how the post-analysis is made so the final compensation will be based on trust.
  3. Scenario C:- The job is completely like a normal translation in a CAT tool, and the compensation is based on the translator’s offer (word price or packet price); a TM and a customary TM matches analysis may be involved (with the common adjustment of segment prices). However, the translator can also – on his or her own accord – use MT; depending on the need for confidentiality it may be an in-house engine using only the translator’s own TMs; or it may be online engines with confidentiality guaranteed; or it may be less (but still reasonably) confidential online engines. Whatever the case, the translator stands to win some time thanks to the MT resources without having to lower his or her pricing.
In addition to this, there are differences between scenarios A and B in how the work is done. For instance, in A you can use Find & replace to make changes in all target segments; not so in B (unless you start by pre-translating the whole text using MT) – but there you may have some assistance by various other functions offered by the CAT tool and also by using Regular expressions. And if it’s a big job, it might be worthwhile, in scenario A, to create a TM based on the texts and then redo the translation using that TM plus any suitable CAT tool features (and regex).

Theoretically possibly, but practically not

There is also the difference between “full” and “light” post-editing: Briefly, the former means that the resulting text is comprehensible and accurate, but the editor need not – in fact, should not – strive for a much “better” text than that, and should use as much of the raw MT version as possible. The purpose is to produce a reasonably adequate text with relatively little effort. The latter situation means that the result should be of “human” translation quality. (Interestingly, though, there are conflicting views on this: some sources say that stylistic perfection is not expected and that clients actually do not expect the result to be comparable to “human” translation.) Of course these categories are only the end-points on a continuous scale; it is difficult to objectively test that a PEMT text fulfils the criteria of one or the other (is the light version really not above the target level? is the full version really up to the requirements?), even if such criteria are defined in ISO 18587 (and other places).

Furthermore, all jobs involving “light-edit” quality is likely to be avoided by most translators 

Source: Common Sense Advisory

These categories mainly come into play in scenario A; I don’t believe any translation agency will be asking for anything but the “full” quality in scenario B. Furthermore, all jobs involving “light” quality is likely to be avoided by most translators. Not only does it go against the grain of everything a translator finds joy in doing, i.e. the best job possible; experience also shows that all the many decisions that have to be made regarding which changes need to be made and which not often take so much time that the total effort with “light” quality editing is not much less than that with “full” quality.

Furthermore, there are some interesting research results as to the efforts involved, insights which may be of help to the would-be editor. It seems that editing medium quality MT (in all scenarios) takes more effort than editing poor ones – this is cognitively more demanding than discarding and rewriting the text. Also, the amount of effort needed to detect an error and decide how to correct it may be greater than the rewriting itself and reordering words and correcting mistranslated words takes the longest time of all. Furthermore, it seems that post-editors differ more in terms of actual PE time than in the number of edits they make. Interestingly, it also seems that translators leave more errors in TM-matched segments than in MT-matched ones. And the mistakes are of different kinds.

These facts, plus the fact that MT quality today is taking great steps forward (not least thanks to the fast development of neural MT, even taking into account the hype factor), are likely to speed up the current trend, which according to Arle Lommel, senior analyst at CSA Research and an expert in the field, can be described thus:
"A major shift right now is that post-editing is being replaced by “augmented translation.” In this view, language professionals don't correct MT, but instead, use it as a resource alongside TM and terminology. This means that buyers will increasingly just look for translation, rather than distinguishing between machine and human translation. They will just buy “translation” and the expectation will be that MT will be used if it makes sense. The MT component of this approach is already visible in tools from Lilt, SDL, and others, but we're still in the early days of this change."

In addition, this will probably mean that we can do away with the “post-editing” misnomer – editing is editing, regardless of whether the suggestion presented in the CAT tool interface comes from a TM or an MT engine. Therefore, the term “post-editing” should be reserved only for the very specific case in scenario A, otherwise, the concept will be meaningless. This view is taken in, for instance, the contributions by a post-editor educator and an experienced post-editor in the recently published book Machine Translation – What Language Professionals Need to Know (edited by Jörg Porsiel and published by BDÜ Fachverlag).

Thus it seems that eventually we will be left with mainly scenarios B and C – which leaves the matter, for translators, of how to come to grips with B. This is a new situation which is likely to take time and discussions to arrive at a solution (or solutions) palatable to everyone involved. Meanwhile, we translators should aim to make the best possible use of scenario C. MT is here and will not go away even if some people would wish it to.


-------------



Mats Dannewitz Linder has been a freelance translator, writer and editor for the last 40 years alongside other occupations, IT standardization among others. He has degrees in computer science and languages and is currently studying national economics and political science. He is the author of the acclaimed Trados Studio Manual and for the last few years has been studying machine translation from the translator’s point of view, an endeavour which has resulted in several articles for the Swedish Association of Translators as well as an overview of Trados Studio apps/plugins for machine translation. He is self-employed at Nattskift Konsult.

23 comments:

  1. I appreciate the effort, even though I would have liked a deeper analysis of the reasons for useless standards like ISO 18587.
    Also, I'm strongly against terms like "augmented translation".
    This said, I'd also like we do not use "human" any more as a substitute for excellent with "translation." In my experience, translation makes the news only when it is bad, and it is regularly "human." A proof of concept is in the statement that "translators leave more errors in TM-matched segments than in MT-matched ones. And the mistakes are of different kinds."
    Finally, it is true that "post-editors differ more in terms of actual PE time than in the number of edits they make," but this is just another effect of the typical "red-pen syndrome" affecting translators, especially when asked to act as revisors (editors.) And it is acquired (fostered I daresay) in translation schools.

    ReplyDelete
    Replies
    1. I did not see a discussion of ISO 18587 as within the scope of my article, even if I also find it of rather limited use.
      As for the term "human translation", I agree. It's just that this term is frequently used; furthermore I believe most people understand it correctly.
      The "red-pen syndrome" might often be a case of (too) high ambition and certainly sometimes misdirected. It's a question of a sometimes difficult balance, I think.

      Delete
    2. ISO 18587 is, in my opinion, just another source of confusion, and since you cited it in your article, a deeper analysis would have helped. I don't think a comment box is the best place to get further into it, though, and it possibly deserves a separate post.

      Delete
    3. Meanwhile, I'd certainly appreciate your comments on ISO 18587 in a private email; I am sure I would find them interesting.

      Delete
  2. Thank you for this interesting article. In my opinion, PEMT is not made by and for translators but by and for companies who want to save time and money. Why should I help them? What's in it for me as a translator? It's not a win-win deal, is it?

    ReplyDelete
  3. As Herr Dr.phil. Tilmann Kleinau says what do we profit of all this ? I had myself used to this day 60+ languages beyond Hungarian which is my mother-tongue - so I can't see very well why should any but any of us accept MT to work with. MT means giving your knowledge and know-how to the concurrency so they could make big bucks daily while you will be cut out of any... This is not a fair thing from my point of view.

    ReplyDelete
    Replies
    1. Dr. Kleinau and Mr. Horváth-Militicsi: I believe your comments pertain to my scenario A above, and I certainly agree that it may be unfavourable to the translator -- in which case I can only say: don't do it! The same goes for scenario B. But as for using MT as an aid as described in scenario C, I don't see why not if it is useful. And if you don't want to give your translations to the MT provider, just don't! Usually you have the choice not to even if you make use of the MT output.

      Delete
  4. We have introduced too many technological "by-passes" to complete and evaluate a translation work; however, it is a Guild's decision to put all the cards on the table - and invite the counterparts to a thorough analysis of the translation process, its stages and the individual evaluation of each... If the Machine Translation is a heresy in form ans substance - the automated screening and sorting in price categories could prove to be a good idea, after all

    ReplyDelete
    Replies
    1. I don't think I agree with you about the technological "by-passes" (if I understand you correctly). And unfortunately, I am sure I don't understand what you mean by your last sentence. But another thourough analysis of the translation process... why not?

      Delete
  5. This poorly written article does not educate as much as it pontificates. The MT industry continues to struggle to balance itself on the backs of human translators.

    ReplyDelete
    Replies
    1. It seems from your comment that you have seen these views expressed before. I would be grateful if you could tell me where.

      Delete
  6. I agree with this, for personal experience: "It seems that editing medium quality MT (in all scenarios) takes more effort than editing poor ones – this is cognitively more demanding than discarding and rewriting the text. Also, the amount of effort needed to detect an error and decide how to correct it may be greater than the rewriting itself and reordering words and correcting mistranslated words takes the longest time of all."

    By the way, I don't understand the approach of so many of my colleagues. MT was developed and exists because of the companies that want to translate more spending less. And the outcomes of this technology are so outstanding that it is spreading at all levels of our industry. Full stop.

    They remind me my grandfather who thought the bike was a devil work.

    ReplyDelete
  7. Mats’ scenarios are a good snapshot of today’s constantly changing use of evolving technologies. As change continues, his scenario A will fade to obscurity. Scenario B will serve the mass commodity market where editors with lower skills work for low pay, because skilled translators will refuse to sign up for unpredictable terms and conditions.

    Scenario C will become the norm for translators who choose their tools from vendors who improve the tools to serve the translator.

    As Mats points out, changes are moving to a market where “editing is editing, regardless” of where the suggestions come from. Maybe soon, we can add the misnomer “post-editing” term to the trash heap where it belongs.

    ReplyDelete
  8. Thank you, very interesting article on this topic. And I fully agree with the statement: "editing is editing, regardless of whether the suggestion presented in the CAT tool interface comes from a TM or an MT engine". We'll need to invoice the actual time spent on this task, not on a per-word basis which does not take into account the amount of work involved in the task.

    ReplyDelete
  9. In my opinion, post-editing is only acceptable if it's performed following human translation (I dislike this term just as much as I dislike the word "organic" in food industry) and the client requires someone who specializes in the given field to ensure correct terminology has been used. Otherwise, excuse my language, it is nothing other than "turd polishing".

    ReplyDelete
    Replies
    1. @Nagme, you can’t polish a turd You can only roll it in glitter. Today's post-editing’s “best practices” date back to the early 1960’s. As Mats points out, they include gist, medium and human revision levels. Those have seen little innovation and woefully few updates Un 50 years despite improvements in the technology that they were created to serve. All the hype around MT improvements and benefits of PEMT are simply our industry's way to roll the turd in glitter. Mats’ astute observations about scenario C are the first creative rewrite of editing rules in half a century!

      Delete
  10. It would interest me to see research looking at a different aspect of Case C:

    You emphasize that using MT would speed up our working process by replacing the step of raw translation and allowing us to invest more time in refining the raw translation (and/or increasing our earnings by increasing our output per hour).

    I would emphasize that working on the basis of MT is likely to fundamentally affect the nature of the resulting translations. Among other things, I think it would probably seduce good translators into using all kinds of problematic or even incorrect phrases and sentence structures. Intensely mentally processing a text 2 or 3 times also seems essential to catching the meaning of peculiar elements in the original text and the subtle misunderstandings or ambiguities we (or MT) introduce into the translation. That is why I think it would also involve more errors or inadequacies at the level of content.

    It seems fair to assume that MT-supported translation not only brings faster results, but also different results than translation without MT. My gut assumption and thesis is that there would be a significant reduction in quality for significant parts of the translation market, but I'd like to see this thesis tested.

    ReplyDelete
    Replies
    1. Michael, While I cannot confirm that the research involved use in a Scenario C model, I am aware that large scale PEMT projects tend to show a few things very regularly: 1) Greater consistency in style and terminology which is highly valued by buyers and actual users, 2) In properly implemented use cases there is also less of a tendency to over-correct. But this requires training with very specific error types and correction strategies.

      Delete
  11. I would agree with Kirti here. To put it facetiously: there is MT and there is MT and then there is every type of MT in between. An example from my own experience, where I translated some 100 pages of popular science: some MT suggestions could be immediately discarded, others (although seldom more than 10 words) were perfect. In very many cases I had to reorder some words, change some, etc. -- but still saved typing time! In another example, involving 25 pages of software agreement, the MT gave me about 90% correct text. So there is definitely a case to be made for trying. But as Kirti says, you have to adjust your way of reading the suggested translations and correcting them. And you need to take great care with your proofreading. (But then of course you should always do that, regardless of the source of the translated text.)

    ReplyDelete
  12. I just want to confirm that your account accurately reflects my experience and my view as an Italian linguist active in various capacities since 1988.

    ReplyDelete
  13. Perhaps I should add a couple of facts to my comment above:

    1. My language combination was En > Sv, which is pretty advantageous. (Most other European languages would work just as fine, but Finnish would be hell, and German, with its strange placement of verbs, would also create problems.)

    2. Software agreement... but what about confidentiality, you might ask. I used the SDL Language Cloud, which guarantees full confidentiality. (I also paid for the service, so it was not a complete gift from above.)

    ReplyDelete
  14. The trouble with the term PEMT is that it also covers pre-editing, which in fact is essential for good MT. All the more so if the text is to be translated into multiple languages, which is one of the strengths of MT. I would agree that the amount of editing can take about the same amount of time, regardless of the quality, as a text that is relatively good but needs some polishing can take as long s a bad text. The problem is that the time savings are not necessarily that much after all, and our income is in the last resort based on the length of time that we work.

    ReplyDelete
  15. MT is good for translation for information where content is important and not style. This kind of text can be corrected by less skilled translators, which immediately means that the prices offered will be lower. The problem then arises that end users think in terms of the going prices for corrected MT work and balk at the prices asked for human translation. Against that, MT means that texts can be translated that have relatively little value and do not justify the cost of formal translation when it is only necessary to know the gist. This is another market, although it may well simply end up as a tool that everyone can use and human translators will be reserved for the texts that even the best MT cannot handle.

    ReplyDelete