Everyone had heard of the Turing test. The idea there is to see if a chat bot can pass as human. What if there were a similar test or competition for machine interpreting?
To make life interesting and fair, I would like to suggest that we have a single, authentic event but supply interpreting in the same language pairs provided by professional human interpreters in person, professional human interpreters via a remote interpreting platform that allows the interpreters to have a video feed, and one or more machine interpreting solutions.
It would then be a simple matter of providing one audio channel for each of the human teams (one in-person, one remote) and any machine interpreting software being tested. The audience could then choose to listen to their feed of their choice and record their impressions.
Vitally, they would be asked to suggest which channel was which (human in-person, human remote, machine(s)). Their impressions could then be cross-checked by looking at which channels were listened to the most.
As too many machine interpreting developers have discovered, laboratory results are simply not a reliable guide to real-world performance. The only way to truly test the state of machine interpreting and make useful improvement is to run a field test. And, my view is that the field test should be as realistic and comprehensive as possible.
This is why I would suggest the following setup for the event:
2 hours of simultaneous interpreting, alternative between languages every 20-30 minutes (so interpreting English to German then German to English then back again) followed by,
a 1-2 hour factory tour (Wi-Fi signal not guaranteed) with the audience hearing on a system similar to the commercially available tour guide systems followed by returning to the conference hall for,
a 1-2 hour discussion with the questions unknown beforehand to any of the teams or the speakers but still on the themes covered by the event.
The two sessions in the conference hall would, ideally be live-streamed, to get the reactions of those outside the event. Following the success of the Heriot-Watt University Multilingual Debate, it should be possible to put on a conference that would be interesting in its own right and otherwise identical to one the interpreters would normally work at. This breadth of work represents work that will be familiar to most freelance conference interpreters.
In fact, there is no real reason why a company could not use such an event to give an insight into their products. A whisky company could talk about their environmental policies, fishing industry representatives could talk about regulations and give a tour of a working boat, a manufacturing company could showcase their innovation.
Since few clients have large pre-aligned bilingual databases, I would also suggest that every one of the teams receive identical briefing documents from the client. And, of course, if a speech doesn’t arrive on time and some creativity is needed, that just adds to the realism.
Given what we know about interpreter motivation, the interpreters and AV team should be paid at their normal rates and should be chosen by a consultant in the same way as they would be chosen for a normal job. There should perhaps also be some sort of monetary award for the winners too, to encourage further developments.
I believe that this would not only give us an accurate view of where machine interpreting is but would encourage developments in the field, showcase excellent intepreting and provide a platform for a company with interesting products to show what they do. There would likely be significant press interest, just as there has been when machine translation companies claim to match humans or when literary translators are pitched against their digital counterparts.
It should not be difficult to find a venue to host it and I am sure that any of the good interpreting AV companies would relish the challenge of finding a way to keep the test fair and hide the identity of who was on each channel, even during the factory tour.
It would just need a corporate sponsor and a company or organisation willing to be guinea pigs. If you know anyone who would be interested in playing any role, please drop me an email. I would welcome any feedback on the idea.