Deepfake is a technique based on artificial intelligence that allows audio or video files to be superimposed on other existing files in order to reproduce a person’s voice or change their face and thus make them say things or do things that they have never done. This technology is often used to create hoaxes or to spread fake news. However, it could be put to good use in the film industry for dubbing actors in multiple languages.

AI software to get around dubbing and lip articulation problems

Typically, deepfake is used to alter a person’s facial features, gestures or voice in an image or video, in order to make them say something they never actually said or did. If this practice is actually associated with intoxicants and internet memes, it could eventually be used for the cinema to allow a more realistic and easy to realize dubbing.

The British company Flawless has announced the development of a software program, called TrueSync, which can analyze the facial patterns of actors and modify them so that they can be integrated into the vocalization of another language. To be more precise: one of the recurring problems in dubbing is to make sure that the words in the dubbed language can match the movement of the lips. A difficulty that the AI experts at the British company want to alleviate.

Scott Mann, one of the directors who decided to exploit Flawless’ technology, explains:

“I’m baffled when I look at films I’ve directed but dubbed into other languages, because the original script is rarely retained. If you make a small change in a word or a performance, you can have a big change in a character’s personality or in the pace of the story, and therefore on the understanding of the film.”

Deepfake, a useful practice in some circumstances, subject to ethical questions

TrueSync allows for realistic synchronization by matching the original actors’ mouth and lip movements to the articulation that can be found in many foreign languages. A few weeks ago, Flawless published a video showing some of the results of using the tool to dub movies with famous actors such as Robert de Niro, Tom Hanks, Jack Nicholson and Tom Cruise.

However, the use of software like this presents several ethical issues. Some professionals have noted that if the software falls into the wrong hands, it could be used to commit crimes. In addition, it must be assumed that actors must accept that their lips (and therefore part of their body) be modified to facilitate dubbing into other languages.

Several questions that will undoubtedly push back the future and massive use of deepfakes techniques in cinema.

Translated from Cinéma et Deepfake : zoom sur la synchronisation des lèvres avec le doublage en langue étrangère