Video game developers are making virtual worlds as realistic as possible, and some researchers have succeeded in making these environments even more believable. As part of the Enhancing Photorealism Enhancementproject, the Intel ISL research group has created a machine learning model that can make images from a video game look so realistic that they appear to be from the real world. Example with these videos taken from the video game Grand Theft Auto V modified in such a way that we can believe that the game takes place in real life.

A machine learning model performing a frame-by-frame analysis

Stephan R. Richter, Hassan Abu AlHaija and Vladlen Koltun are the researchers who developed this machine learning model to improve the realism of video games. Currently, the most graphically advanced video games are realistic, but not so realistic that the game environment is mistaken for reality. With this new tool, the research team has developed an image that comes as close as possible to what you would see in the real world by using artificial intelligence.

In order to work, the system analyzes each of the images generated when playing the video game to improve them. The model relies on a series of factors and data that exist in each image, such as the depth of the elements, their saturation, the direction of the light or the shading of certain objects.

Thanks to this, we end up with photorealistic images like this one:

A system trained with a database of urban landscapes

The artificial intelligence also compares the frames with real-world images containing similar objects and environments. This is possible thanks to the Cityscapes database, which allows the software to recognize a traffic light and to be inspired by one of the traffic lights in the database to model it in a photorealistic way. The image is thus very detailed.

As you can see in the image above, the tone and colors are a little less saturated than in the video game. This is due to the fact that the real images in the Cityscapes database were taken in Germany, a colder country than the Californian region, scene of the video game Grand Theft Auto V. Also, some details are quite different: the asphalt is more “grainy” in the game, as well as the cars which have much more reflections. However, the vegetation is much more detailed in the photorealistic image than in the game.

The tool was designed, in principle, to be able to generate images in real time while a user is playing the game. However, it has not yet been put into practice in this situation and has not been released publicly for players to experiment with. Only a few videos and images are available at the moment.

Translated from Intel ISL dévoile un modèle de machine learning rendant les images de jeux vidéo plus réalistes