This article is from WeChat official account:Big Data Digest (ID: BigDataDigest), author: Niu Yang Wan, the original title: “one picture, two time lines! Using AI to rewrite the time of the different protagonists in the video, the effect is comparable to the blockbuster”, the title picture comes from: “creed”
Have you watched the popular movie “Creed” a while ago?
No matter from the plot or the production of this movie, there has been a wave of public opinion. The most impressive thing in the film is undoubtedly several “time clamp wars”, which highly restored the entire process of time reversal, rather than jumping directly to a certain point in the past.
In order to truly show the movement of time in different directions, director Nolan stated that there are less than 300 special effects shots used in this movie, which may even be fewer than most romantic comedies. It really makes people real name Risby.
Nolan’s insistence on real shooting as much as possible is also his own “creed”, but With the advancement of technology, it is undeniable that technology can do more and more things.
No, Google and Oxford University researchers recently published a “new video editing tool” that can “rewrite time“!
In the video, you can fast forward, slow down, or even delete the actions of specific characters, without affecting other characters on the screen. A variety of special dynamic effects can also be achieved. Come and have a look with Digest Bacteria.
When deep neural networks learn to “control the timeline”
Let’s look at an example first. Here is a video of kids diving. In the original video, they jumped into the water separately:
Researchers used a new deep neural network for video processing, and successfully made them jump into the water together! Moment to witness the miracle:
Is there no trace of modification at all, it looks so natural and smooth.
This is the “temporal rewriting technique” proposed by researchers at Google and Oxford University.They trained a deep neural network to learn how to decompose video in layers.
This model not only separates the movement of people in different layers, but can also capture various scene elements related to those people (for example, children splashing in the water, shadows, reflections). When people in the video are “rewritten in time”, these related elements will also be automatically retimed with them, which allows researchers to create real video recolors for various retimed effects.
Let’s take a look at this “Time Rewriting” Dafa and what else can be done without any violation of the video~
“Freeze Time”
As shown in the picture below, here are two pairs of children dancing Latin dance. You can see that they have been dancing since the beginning of the video in the original video.
Next, it’s time to “freeze time”! As we have seen in the movie, the protagonist with super power can move when everyone is still. Then this kind of special effect seems to be safe to hand over to AI in the future!
Just like this, AI can offset the time. For example, the offset time is 1 second. We can see that the pair of kids on the right side stood still for 1 second before they started dancing. Naturally, their dance moves are compared to the left side. The group will also be delayed by 1 second.
In addition, this neural network can also control who “freeze” and the “freeze” time as you like, such as jumping halfway to stop the pair on the right~
The pair on the left will stop and jump again~
You think this is it? ? Of course more than that. This neural network can also make the characters in the video become a “ghost” effect.
Character dissolve
When editing a video, there is a transition effect called “Dissolve”. What does that mean? It is a transition method adopted to make the switching effect more natural when switching between two clips in the video. In other words, the end of the previous segment overlaps with the beginning of the next segment.
The neural network trained by researchers at Google and Oxford University can achieve a similar effect to the “dissolve” transition. Researchers call this effect-Duplication. Different from video transitions, this effect can be used to dissolve the actions of characters in a video by rewriting the time.
Let’s look at an example. The little girl in pink pants in the original video did a cartwheel:
Through the neural network rendering, the following effects can be turned into:
Is the action of the little girl turning the cart a bit cool after dissolving it~ In addition, the girl in the blue dress on the right uses the freeze effect. From the comparison of the two videos, there is no difference in the surrounding environment at all. This kind of special effect can be said to be very successful~
Through layered neural rendering, only the character timeline is redefined, and the video has no sense of contradiction
All these effects are achieved by a new model based on deep neural network, the core of this technology is layered neural rendering.
That is, the model can be optimized according to the video, and each frame of image is decomposed into a set of layers. Each layer is composed of an RGB color image and an opacity mask α (collectively referred to as “RGBA”). Single/multiple characters are associated.
Background layer and 1-3 layers
It should be noted that researchers only focused on rewriting time in this study. In other words, the poses of the characters in the output video appear in the original video, They do not generate new, invisible poses or viewpoints.
It is worth mentioning that their method does not require manual annotation or explicit representation of dynamic scene elements, such as shadows, splashes, and trampoline deformation;The operator performs rough parameterization, and then the model automatically learns to reconstruct the scenes related to the characters in groups. Importantly, the retiming effect can be produced by simple operations on layers (removing, copying or interpolating specific layers) without additional training or processing.
Layered Neural Rendering
For more theoretical details, please refer to the paper “Layered Neural Rendering for Retiming People in Video 》
The relevant code of this research will also be released at SIGGRAPH Asia 2020, and the conference is expected to be held on December 4.
Finally, Digest Bacteria is also looking forward to rubbing his hands, hoping that AI can join the movie special effects army in the future~
Related reference: https://retiming.github.io/
This article is from WeChat official account: big data Digest (ID: BigDataDigest) , author: Niu Wan Yang