I have AI, do you have a story?
Editor’s note: This article is from WeChat public account “ Quabit ” (ID: QbitAI ), The author Guo Yichen.
Seeing pictures is always easier than reading words.
For example, it is easier and easier to read a novel adapted from a novel than to read a text version of a novel.
So how do you automatically turn a story into a comic?
AI can already do it. Give it a story, it can tell the story with pictures, with a little modification, it becomes a set of comics.
A story like this:
A long time ago, a little boy lost his way in the forest. He walked for a long time and was all hungry. He wanted to eat the cakes made by his mother, but this time the little boy couldn’t go back, so he was on the ground. I drew a big loaf and ordered sesame seeds on the loaf.
The little boy looked at him as if he had eaten a big pie, so he didn’t feel very hungry, and stood up again and walked along a small path …
Soon, the little boy’s parents found him, and they came home together and ate a real cake.
AI can turn it into a comic like this:
In addition, this comic form can also serve as a storyboard in the film industry to assist filmmakers in artistic creation.
Find a storytelling picture
So how does this work?
First of all, these pictures are not drawn by AI, but they take a simpler and easier way:
Find out some similar composition from the ready-made gallery and change it.
The gallery here is called the GraphMovie dataset, and the data source is some movie review websites.
But there are many pictures in the data set. How can you use AI to automatically find the picture that best matches your story?
A model is used here, called Contextual-Aware Dense Matching model (CADM).
It looks like this:
According to the content of the story, CADM found some of these images:
In addition, there is a model called No Context, which is the best performing AI in the field of “looking at characters and looking for pictures”.
No Context also found some images:
However, the pictures are still a little bit. Others are also a complete story. You ca n’t take such a few pictures and finish them. The story has no details and the reader ’s experience is not good.
Now, please come up with a third model: the greedy decoding algorithm, which is responsible for finding the available pictures based on the details in the story.
After making it come and make up for it, the story is complete immediately:
Unify the style of pictures
However, with this effect, you will be confused, as if it cannot convey the story in the previous text.
Where’s the problem?
The first problem is that there are many background, environment and other related elements in the picture, which have nothing to do with the main line of the story. Seeing it will also affect your understanding of the story.
You need to delete these redundant elements. Here, He Kaiming’s famous work Mask R-CNN is used to segment the area and delete the parts of the picture that are not related to the story.
Now these pictures look like this:
Second question, the style of these pictures is really differentIt’s too big. If you take out such a comic, it will be hit by readers.
So, we need to unify the styles of the pictures. Here is a tool called CartoonGAN, which can be understood literally. This is a GAN (Generate Adversarial Network) that turns pictures into cartoon style.
After the cartoon GAN processing, this set of pictures becomes like this:
It seems better, but there is another big bug: these people look different! You say they are the protagonists of the same story, and I don’t believe it.
So, the third question comes, how can we make these people look the same?
Here, the researchers directly found a software called Autodesk Maya, which is a software for processing 3D images in movies. It uses it to create 3D scenes, characters and props, and uses a semi-manual method to transfer 9 pictures. The people in it all look the same.
However, the author of the paper said that the process of making 3D images in the future is expected to be fully automatic.
This operation is really as fierce as a tiger. Nine pictures with eight poles can’t hit the edge. Now the style is consistent, the story is smooth, and even the background and landscaping are added.
Used in the film industry
Actually, the “comic” generated is not the end result.
It’s actually used to make movies.
During the preparation of the movie, you need a demo called “storyboard”.
