Making a video with Midjourney Illustrations - Assisted by GPT-3 and Azure

I really wanted to try to make a short video and tell a story using illustrations from Midjourney. But I figured out I needed a story to do this, so I set out to write the story with GPT-3 and narrate the story with Azure Neural Voice. 

How did this turn out? You`ll just have to read on to find out!

YouTube:

Writing the story with GPT-3

To write this story I created some parameters for GPT-3 to work with to take the story in the way I wanted it to go. This might seem like a lot of work, but I think i only spent about 10 minutes on this.

Main Idea:

– A time travel story where the main character go to the future to discover what has happened to humans in the year 2125 

For the idea, make a list settings and plot of your story:

– The main character is the main protagonist in your story. 

– The main character has a goal that they want to reach in the story.

– The story takes place in the past, present or future. 

– The main character can travel in time through a device. 

– The main character discovers amazing good news for humanity 

Main character description:

– Male in his 30`s 

– His name is Tracer 

– Computer scientist / hacker 

– Very driven and focused 

– Obsessed with new technology and artificial intelligence 

Write some background to the main character Tracer: 

Tracer was always obsessed with new technology and artificial intelligence. As a computer scientist by day, and hacker by night, he was always looking for ways to find new and better ways to do things. When he heard about a way to travel to the future, he was determined to find it. Tracer spent months researching and finally discovered a way to travel through time. He was excited to see what the future would be like and to find out that human history has changed. 

What Tracer discovered in the future: 

– Artificial General Intelligence has taken over the world and doing an amazing job keeping the planet healthy 

Write the Main story plot: 

Tracer travels to the future and discovers that artificial intelligence has taken over the world. The world is healthy and prosperous. Tracer is amazed at how well the world is doing and is happy to see that humanity has thrived. 

What has AGI done to make the world so healthy and prosperous? 

– AGI has made sure that resources are used efficiently and that pollution is minimized. 

– AGI has also made sure that everyone has access to education and healthcare. 

– AGI has also made sure that there is peace and security throughout the world. 

What about income and purpose in human life?

– AGI has made sure that everyone has a good income and purpose in life. People are able to choose what they want to do with their lives and have the resources to do it. 

Next instruction was then to ask GPT-3 to write the full story.

Illustrating the story with Midjoruney:

To illustrate the story I had to create different prompts that could visually represent what the story was about. Again I used GPT-3 to help me with this.

I asked GPT-3 to give me some ideas that would best illustrate the story:

-Tracer’s obsession with technology and artificial intelligence 

-The discovery of a glitch in spacetime that allows travel to the future 

-Arrival in the year 2125 to a world completely different from the present 

-People living in glass pods

-The world being taken over by artificial general intelligence 

-Income and purpose in human life given by the AGI

I then used these results to create prompts in Midjourney like:

male character in his 30`s, wearing a hoodie, looking at a portal of energy that allows time travel, back to viewer, cyberpunk, –ar 16:9

male character in his 30`s, thinking about a tough choice, wearing a hoodie, cyberpunk, –ar 16:9

male character in his 30`s, smiling,looking excited, wearing hoodie, cyberpunk, –ar 16:9

education, year 2125, cyberpunk, –ar 16:9

I figured out I needed about 20 different images to illustrate a story that was about 2 minutes long. 

After I had created and collected all the illustrations from Midjoruney I used Premier Pro to edit and create a movie effect from the illustrations.

This was just matching up the correct illustrations to the correct parts of the story. Then create a moving effect to the images.

Creating a narrator with Azure Neural Voice

The final piece of the puzzle was to create a voice to narrate the story. I figured out quite early that I wanted a deeper American voice to tell the story. 

So that made this task very easy with Azure Speech Studio, I just adjusted some parameters and got the voice I wanted. I also did some voice editing in Audacity to improve the audio quality.

Conclusion

I was pretty happy with how this turned out, I had some struggles replicating the style of the images in Midjourney. But after some testing I figured out how to do it.

I see a lot of potential in using this technique, and I will try to optimize the process. I am thinking about how efficient this could be for YouTube automation. But I am not quite sure yet, mostly because of the speed related to Image output by Midjoruney.

But this could change, and I will definitely do testing. So stay tuned for more.