Remastering Video Game Cutscenes with Stable Diffusion Temporal Kit (and ControlNet and Ebsynth)

FFmpeg:

FFmpeg easy install video:

Ebsynth:

Temporal Kit:

ControlNet:

ControlNet models:

TABLE OF CONTENT
0:00:00 Introduction
0:00:43 Prerequisites
0:02:55 Workflow Overview
0:03:52 Pre-Processing
0:05:27 img2img
0:11:11 Ebsynth-Process
0:11:46 Ebsynth
0:12:30 Results
0:14:05 Possible improvements

In this video, I show you how to remaster old video game cutscenes using Stable Diffusion, Temporal Kit, Control Net and Ebsynth. I’ll walk you through the process of using these tools to create stunning remastered cutscenes that look better than ever.

Stable Diffusion is a powerful tool that allows you to create stunning visual effects with ease. Temporal Kit is a plugin that helps you create temporally coherent animations. Control Net is a game-changing method that allows artists to take large pre-trained diffusion models (like Stable Diffusion 1.5) and extend them to support additional input conditions. Ebsynth uses “non-parametric example-based synthesis” to propagate keyframes and create temporally coherent end results across the clip.

If you’re looking to remaster old video game cutscenes, this is the video for you. I’ll show you how to use these powerful tools to create visuals that are sure to impress.

[clenzen]

Thanks for the video. I hope you keep it up. Lots of good info.

[-Belshazzar-]

Hey thanks for the tutorial, I a bit confused though, if I understand correctly, the first process with EBsynth is just to reduce time generating the images for the second phase since it's "guessing" the in between frames for you. But in the second phase (where you use the img2img) you use the frames generated from EBsynth but they look the same as the original extracted frames (before passing to EBsynth)?

So I am kind of lost why this step at all, and why even using mmfpeg and temporalKit?
I mean the actual stylizing is done in the second phase in img2img where I see no connection to EBsynth or temporal kit? can't i just do a batch process directly in img2img?

Am I missing something here?

Thank you again!

[zhenthewei]

Hi there, I uses this exact method. But I have one problem, when I put all my crossfade videos together, I noticed there's a significant missing frames in between the crossfade videos. How could I fix this? I put 24 for FPS in Ebsynth-Process, is this causing the problem, should I not change it and leave it at 30fps?

[ppn]

hi thanks for the video, when i split my video, i got input video with 1sec instead of 3. Can i change this?

[joaopaulodepretto]

my ebsynth is not loading the frames after i drag the frames folder to it, ... any idea?

[Ekkivok]

hmmm you forgot to put the reference image on controlnet, so it cannot work without it. your video is great but without those missing video is useless for people don't know control net.

[kinkyqt]

Stop using white mode on everything I beg of you 😢