Universal Neural Style Transfer | Two Minute Papers #213

The paper "Universal Style Transfer via Feature Transforms" and its source code is available here:

Recommended for you:

We would like to thank our generous Patreon supporters who make Two Minute Papers possible:
Andrew Melnychuk, Brian Gilman, Christian Ahlin, Christoph Jadanowski, Dave Rushton-Smith, Dennis Abts, Eric Haddad, Esa Turkulainen, Evan Breznyik, Kaben Gabriel Nanlohy, Malek Cellier, Marten Rauschenberg, Michael Albrecht, Michael Jensen, Michael Orenstein, Raul Araújo da Silva, Robin Graham, Steef, Steve Messina, Sunil Kim, Torsten Reil.

One-time payments:
Bitcoin: 13hhmJnLEzwXgmgJN7RB6bWVdT7WkrFAHh

Károly Zsolnai-Fehér's links:

[evandavis]

A couple of commenters have asked what's so special about this method as compared to what's been around for the past two years:

The original algo from Gatys in 2015 still produces the most visually pleasing results as far as I know. It poses style transfer as an optimization objective where a noise image is fed forward through a pre-trained image classification net like VGG16, and the loss balances between content/style losses that encourage it to match the VGG features of given content/style images at multiple layers. The error is backpropagated and the noise image is updated with a gradient descent step. The forward/backward/update process is repeated up to hundreds of times, and so this is typically slow and can't stylize in real-time.

jcjohnson in 2016 introduced a 'fast' style transfer approach that addresses the speed issue with somewhat lower quality results. It uses a separate 'image transformation network' that's trained to apply a single style to input content images with only a forward pass. It also uses VGG to calculate a similar loss during training, but at test time only the transformation net is needed. Most style transfer mobile apps likely use a variation of this.


This paper is 'universal' in the sense of having the flexibility to generalize to arbitrary styles while also being fast. In fact, there is *no explicit style transfer objective* and style images aren't even needed for training. The architecture is simply an autoencoder trained to decode VGG features to reconstruct the image that's fed in.

[li_tsz_fung]

It's really delightful hearing you say "See you next time"

[Reavenk]

Waiting for the day style transfer is used as a post effect for a game. Maybe a game where you can jump around different artworks. While there's a lot of stuff you can do with post shaders, there's got to be a bunch of amazing effects and animations that can only be done with style transfers.

[submagr]

What a time it is to be alive! Indeed!

[Peacepov]

this + augmented reality = mind blowing

[FelheartX]

2:28 It is not clear at all how the two inputs are combined. How does this "Feature Transforms" step work?

[BrianMPrime]

Oh man, I just tried this on some photographs. Stunning!

[centar]

Amazing episode!!! Thank you so much, and I have no idea whether you read my comment on the last episode, but this one really had the perfect amount of explanation! Thank you Karoly from a Viennese Fellow Scholar/Data Scientist ;) I only miss how the two bottleneck representations are combined, but -> paper reading it is :)

[ToriKo_]

Wow, I really liked this episode, good job!

[karlkastor]

How are the encodings of both source images combined? Simply adding and averaging wouldn't make a difference between the content image and the style image.

[ProGamerGov]

The results don't seem to look as good as jcjohnson's Neural-Style. This seems to be more towards "Fast" style transfer, which produces lower quality, but faster outputs. Reminds me a lot of AdaIN, Style-swap, and Fast-Neural-Style. These sorts of style transfer networks seem best suited for devices like phones, and those without access to high end GPUs, but they still can't compete with the original Neural-Style.


So I don't think it's accurate to say that you couldn't tune the output artistically to your liking, in previous style transfer algorithms. It's more accurate to say that you couldn't really tune "Fast" style transfer outputs to your liking as easily, in previous "Fast" style transfer algorithms.

[shreeyaksajjan]

I didn't really understand the difference between the old method and the new one. Could you elaborate a bit on this in the comments?

[frankx]

What about the hypothesis that the essence of idealisation in the brain is merely (usually) the result of this "bottlenecking", (or just heavy data compression achieved by some means)?

[J_Dubois]

This is amazing, im new to this and slightly confused on how to get started using this on mac. How do I get started?

[donovankeating]

Hi Karoly. I love your content. Can you do some stuff on NLP? :)

[WannabePianistSurya]

Implementing this on Android app for an research internship. Let's hope the results are decent :D

[JoaquinRocco]

There are apps that are performing real-time style transfer on phone already. Check out envision and dreamsnap! Both run on the GPU of iPhones using a framework called Bender

[chkone]

Even If I see the interest of this work, depending of the style I still prefer (qualitatively) the other methods ( :

[calebray]

How do I install this on Windows??
Is it possible?

[supercables]

Downloaded the code, how do i use it?