NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ML Research Paper Explained)

#nerf #neuralrendering #deeplearning

View Synthesis is a tricky problem, especially when only given a sparse set of images as an input. NeRF embeds an entire scene into the weights of a feedforward neural network, trained by backpropagation through a differential volume rendering procedure, and achieves state-of-the-art view synthesis. It includes directional dependence and is able to capture fine structural details, as well as reflection effects and transparency.

OUTLINE:
0:00 - Intro & Overview
4:50 - View Synthesis Task Description
5:50 - The fundamental difference to classic Deep Learning
7:00 - NeRF Core Concept
15:30 - Training the NeRF from sparse views
20:50 - Radiance Field Volume Rendering
23:20 - Resulting View Dependence
24:00 - Positional Encoding
28:00 - Hierarchical Volume Sampling
30:15 - Experimental Results
33:30 - Comments & Conclusion

Abstract:
We present a method that achieves state-of-the-art results for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. Our algorithm represents a scene using a fully-connected (non-convolutional) deep network, whose input is a single continuous 5D coordinate (spatial location (x,y,z) and viewing direction (θ,ϕ)) and whose output is the volume density and view-dependent emitted radiance at that spatial location. We synthesize views by querying 5D coordinates along camera rays and use classic volume rendering techniques to project the output colors and densities into an image. Because volume rendering is naturally differentiable, the only input required to optimize our representation is a set of images with known camera poses. We describe how to effectively optimize neural radiance fields to render photorealistic novel views of scenes with complicated geometry and appearance, and demonstrate results that outperform prior work on neural rendering and view synthesis. View synthesis results are best viewed as videos, so we urge readers to view our supplementary video for convincing comparisons.

Authors: Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng

Links:

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

[YannicKilcher]

OUTLINE:
0:00 - Intro & Overview
4:50 - View Synthesis Task Description
5:50 - The fundamental difference to classic Deep Learning
7:00 - NeRF Core Concept
15:30 - Training the NeRF from sparse views
20:50 - Radiance Field Volume Rendering
23:20 - Resulting View Dependence
24:00 - Positional Encoding
28:00 - Hierarchical Volume Sampling
30:15 - Experimental Results
33:30 - Comments & Conclusion

[Jianju]

This type of pre-digestion for a complex technical paper is very expedient. Thank you.

[thierrymilard]

First time I really truely understand NERF. Wonderfull simple explanation. Thanks a lot !

[TheDukeGreat]

Wait, did Yannic just release a review of a paper I already read? So proud of myself :D

[adriansalazar]

One of the best NeRF explanations available. Thank you so much, it helped a lot.

[gravkint]

Gotta present this paper for a seminar at uni so this video makes it so much easier. Thank you so much for this!

[gqynlig]

Man you got many clear notes to explained papers. I got tons of helps from your videos.

[peter]

My guy, this has to be the best tutorial on NeRF I've seen, finally understood everything

[aayushlamichhane]

Awesome explanation! Please dont stop making these.

[vslaykovsky]

"Two papers down the line" we'll probably see a paper that also infers positions and directions of photos.

[dsp]

Excellent explanation. Realtime 3D Street View should be right around the corner now.

[nitisharora]

Thanks for creating such a detailed video on NerF

[kameelamareen]

Beautiful and Super Intuitive video ! Thanks :3

[LouisChiaki]

I feel this approach probably has been used in physic and 3D image reconstruction for a long time with the Fourier decomposition technique (that is renamed as the positional encoding here). The main point is that it is 1 model per object so I feel like it is a curve fitting problem. Though using gradient descent and neural network like framework probably makes it much easier to model.

[muhammadaliyu]

UC Berkeley - I salute this University when it comes to A.I. research. In most big paper, you will definitely see one or more scholars from it.

[trejohnson]

The "overfitting" is one of the core principles in Functional Programming/Dataflow Programming. Very awesome to see, wil have to check whether or not it was a locally unique idea, or if it is directly pulling from the aforementioned knowledgebases.

[NoobMLDude]

Thanks for the Great explanation. Finally understand the central ideas behind NeRF.

[howdynamic]

Thank you for the clear-cut and thorough explanation! I was able to follow and that is definitely saying something because I come from a different world, model-based controls :)

[oiuusqk]

Pretty clear and great thanks to you!!

[tnmygrwl]

Had been waiting for this for a while now. 🔥