Matthew Tancik: Neural Radiance Fields for View Synthesis

Talk @ Tübingen seminar series of the Autonomous Vision Group

Neural Radiance Fields for View Synthesis
Matthew Tancik (UC Berkeley)

Abstract: In this talk I will present our recent work on Neural Radiance Fields (NeRFs) for view synthesis. We are able to achieve state-of-the-art results for synthesizing novel views of scenes with complex geometry and view dependent effects from a sparse set of input views by optimizing an underlying continuous volumetric scene function parameterized as a fully-connected deep network. In this work we combine the recent advances in coordinate based neural representations with classic methods for volumetric rendering. In order to recover high frequency content in the scene, we find that it is necessary to map the input coordinates to a higher dimensional space using Fourier features before feeding them through the network. In our followup work we use Neural Tangent Kernel analysis to show that this is equivalent to transforming our network into a stationary kernel with tunable bandwidth.

[animeshkarnewar]

Love the way the presentation is arranged. Didn't get distracted at all. Awesome job!

[alanmelling]

Great tour of recent research. And brilliant work with NeRF

[Klarpimier]

When do you think Zillow will incorporate this tech for like house tours?

[animeshkarnewar]

The analysis in the 2nd project is really insightful. Although I think the whole NTK theory is not applicable when the loss (objective) function is not some variant of l2 loss. For instance if you are minimizing a discriminator's adversarial loss or a VGG's perceptual loss, then in that case, the same neural network without these mappings is able to represent the high-frequency details that we care about. It would be really interesting to consider this effect in the analysis too. I look forward to the follow-up work. Thanks a lot for sharing the presentation.

[benibachmann]

Thank you! This is a really exciting paper which I see a number of practical applications for. For example product visualization where no 3D model exists; especially if the inference time could be reduced to sub second time frames. This approach could also lead to improved photogrammetry tools by reducing the number of photos needed and tackling challenges with reflective surfaces.