Learning Controls through Structure for Generating Handwriting and Images

Exposing meaningful interactive controls for generative and creative tasks with machine learning approaches is challenging: 1) Supervised approaches require explicit labels on the control of interest, which can be hard or expensive to collect, or even difficult to define (like 'style'). 2) Unsupervised or weakly-supervised approaches try to avoid the need to collect labels, but this makes the learning problem more difficult. We will present methods that structure the learning problems to expose meaningful controls, and demonstrate this across two domains: for handwriting - a deeply human and personal form of expression - as represented by stroke sequences; and for images of objects for implicit and explicit 2D and 3D representation learning, to move us closer to being able to perform `in the wild' reconstruction. Finally, we will discuss how self-supervision can be a key component to help us model and structure problems and so learn useful controls.

00:00 Intro
10:29 Part 1 - Atsunobu Kotani (Atsu) - Generating Handwriting via Decoupled Style Descriptors (ECCV 2020)
15:39 Decoupled Style Descriptors (DSD)
26:16 Subsequences rather than single characters
28:51 Combined method with sampling
34:04 Few-Shot learning of New Characters
36:20 BRUSH dataset
37:05 Conclusions and discussion
51:05 Part 2 - James Tompkin - Unsupervised Attention (NeurIPS 2018) and Generative Object Stamps (CVPR W 2020)
51:40 Constrained dataset example - Faces
56:16 Now-standard cycle approach: Two GANS
57:55 Neural Style Transfer Struggles not to Deform Background
59:07 Attention helps separate foreground/background distributions
01:03:02 Why does this work?
01:05:31 We trying make 'image domain' that doesn't exist!
01:11:01 Generative Object Stamps
01:21:34 Existing approaches handle dataset variation
01:23:58 James This is such hack! Only 2D, pose variation is 'random'.
01:25:37 GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes
01:27:10 Interactive Editing
01:30:41 Our approach: Mixture of Anisotropic Gaussians
01:38:31 Discussion

[Chapters were auto-generated using our proprietary software - contact us if you are interested in access to the software]

The talk is based on the papers:

Generating Handwriting via Decoupled Style Descriptors (ECCV 2020)

Unsupervised Attention-guided Image to Image Translation (NeurIPS 2018)

Generating Object Stamps

Presenters BIO:

Dr. James Tompkin is an assistant professor of Computer Science at Brown University. His research at the intersection of computer vision, computer graphics, and human-computer interaction helps develop new visual computing tools and experiences. His doctoral work at University College London on large-scale video processing and exploration techniques led to creative exhibition work in the Museum of the Moving Image in New York City. Postdoctoral work at Max-Planck-Institute for Informatics and Harvard University helped create new methods to edit content within images and videos. Recent research has developed new machine learning techniques for view synthesis for VR, image editing and generation, and style and content separation.

Atsunobu Kotani (Atsu) recently graduated from Brown University and will be starting my PhD in EECS at UC Berkeley this Fall.
He is interested in style learning, particularly in artistic domains, such as painting, sculpture and calligraphy. He also works with robots to investigate collaborative art production as well as conservation.

-------------------------
Find us at:

[ddai]

Would love to hear any comment or input you have. Did you find the content interesting? Is there a place for improvement? Please leave any feedback you have here in a comment :)