Electrical Engineering and Computer Science (EECS) Seminar Series: Dr. Sergey Tulyakov (Snap Research)

Title: Representations for Content Creation, Manipulation and Animation

Abstract: “What I cannot create, I do not understand” said the famous writing on Dr. Feynman’s blackboard. The ability to create or to change objects requires us to understand their structure and factors of variation. For example, to draw a face an artist is required to know its composition and have a good command of drawing skills (the latter is particularly challenging for the presenter). Animation additionally requires the knowledge of rigid and non-rigid motion patterns of the object. This talk shows that generation, manipulation and animation skills of deep generative models substantially benefit from such understanding. Moreover we see, the better the models can explain the data they see during training, the higher quality content they are able to generate. Understanding and generation form a loop in which improved understanding improves generation, improving understanding even more. To show this, I detail our works in three areas: video synthesis and prediction, image animation by motion retargeting, and 3D object manipulation and reconstruction. In each of these works, the internal representation was designed to facilitate better understanding of the task, resulting in improved generation abilities. Without a single labeled example, our models are able to understand factors of variation, object parts, their 3D shapes, their motion patterns and perform creative manipulations previously only available to trained professionals equipped with specialized software and hardware.

Bio: Sergey Tulyakov is a Lead Research Scientist heading the Creative Vision team at Snap Research. His work focuses on creating methods for manipulating the world via computer vision and machine learning. This includes style transfer, photorealistic object manipulation and animation, video synthesis, prediction and retargeting. His work has been published as 20+ top conference papers, journals and patents resulting in multiple innovative projects, including Snapchat Pet Tracking, OurBaby Snappable and Real-time Neural Lenses (gender swap, baby face, aging lens) and many others. Before joining Snap Inc., Sergey was with Carnegie Mellon University, Microsoft, NVIDIA. He holds a PhD degree from the University of Trento, Italy.

For more information, please contact Ming-Hsuan Yang: mhyang@ucmerced.edu