Towards Better Global Landscape of GAN: How Two Lines of Code Change Makes a Difference

Our understanding of GAN (generative adversarial net) training is still very limited since it is a non-convex-non-concave min-max optimization. As a result, most recent studies focused on local analysis. In this talk, we discuss how to perform a global analysis of GANs. We prove that the original JS-GAN has exponentially many bad strict local minima which are perceived as mode-collapse. We show that a 2-line modification to JS-GAN called relativistic standard GAN (RS-GAN) eliminates all bad basins. We extend the two results to a large class of losses as well: for separable GANs (including JS-GAN, hinge-GAN, LS-GAN) exponentially many bad basins exist, while for R-GANs (relativistic GANs) no bad basins exist.

The effectiveness of R-GANs were verified by a few empirical works before (e.g. ESR-GAN in super resolution). Based on theory, we predict that R-GANs has a bigger advantage for narrower neural nets, and our experiments verify that R-GANs (e.g. RS-GAN) can beat their separable counter-parts (e.g. JS-GAN) by 5-10 FID scores in narrower nets. We demonstrate that for high-resolution images like LSUN, while JS-GAN often only generates noise, RS-GAN can generate quite good images.

Prof. Ruoyu Sun

Assistant Professor, University of Illinois at Urbana-Champaign (UIUC) on October 9, 2020 at 11:45 AM in Zoom Webinar
Join Zoom Webinar

Ruoyu Sun is an assistant professor in the Department of Industrial and Enterprise Systems Engineering (ISE) and affiliated with Coordinated Science Lab (CSL) and Department of Electrical and Computer Engineering (ECE), University of Illinois at Urbana-Champaign (UIUC). Before joining UIUC, he was a visiting research scientist at Facebook AI Research, and was a postdoctoral researcher at Stanford University. He obtained Ph.D. in electrical engineering from University of Minnesota, and B.S. in mathematics from Peking University. His current research interests lie on optimization and machine learning, especially deep learning and large-scale optimization.

The Department of Electrical and Computer Engineering hosts a regularly scheduled seminar series with preeminent and leading reseachers in the US and the world, to help promote North Carolina as a center of innovation and knowledge and to ensure safeguarding its place of leading research.