Yuchao Gu

Ph.D. Student

Show Lab
National University of Singapore

Email: yuchaogu9710 [at] gmail.com

Biography

Hi there! I am a third-year Ph.D. student in Show Lab @ NUS, working with Prof. Mike Shou. Before that, I received master's degree in Nankai University in 2022, working with Prof. Ming-Ming Cheng. In 2019, I received my bachelor degree from Beijing University of Chemical Technology , working with Prof. Wei Hu.

My research interests focus on visual generation. While visual generation has advanced significantly with text-to-image and text-to-video models, these approaches are often limited by the less controllable nature of language. I have worked on enhancing control in visual generation across various aspects, including identity control (Mix-of-Show, NeurIPS 2023), point trajectory control (VideoSwap, CVPR 2024), and instance control (ROICtrl, 2024). My current research interests lie in promoting physics consistency in visual generation and developing next-generation generative models.

I also collaborate on several exciting research directions related to visual generation.

Unified Understand and Generation: Show-O
Foundation Models for Visual Generation: Show-1 (IJCV24), EvolveDirector (NeurIPS, 24)
Downstream Applications of Visual Generation: MotionDirector (ECCV 24, Oral), DragAnything (ECCV 24), DynVideo-E (CVPR 24), Tune-A-Video (ICCV 23)

I firmly believe that visual generation is an indispensable step toward AGI. I am always open to discussion and collaboration. Feel free to reach out.

News

2024 Sept: One paper (EvolveDirector) got accepted by NeurIPS 2024.
2024 July: Two papers (MotionDirector, DragAnything) got accepted by ECCV 2024, with MotionDirector selected for Oral presentation.
2024 Feb: Four papers (VideoSwap, RethinkVQ, DynVideo-E, MaskINT) got accepted by CVPR 2024.
2023 Sept: Two papers (Mix-of-Show, DatasetDM) got accepted by NeurIPS 2023.
2023 Jun: Joined Meta GenAI as a Research Intern in Menlo Park, CA.
2023 July: One paper (Tune-A-Video) got accepted by ICCV 2023.
2022 July: One paper (VQFR) got accepted by ECCV 2022 as Oral.
2022 July: Joined Show Lab @ NUS to start my Ph.D. journey!
2021 Sep: Joined Tencent ARC Lab as a Research Intern.

Publications

	Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, Mike Zheng Shou. Arxiv, 2023 [project] [paper] [code]
	MotionDirector: Motion Customization of Text-to-Video Diffusion Models Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Junhao Zhang, Jiawei Liu, Weijia Wu, Jussi Keppo, Mike Zheng Shou. Arxiv, 2023 [project] [paper] [code]
	VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jiawei Liu, Rui Zhao, Jay Zhangjie Wu, David Junhao Zhang, Mike Zheng Shou, Kevin Tang. IEEE Computer Vision and Pattern Recognition Conference (CVPR), 2024 [project] [paper] [code]
	DynVideo-E: Harnessing Dynamic NeRF for Large-Scale Motion- and View-Change Human-Centric Video Editing Jia-Wei Liu, Yan-Pei Cao, Jay Zhangjie Wu, Weijia Mao, Yuchao Gu, Rui Zhao, Jussi Keppo, Ying Shan, Mike Zheng Shou. IEEE Computer Vision and Pattern Recognition Conference (CVPR), 2024 [project] [paper] [code]
	Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis Yuchao Gu, Xintao Wang, Yixiao Ge, Ying Shan, Xiaohu Qie, Mike Zheng Shou. IEEE Computer Vision and Pattern Recognition Conference (CVPR), 2024 [paper] [code]
	Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge, Ying Shan and Mike Zheng Shou. Neural Information Processing Systems (NeurIPS), 2023 [project] [paper] [code]
	Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie and Mike Zheng Shou. International Conference on Computer Vision (ICCV), 2023 [project] [paper] [code]
	VQFR: Blind Face Restoration with Vector-Quantized Dictionary and Parallel Decoder Yuchao Gu, Xintao Wang, Liangbin Xie, Chao Dong, Gen Li, Ying Shan and Ming-Ming Cheng. European Conference on Computer Vision (ECCV), 2022 Oral (2.7%) [project] [paper] [code]
	iNAS: Integral NAS for Device-Aware Salient Object Detection Yuchao Gu, Shang-Hua Gao, Xu-Sheng Cao, Peng Du, Shao-Ping Lu and Ming-Ming Cheng. IEEE International Conference on Computer Vision (ICCV), 2021 [paper] [code]
	DOTS: Decoupling Operation and Topology in Differentiable Architecture Search Yuchao Gu, Li-Juan Wang, Yun Liu, Yi Yang, Yu-Huan Wu, Shao-Ping Lu and Ming-Ming Cheng. IEEE Computer Vision and Pattern Recognition Conference (CVPR), 2021 [paper] [code]
	Pyramid Constrained Self-Attention Network for Fast Video Salient Object Detection Yuchao Gu, Li-Juan Wang, Ziqin Wang, Yun Liu, Shao-Ping Lu and Ming-Ming Cheng. Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020 [paper] [code]