The central problem we aim to address is how we can leverage both the expressiveness power of deep neuralmodels and the robustness, scalability, and explainability of traditional algorithms that take advantage of structural reasoning. Focusing on the tasks of image inpainting and face retargeting, we demonstrate that with carefully designed frameworks, it is possible to combine the strength of both worlds. Firstly, for image inpainting, we adapt the idea of patch match and apply it to deep neural features. Secondly, for face retargeting, our prior knowledge about the face structure as defined by the facial landmarks can ease the difficulty of disentangling various latent
factors such as identity and expressions. Finally, we conclude that combining deep learning models with structural knowledge and reasoning, we can not only alleviate the “date-hungry” issue of deep neural networks but can also encourage the output to be consistent with the human understanding of the real world.


Chao “Harry” Yang is a PhD student in computer science at University of Southern California. His research interests are deep generative models and their computer vision applications, such as image inpainting, image translation and human retargeting. He is especially interested in developing new methods that combine the power of deep learning and human reasoning. He has published in top-tier computer vision conferences such as CVPR, ICCV and ECCV. In particular, his work of image inpainting using neural patch synthesis is published as one of the first deep-learning based image inpainting papers, and has attracted wide interests from the research community. He has authored multiple open-source projects, and many of them are extensively starred and forked. He has interned at Adobe, Butterfly and Facebook, and has been invited to give talks at CalTech, CMU and Harvard. He is a sub-finalist of the entrepreneurial computer vision challenge at LDV vision summit 2017, New York. Other than doing research and building machine learning systems, he enjoys reading, hiking and traveling to different places of the world.