Learning 3D features from 2D images with Generative Models

AI Technology: Vision (image recognition, image generation, text recognition OCR, etc.)

The project's research focuses on advancing 3D image generation, a field with vast potential in virtual reality, movies, robotics simulations, and autonomous driving. 

While 2D image generation has reached a mature stage, 3D generation poses greater challenges, particularly in data collection, which is expensive and often limited to lab environments with specialized hardware. This project aims to develop innovative methods for accurate 3D reconstruction and generation from 2D images, addressing the limited information they provide.

The team will explore two generative models: 3D-aware Generative Adversarial Networks (GANs) and diffusion models trained on large datasets. 3D-aware GANs learn 3D features directly from data, while diffusion models leverage strong image priors for high-quality 3D generation. 

The research objectives include:

1. Learning 3D Features Encoder for 3D-aware GAN Models: Developing an encoder to estimate 3D structures from single image inputs using innovative training strategies and novel loss objectives.
2. Diverse Reconstruction of 3D Textured Models: Achieving high-fidelity reconstructions with realistic variations in invisible parts, using autoregressive models and variational inference.
3. Controlled Generations of 3D Textured Models: Generating novel 3D objects with style image and text-based controls, employing CLIP embedders and multi-diffusion schemes for consistent, high-quality outputs.