Descripció del projecte
Recently, the state of the art in oblect reconstruction from multiple images has been set using methods based on neural fields and differentiable rendering [1]. In particular, these methods have been applied to obtain high quality reconstructions of human heads [2] and bodies [3]. While these methods can yield impressive reconstructions, their performance is conditioned by the quality of certain inputs to the algorithm, such as segmentation masks and camera parameters. Since these assets are commonly predicted using deep learning models, the quality of these intermediate models becomes the bottleneck that determines the quality of the entire system.
In this thesis, we will address the question of building robust 3D reconstruction pipelines, acknowledging and addressing the imperfections that exist in intermediate steps (pose regression, segmentation…). Specifically, we will apply these pipelines to obtain 3D reconstructions of humans (head and body). Adopting a system perspective, we intend to improve the final 3D reconstructions by matching the strengths of the auxiliary neural networks to the requirements of the full pipeline. In addition, we will implement strategies to attenuate the impact of intermediate errors in the quality of the final reconstructions. To achieve this, we will take advantage of the capacity of Crisalix to capture, annotate and process large amounts of data involving human faces and bodies. The outcome of this thesis will be integrated in the company product, an intelligent tool to assist professionals in plastic and cosmetic surgery.
This project will be carried out in cooperation with the following:
– Crisalix, in Barcelona
– Institut de Robotica i Informatica Industrial (IRI) in Universitat Politêcnica de Catalunya (UPC)
The scientific advisor at IRl will be Dr. Jordi Sanchez Riera
The responsible person on the company side will be Dr. Gil Triginer Garcés.
[1] L. Yariv et al, Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
[2] E. Ramon et al. H3D-Net: Few-Shot High-Fidelity 3D Head Reconstruction, ICCV 2021
[3] ICON: Implicit clothed humans obtained from normals, CVPR 2022. Th
