PersonaCraft: Personalized Full-Body Image Synthesis for Multiple Identities from Single References Using 3D-Model-Conditioned Diffusion

Gwanghyun Kim1*, Suh Yoon Jeon1*, Seunggyu Lee1, Se Young Chun1, 2
1Dept. of Electrical and Computer Engineering, 2Interdisciplinary Program in Artificial Intelligence
*Authors contributed equally.
Seoul National University, Korea
Paper arXiv Code

We propose PersonaCraft, a novel approach that integrates diffusion models with 3D human modeling to address occlusions and enable full-body personalization for multiple individuals in a scene. Furthermore, we introduce a novel feature for user-defined body shape control.


Abstract

Personalized image generation has been significantly advanced, enabling the creation of highly realistic and customized images. However, existing methods often struggle with generating images of multiple people due to occlusions and fail to accurately personalize full-body shapes. In this paper, we propose PersonaCraft, a novel approach that combines diffusion models with 3D human modeling to address these limitations. Our method effectively manages occlusions by incorporating 3D-aware pose conditioning with SMPLx-ControlNet and accurately personalizes human full-body shapes through SMPLx fitting. Additionally, PersonaCraft enables user-defined body shape adjustments, adding flexibility for individual body customization. Experimental results demonstrate the superior performance of PersonaCraft in generating high-quality, realistic images of multiple individuals while resolving occlusion issues, thus establishing a new standard for multi-person personalized image synthesis.



Method

Overview of PersonaCraft. We extract face and body embeddings from reference images, then perform personalized image synthesis using controllable modules for face identity and body shape customization.

SMPLx-ControlNet (SCNet) leverages 3D-aware pose conditioning using SMPLx models to accurately capture body shape and pose.


Comparision

Comparison of PersonaCraft with baselines. Yellow arrows show anatomical inconsistencies in poses and occlusions. Pink arrows highlight identity mixing or duplication in baselines. Blue arrows refer to the people for checking correct body shape preservation.PersonaCraft excels in face identity, body shape consistency, and naturalness compared to the baselines.


User-Defined Body Shape Control

Result of PersonaCraft's user-defined body shape control. (a) Reference-based body shape control. (b) Interpolation and extrapolation-based control, where γ controls the interpolation or extrapolation between the reference body shapes..


PersonaCraft with Style LoRAs