DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation

Jiwook Kim*, Seonho Lee*, Jaeyo Shin, Jiho Choi, Hyunjung Shim
Graduate School of Artificial Intelligence, KAIST
* Indiciates Equal Contribution
arXiv Code

Demo Video 1

Demo Video 2

Abstract

Score distillation sampling (SDS) has emerged as an effective framework in text-driven 3D editing tasks due to its inherent 3D consistency. However, existing SDS-based 3D editing methods suffer from extensive training time and lead to low-quality results, primarily because these methods deviate from the sampling dynamics of diffusion models. In this paper, we propose DreamCatalyst, a novel framework that interprets SDS-based editing as a diffusion reverse process. Our objective function considers the sampling dynamics, thereby the optimization process of DreamCatalyst is an approximation of the diffusion reverse process in editing tasks. DreamCatalyst aims to reduce training time and improve editing quality. DreamCatalyst presents two modes: (1) a faster mode, which edits the NeRF scene in only about 25 minutes, and (2) a high-quality mode, which produces superior results in less than 70 minutes. Specifically, our highquality mode outperforms current state-of-the-art NeRF editing methods both in terms of speed and quality.

Architecture of DreamCatalyst

BibTeX

@misc{kim2024dreamcatalystfasthighquality3d,
      title={DreamCatalyst: Fast and High-Quality 3D Editing via Controlling Editability and Identity Preservation}, 
      author={Jiwook Kim and Seonho Lee and Jaeyo Shin and Jiho Choi and Hyunjung Shim},
      year={2024},
      eprint={2407.11394},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.11394}, 
}