SyncTweedies: A General Generative Framework Based on Synchronized Diffusions

NeurIPS 2024

Jaihoon Kim*, Juil Koo*, Kyeongmin Yeo* Minhyuk Sung

KAIST
(* denotes equal contribution.)
PDF arXiv Code
architecture

Abstract

We introduce a general diffusion synchronization framework for generating diverse visual content, including ambiguous images, panorama images, 3D mesh textures, and 3D Gaussian splats textures, using a pretrained image diffusion model. We first present an analysis of various scenarios for synchronizing multiple diffusion processes through a canonical space. Based on the analysis, we introduce a novel synchronized diffusion method, SyncTweedies, which averages the outputs of Tweedie’s formula while conducting denoising in multiple instance spaces. Compared to previous work that achieves synchronization through finetuning, SyncTweedies is a zero-shot method that does not require any finetuning, preserving the rich prior of diffusion models trained on Internet-scale image datasets without overfitting to specific domains. We verify that SyncTweedies offers the broadest applicability to diverse applications and superior performance compared to the previous state-of-the-art for each application.



3D Mesh Texturing

🎬 3D Mesh


"A dumpster"

"A clutch bag"

"A lemon"

"A hand carved wood turtle"


🎬 Qualitative Results

"A nascar"

"A hamburger"

"An hourglass"

"A jeep"


🎨 Luma AI 3D Mesh Re-Texturing

"A turtle"

âž¡

"A golden statue
of a turtle"

"A car"

âž¡

"A luxurious
red sports car"

"A lantern"

âž¡

"A chinese style lantern"

"A nascar"

âž¡

"A car with graffiti"

"An elephant"

âž¡

"An african elephant"

"An axe"

âž¡

"A wooden axe"


3D Gaussian Splat Texturing

🎬 Qualitative Results


"A majestic red chair"

"A photo of cucumbers"

"A photo of a yellow excavator covered in snow"

"A photo of a white cruise ship at sea"

"A leather chair"

"A photo of corns"

"A white drum kit"

"A photo of a pirate ship at sea"


Ambiguous Images

🎬 Qualitative Results

Clockwise 90° Rotation

Color Inversion

Patch Permutation


Panorama Generation

🎬 Qualitative Results

"A photo of a mountain range at twilight"

"A photo of a beautiful ocean with coral reef"

"A photo of a lake under the northern lights"


Depth-to-360-Panorama Generation

🎬 Qualitative Results

"A house at night"

"An old looking library"

"A room that has been painted gold"


💡 Comparison with Other Methods

🚀 3D Mesh Texturing

Case1

Case2

(SyncTweedies)

Case3

Case4

Case5

Paint-it

TEXTure

Text2Tex

"Baseball glove"

Case1

Case2

(SyncTweedies)

Case3

Case4

Case5

Paint-it

TEXTure

Text2Tex

"Minivan"

Case1

Case2

(SyncTweedies)

Case3

Case4

Case5

Paint-it

TEXTure

Text2Tex

"iPod"

Case1

Case2

(SyncTweedies)

Case3

Case4

Case5

Paint-it

TEXTure

Text2Tex

"Pigeon"


🚀 3D Gaussian Splat Texturing

Case2

(SyncTweedies)

Case5

SDS

IN2N

"A photo of a tree with multicolored leaves"

Case2

(SyncTweedies)

Case5

SDS

IN2N

"A photo of a wooden carving of a microphone"

Case2

(SyncTweedies)

Case5

SDS

IN2N

"A photo of an intricate wooden carving of a ship"

Case2

(SyncTweedies)

Case5

SDS

IN2N

"A photo of a purple chair"

Case2

(SyncTweedies)

Case5

SDS

IN2N

"A photo of carrots"

Case2

(SyncTweedies)

Case5

SDS

IN2N

"A photo of a tree covered in snow"


BibTeX

@article{Kim2024SyncTweedies,
title = {SyncTweedies: A General Generative Framework Based on Synchronized Diffusions},
author = {Kim, Jaihoon and Koo, Juil and Yeo, Kyeongmin and Sung, Minhyuk},
year = {2024},
journal = {arXiv:2403.14370},
}