Sketch-guided Cage-based 3D Gaussian Splatting Deformation

Tianhao Xie1, Noam Aigerman2, Eugene Belilovsky1,3, Tiberiu Popa1,
1Concordia University, 2Université de Montréal, 3MILA
Arxiv
teaser image.

Deformations of 3D Gaussian splats (GS) using our sketch-guided deformation method. Given a 3D GS scene, the user can deform the 3D GS by drawing a deformed silhouette sketch of a single view. (A) and (B) show examples with synthetic data and real-world large-scale data respectively. We can also produce an animation of a of 3D GS by few keyframe sketches, as shown in (C).

Abstract

3D Gaussian Splatting (GS) is one of the most promising novel 3D representations that has received great interest in computer graphics and computer vision. While various systems have introduced editing capabilities for 3D GS, such as those guided by text prompts, fine-grained control over deformation remains an open challenge. In this work, we present a novel sketch-guided 3D GS deformation system that allows users to intuitively modify the geometry of a 3D GS model by drawing a silhouette sketch from a single viewpoint. Our approach introduces a new deformation method that combines cage-based deformations with a variant of Neural Jacobian Fields, enabling precise, fine-grained control. Additionally, it leverages large-scale 2D diffusion priors and ControlNet to ensure the generated deformations are semantically plausible. Through a series of experiments, we demonstrate the effectiveness of our method and showcase its ability to animate static 3D GS models as one of its key applications.

Method

method image.

Results

Source
Deformed
Sketch
Image description
Source
Deformed
Sketch
Image description
Image description
Image description
Image description
Image description
Image description
Image description
Image description
Image description

Acknowledgement

The source code of this project is based on some amazing projects:

3D Gaussian Splatting

DreamGaussian4D

TextDeformer

Neural Jacobian Fields