HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes

NeurIPS 2022

Zan Wang1,2

Yixin Chen2

Tengyu Liu2

Yixin Zhu3*

Wei Liang1,4*

Siyuan Huang2*

* indicates corresponding authors

1School of Computer Science & Technology, Beijing Institute of Technology
2Beijing Institute for General Artificial Intelligence (BIGAI)
3Institute for Artificial Intelligence, Peking University
4Yangtze Delta Region Academy of Beijing Institute of Technology, Jiaxing

Paper  |  arXiv  |  Code  |  Data


Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human-Scene Interaction(HSI); they only have limited scale/quality and lack semantics. To fill in the gap, we propose a large-scale and semantic-rich synthetic HSI dataset, denoted as HUMANISE, by aligning the captured human motion sequences with various 3D indoor scenes. We automatically annotate the aligned motions with language descriptions that depict the action and the unique interacting objects in the scene; e.g., sit on the armchair near the desk. HUMANISE thus enables a new generation task, language-conditioned human motion generation in 3D scenes. The proposed task is challenging as it requires joint modeling of the 3D scene, human motion, and natural language. To tackle this task, we present a novel scene-and-language conditioned generative model that can produce 3D human motions of the desirable action interacting with the specified objects. Our experiments demonstrate that our model generates diverse and semantically consistent human motions in 3D scenes.

TL;DR:  We propose a large-scale and semantic-rich human-scene interaction dataset, HUMANISE. It has language description for each human-scene interaction. HUMANISE enables a new task: language-conditioned human motion generation in 3D scenes.

Dataset Gallery

1. walk

walk to the desk walk to the couch

2. Sit

sit on the couch sit on the armchair that is far away from the door

3. Stand up

stand up from the chair that is farthest from the laptop stand up from the coffee table

4. Lie

lie on the bed lie on the bed

5. Place something (supplementary action type)

place something on the coffee table place something on the table that is in the center of the armchair and the kitchen counter

6. Knock (supplementary action type)

knock on the door knock on the door

7. Open (supplementary action type)

open the refrigerator open the cabinet that is in the middle of the shower and the door


If you find our project useful, please consider citing us:

  title={HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes},
  author={Wang, Zan and Chen, Yixin and Liu, Tengyu and Zhu, Yixin and Liang, Wei and Huang, Siyuan},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},