ProcTHOR

ProcTHOR enables Embodied AI to scale by orders of magnitude by procedurally generating interactive 3D environments.

ProcTHOR uses procedural generation to sample massively diverse, realistic, interactive, customizable, and performant 3D environments to train simulated embodied agents. Here is an example of sampling virtual home environments.

HOUSE SELECTION
PROPERTY TYPE
1
2
3
4
Drag to Rotate
PIN LOCATION
1
2
3
4

Customizability. ProcTHOR can construct custom scene types, such as classrooms, libraries, and offices.

Classroom

Classroom Environment

Library

Library Environment

Office

Office Environment

Results. Pre-training with ProcTHOR improves downstream performance. Zero-Shot performance, from models pre-trained on ProcTHOR, often beats the same models trained on the training data from the benchmark it is evaluated on.

RoboTHOR ObjectNav

Semantic Object Navigation in Dorm Styled, 3D Artist-Designed Scenes

RoboTHOR ObjectNav Task
20% Success Rate
80%
Baseline trained on RoboTHOR
Zero-shot trained on ProcTHOR
Pre-Trained on ProcTHOR, fine-tuned on RoboTHOR

Habitat ObjectNav

Semantic Object Navigation in Multi-Floor, 3D Matterport Scanned Scenes

Habitat ObjectNav Task
0% SPL
40%
Baseline trained on Habitat
Zero-shot trained on ProcTHOR
Pre-Trained on ProcTHOR, fine-tuned on Habitat

AI2-iTHOR ObjectNav

Semantic Object Navigation in Room-Sized, 3D Artist-Designed Scenes

AI2-iTHOR ObjectNav Task
55% Success Rate
85%
Baseline trained on AI2-iTHOR
Zero-shot trained on ProcTHOR
Pre-Trained on ProcTHOR, fine-tuned on AI2-iTHOR

ArchitecTHOR ObjectNav

Semantic Object Navigation in Single-Floor House-Sized, 3D Artist-Designed Scenes

ArchitecTHOR ObjectNav Task
10% Success Rate
50%
Baseline trained on AI2-iTHOR
Zero-shot trained on ProcTHOR

AI2-THOR Rearrangement

Interactive Rearrangement in Room-Sized, 3D Artist-Designed Scenes

AI2-THOR Rearrangement Task
10% Percentage-Fixed Strict
35%
Baseline trained on AI2-THOR
Zero-shot trained on ProcTHOR
Pre-Trained on ProcTHOR, fine-tuned on AI2-THOR

ManipulaTHOR ArmPointNav

Interactive Manipulation in Room-Sized, 3D Artist-Designed Scenes

ManipulaTHOR ArmPointNav Task
10% Success Rate
50%
Baseline trained on ManipulaTHOR
Zero-shot trained on ProcTHOR

Scale Improves Performance. Scaling the number of training houses consistently improves zero-shot performance.

Floorplan Diversity. ProcTHOR houses sample extremely diverse floorplans. Here are examples of sampled floorplans with between 1 and 10 rooms.

Bedroom
Bathroom
Kitchen
Living Room
...

Object Diversity. ProcTHOR includes 1,633 interactive household objects across 108 categories. A small subset of these objects is shown below.

...

Material Augmentation. ProcTHOR includes 3,278 materials that can be used to visually augment objects, walls, floors, and ceilings.

SCENE VIEW

Object Placement. ProcTHOR houses sample realistic and diverse placement of objects.

HOUSE SELECTION

Lighting Variation. Environment lighting can be rendered with significantly variation to simulate real-world lighting conditions at any time of day.

TIME OF DAY
1
2
3
Drag to Rotate
PIN LOCATION
1
2
3

Interactivity. ProcTHOR objects are highly interactive, supporting object state changes, robotic arm manipulation, and multi-agent interaction.

OBJECT STATE CHANGES

ROBOTIC ARM MANIPULATION

MULTI-AGENT INTERACTION

Get Started. ProcTHOR is fully open-source and available to the Embodied AI community. We are excited to see what you build!

We have provided a Google Colab notebook to get started using ProcTHOR-10K. More comprehensive documentation, code, and examples will be available after CVPR 2022.

Team. ProcTHOR was created by the PRIOR team at the Allen Institute for AI.

Matt Deitke
Eli VanderBilt
Alvaro Herrasti
Luca Weihs
Jordi Salvador
Kiana Ehsani
Winson Han
Eric Kolve
Ali Farhadi
Ani Kembhavi
Roozbeh Mottaghi