SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting

Xinshuo Weng1, Jianren Wang1, Sergey Levine2, Kris Kitani1, Nick Rhinehart2

1Robotics Institute, Carnegie Mellon University
2Berkeley Artificial Intelligence Research Lab, University of California, Berkeley

One-Sentence Summary

By learning to forecast future LiDAR point clouds, we build a new end-to-end pipeline for perception and prediction by inverting the forecasting module.

Demo Video (5 minute spotlight presentation at ECCVW 2020)


Many autonomous systems must forecast the future in order to make decisions. For example, autonomous driving and robotic manipulation systems often forecast future object poses by leveraging supervised learning techniques. A significant downside to these approaches is the labeling bottleneck – they require labeled sequences of agent poses, which are costly to obtain, especially in 3D space. Can we circumvent this labeling bottleneck? We hypothesize yes, and propose inverting the detect-then-forecast pipeline. Instead of detecting, tracking and then forecasting the objects, we propose to first forecast 3D sensor data (e.g., point clouds with 100k points) and then detect and track objects on the predicted point cloud sequences to obtain future pose trajectories, i.e., a forecast-then-detect pipeline. Part of this work’s focus is on the challenging first step – Sequential Pointcloud Forecasting (SPF), for which we also propose an effective approach, SPFNet. To evaluate our proposed forecast-then-detect pipeline and compare with the detect-then-forecast pipeline, we also propose an evaluation procedure and two metrics. Through experiments on a robotic manipulation dataset and two real autonomous driving datasets, we show that SPFNet is effective for the SPF task and that our forecast-then-detect pipeline outperforms the conventional pipeline.

New Task: Sequential Pointcloud Forecasting (SPF)

Approach: SPFNet


author = {Weng, Xinshuo and Wang, Jianren and Levine, Sergey and Kitani, Kris and Rhinehart Nick}, 
journal = {ECCVW}, 
title = {{4D Forecasting: Sequantial Forecasting of 100,000 Points}}, 
year = {2020} 
author = {Weng, Xinshuo and Wang, Jianren and Levine, Sergey and Kitani, Kris and Rhinehart Nick}, 
journal = {arXiv:2003.08376}, 
title = {{Unsupervised Sequence Forecasting of 100,000 Points for Unsupervised Trajectory Forecasting}}, 
year = {2020} 

Page Views since 08/18/2020