Skip to page navigation menu Skip entire header
Brown University
Skip 13 subheader links

A Dataset of Natural Language Commands for Robotic Tasks


A longstanding goal in robotics research is having competent robot assistants that understand natural language (NL) commands for everyday tasks. In existing work, robots convert temporal commands to linear temporal logic (LTL) expressions, a form of logic which uses temporal operators. They use learned reasoning skills to carry out sequential tasks where the location of the target object is unknown. However, there’s a lack of comprehensive datasets to train robots on long-horizon temporal tasks which rely on temporal and spatial reasoning; existing datasets do not combine navigation, manipulation, and perception skills all together, and also lack diversity in the types of environments they use. In this study, we compile NL commands annotated to LTL representing a range of complex tasks that users might ask a robot. We began with a pilot study that uses navigation in the lab environment. We plan to collect more data in a wide range of real and simulated environments in order to expand the settings in which robotic assistants can thrive. We hope this work will lead to better reasoning ability in robots for executing a broader range of tasks.

Access Conditions

Use and Reproduction
All rights reserved
In Copyright
Restrictions on Use
All Rights Reserved


Wernerfelt, Anneke, and Juliani, Sofia; Liu, Jason Xinyu; Shah, Ankit; Jia, Mingxi; Tellex, Stefanie, "A Dataset of Natural Language Commands for Robotic Tasks" (2023). Summer Research Symposium. Brown Digital Repository. Brown University Library.



  • Summer Research Symposium

    Each year, Brown University showcases the research of its undergraduates at the Summer Research Symposium. More than half of the student-researchers are UTRA recipients, while others receive funding from a variety of Brown-administered and national programs and fellowships and go …