Brown Digital Repository | Item

Full Metadata

Overview

Year:: 2023
Contributor:: Juliani, Sofia (creator); Wernerfelt, Anneke (creator); Liu, Jason Xinyu (creator); Shah, Ankit (creator); Jia, Mingxi (creator); Jaafar, Ahmed (creator); Tellex, Stefanie (creator); Stefanie Tellex (advisor); Research Experience for Undergraduates (NSF-REU) (research program)
Genre:: posters
Subject:: Natural language processing (Computer science); Robotics; Data sets
Extent:: 1 poster
DOI: https://doi.org/10.26300/2tes-w353

Files

Description

Abstract:: Natural language (NL) is an intuitive way for humans to collaborate with robots, in environments like the home or workplace. Existing models translating NL commands into robot instructions have a high accuracy of following directions to navigate to a landmark. However, more datasets are needed to train a robot to execute tasks by utilizing manipulation and perception skills in addition to navigation. Our approach grounds NL navigation commands to linear temporal logic (LTL) expressions, which represent temporal tasks. We create a new dataset of free-form NL utterances from participants, paired with corresponding LTL translations, so that the model can learn to convert NL instructions to LTL expressions. We will first train our model in a real-world lab setting and then in multiple simulated environments, including factory and home settings, to increase the variety of NL commands in our dataset. We plan to demonstrate how robots can use navigation, manipulation, and perceptions skills to complete tasks. With this new dataset, we hope to enable personal robots to achieve a wider range of tasks in diverse environments.

Content

Access Conditions

Use and Reproduction: All rights reserved
Rights: In Copyright
Restrictions on Use: All Rights Reserved

Citation

Juliani, Sofia, Wernerfelt, Anneke, Liu, Jason Xinyu, et al., "A Dataset of Robotic Task Execution of Natural Language and Vision" (2023). Summer Research Symposium. Brown Digital Repository. Brown University Library. https://doi.org/10.26300/2tes-w353

Relations

Collection:

Summer Research Symposium

Each year, Brown University showcases the research of its undergraduates at the Summer Research Symposium. More than half of the student-researchers are UTRA recipients, while others receive funding from a variety of Brown-administered and national programs and fellowships and go …
...

A Dataset of Robotic Task Execution of Natural Language and Vision