We seek to create new collaboration strategies between a human and an artificial agent in which the agent enhances the human’s capabilities through actions that increase the causal leverage, or empowerment to influence the environment, of both robot and human. That is, the agent will act to increase its own, as well as the human’s, future options. The key motivations behind this strategy are the inherent safety considerations – the agent will not limit the human’s actions – and the ability to provide goal-agnostic, seamless assistance. We will also explore assistance that transitions between goal-oriented and empowerment-based behaviors, depending on the agent’s confidence in the human’s goals.
Update
Researchers
- Yuqing Du, UC Berkeley
- Stas Tiomkin, UC Berkeley
- Emre Kiciman, Microsoft
- Daniel Polani, University of Hertfordshire
- Pieter Abbeel, UC Berkeley
- Anca Dragan, UC Berkeley
Overview
AvE: Assistance via Empowerment
One difficulty in using artificial agents for human-assistive applications lies in the challenge of accurately assisting with a person’s goal(s). Existing methods tend to rely on inferring the human’s goal, which is challenging when there are many potential goals or when the set of candidate goals is difficult to identify. We propose a new paradigm for assistance by instead increasing the human’s ability to control their environment, and formalize this approach by augmenting reinforcement learning with human empowerment. This task-agnostic objective increases the person’s autonomy and ability to achieve any eventual state. We test our approach against assistance based on goal inference, highlighting scenarios where our method overcomes failure modes stemming from goal ambiguity or misspecification. As existing methods for estimating empowerment in continuous domains are computationally hard, precluding its use in real time learned assistance, we also propose an efficient empowerment-inspired proxy metric. Using this, we are able to successfully demonstrate our method in a shared autonomy user study for a challenging simulated teleoperation task with human-in-the-loop training.