In order for agents trained by deep reinforcement learning to help humans in realistic settings, we will need to ensure that the agents are robust. In collaborative scenarios with humans, evaluating robustness to tail failure cases is nontrivial, as adversarial training (and evaluation) is too conservative of...