Completed

Scene Sketch to Photo Synthesis

Overview

Online shopping has been made easier with Augmented Reality. Through Amazon’s View in your room, you can see how products fit and look in your home before you bring them home. We propose to develop scene understanding techniques that could lead to even more advanced shopping features such as: virtually redecorating a home by removing or replacing existing objects, suggesting items based on the room style, suggesting different furniture layout styles.

Specifically, we leverage our expertise on 2D and 3D scene parsing, sketch-to-photo synthesis, and...

Smart Practice: Learning to Practice Skills from Demonstrations

In order to solve complex tasks, intelligent agents learn to perform a variety of skills. These skills abstract key information necessary for agents to act in real-world environments with high-dimensional state and action spaces. The goal of the Smart Practice project is to develop learning algorithms that...

Combining Causal Reasoning and Information Theory to Empower Humans through Human-Agent Collaboration


We seek to create new...

Knowledge Transferable Bayesian Optimization

Abstract

The need for automated optimization has become very important in many domains including hyper-parameter tuning in ML or in manufacturing industry. In practice, one frequently has to solve similar optimization problems for a specific customized setting, e.g. manufacturing robots optimized for a new customer environment or hyper-parameter optimization for a new classification task....

Autonomous Skill Discovery Through Self-Supervised Exploration

Unsupervised Skill Discovery

Abstract

We introduce Contrastive Intrinsic Control (CIC) - an algorithm for unsupervised skill discovery that maximizes the mutual information between skills and state transitions. In contrast to most prior approaches, CIC uses a decomposition of the mutual information that explicitly incentivizes diverse behaviors by...

Better Visual Representations through Language Supervision

Overview

CLIP [1] demonstrates the power of simple contrastive learning on a large-scale dataset of image and caption pairs collected directly from the internet, without expensive manual labeling. In our project we seek to improve the data efficiency of CLIP and performance when trained on uncurated datasets, as well as explore additional capabilities beyond classification.

Data Efficiency

CLIP achieves impressive results when trained on WIT400M, a dataset of 400M image and caption pairs collected by searching for a curated list of...

Automatic Curriculum Generation and Emergent Complexity via Inter-agent Competition

Reinforcement Learning (RL) has been most successful when agents can collect extensive training experience in a simulated environment [1-4]. However, building simulated environments requires a great deal of manual effort, is error prone, and is unlikely to cover the space of all real world tasks. Inter-agent competition has...

On Theoretical Foundations for Transfer Learning

We aim to provide statistical guarantees for transfer learning in sequential settings under dynamic feedback. This can enable...

Towards Trustworthy Reinforcement Learning

...

Unsupervised Learning of Visual Context from Instance Segmentation

Unsupervised representation learning aims to extract latent information from data that reflects their semantic categories. Contrastive learning has emerged as a direct winning alternative to self-supervised learning. We take a step further and propose to extract latent information that also reflects their visual context.

Overview

Instead of...