X-Ray for Lateral Access Mechanical Search


Efficiently finding an occluded object with lateral access arises in many contexts such as warehouses, retail, healthcare, shipping, and homes. We introduce LAX-RAY (Lateral Access maXimal Reduction of occupancY support Area), a system to automate the mechanical search for occluded objects on shelves. For such lateral access environments, LAX-RAY couples a perception pipeline predicting a target object occupancy support distribution with a mechanical search policy that sequentially selects occluding objects to push to the side to reveal the target as efficiently as possible. Within the context of extruded polygonal objects and a stationary target with a known aspect ratio, we explore three lateral access search policies: Distribution Area Reduction (DAR), Distribution Entropy Reduction (DER), and Distribution Entropy Reduction over Multiple Time Steps (DER-MT) utilizing the support distribution and prior information. We evaluate these policies using the First-Order Shelf Simulator (FOSS) in which we simulate 800 random shelf environments of varying difficulty, and in a physical shelf environment with a Fetch robot and an embedded PrimeSense RGBD Camera. Average simulation results of 87.3% success rate demonstrate better performance of DER-MT with 2 prediction steps. When deployed on the robot, results show a success rate of at least 80% for all policies, suggesting that LAX-RAY can efficiently reveal the target object in reality. Both results show significantly better performance of the three proposed policies compared to a baseline policy with uniform probability distribution assumption in non-trivial cases, showing the importance of distribution prediction.


September 1st, 2021


  • Michael Danielczuk, UC Berkeley, link
  • Huang Huang, UC Berkeley
  • Jeffrey Ichnowski, UC Berkeley, link
  • Kate Sanders, UC Berkeley
  • Anelia Angelova, Google, link
  • Vincent Vanhoucke, Google, link
  • Ken Goldberg, UC Berkeley, link


See the project website links below for overviews, videos, and paper links. We are also planning on open-sourcing code in the future. The X-Ray paper was  presented at IROS 2020 and the LAX-RAY paper will be presented at IROS 2021.

In future work we are pursuing (1) combinations of grasping and pushing actions using a suction cup gripper and (2) pushing multiple objects and leveraging inter-object contacts, among other ideas.