A Bin-Picking Benchmark for Systematic Evaluation of Robotic Pick-and-Place Systems

Pick-and-place operations constitute the majority of today’s industrial robotic applications. However, comparability and reproducibility of results has remained an issue that delays further advances in this field. Evaluation of manipulation systems can be carried out at different levels, but for the final application the performance of the overall system is the critical one. This paper proposes a benchmarking framework for pick-and-place tasks, inspired by a typical task in the logistic domain: picking up fruits and vegetables from a container and placing them in an order bin. The framework uses an easy-to-reproduce environment, a publicly available object set, and guidelines for creating scenarios of different complexity. The proposed benchmark is applied to evaluate the performance of four variants of a robotic system with different end-effector