Advances in AGV perception for people and object detection in industrial warehouses

Abstract

The advances in computer vision are meeting many market requirements in sectors such as automotive, security surveillance and industry 4.0. This thesis presents a research on applications of novel computer vision methods in automated warehouses. The idea is to design the next generation of Automated Guided Vehicles (AGVs), that should be able to perceive the surrounding environment to improve productivity and safety in an intelligent way. The main contributions are: the development of a multi-robot multiple camera system for people detection and tracking; a comparison of deep learning models for 2D pallet detection; a self-supervised method for category-level 6D pose estimation with geometric consistency using optical flow. The first contribution is the introduction of a multi-robot system for people detection and tracking in automated warehouses. Each Automated Guided Vehicle is equipped with multiple RGB cameras that can track the workers' current locations on the floor thanks to a neural network that provides human pose estimation. Based on the local perception of the environment, each AGV can exploit information about the tracked people for self-motion planning or collision avoidance. Additionally, data collected from each robot contribute to a global people detection and tracking system. A warehouse central management software fuses information received from all AGVs into a map of the current locations of workers. The estimated locations of workers are sent back to the AGVs to prevent potential collisions. The proposed method is based on a two-level hierarchy of Kalman filters. Experiments performed in a real warehouse show the viability of the proposed approach. A second contribution is related to the problem of automatic pallet detection in industrial environments using a single RGB camera. The problem is relevant in the context of Autonomous Guided Vehicle navigation, and for tasks like pallet storage and retrieval. In particular, an approach based on a convolutional neural network (CNN) followed by a decision-making step is presented. The convolutional neural network is trained to recognize two elements of a pallet, namely the pallet front side and its pockets. Also, a comparative study is carried out between three state-of-the-art CNNs: Faster R-CNN, SSD and YOLOv4. For training and evaluation, a dataset was collected in a warehouse. The dataset contains images of pallets in different configurations, either on the ground or on racks, and with arbitrary orientation. Overall, results indicate that Faster R-CNN and SSD perform better than YOLOv4. Lastly, the problem of category-level 6D object pose estimation with deep learning was investigated using benchmark datasets. Category-level 6D object pose estimation aims at determining the pose of an object of a given category. Most current state-of-the-art methods require a significant amount of real training data to supervise their models. Moreover, annotating the 6D pose is very time consuming, error-prone, and it does not scale well to a large amount of object classes. Therefore, a handful of methods have recently been proposed to use unlabelled data to establish weak supervision. The proposed approach leverages the 2D optical flow as a proxy for supervising the 6D pose. To this purpose, the 2D optical flow between consecutive frames based on the 6D pose estimation is computed. Then the framework harnesses an off-the-shelf optical flow method to enable weak supervision using a 2D-3D optical flow based consistency loss. Experiments show that the proposed approach for self-supervised learning yields state-of-the-art performance on the NOCS benchmark, and it reaches comparable results with some fully-supervised approaches.