Anomaly Detection in Track Scenes

Within the sector initiative “Digitale Schiene Deutschland”, our client Deutsche Bahn is developing an automated driving system for trains. As a part of the efforts towards such a system we developed, together with Deutsche Bahn, a machine learning solution to detect anomalous and hazardous objects on and around the tracks using onboard RGB cameras. It is intentionally required that this system does not simply detect objects within a given collection of classes (such as people, signals or vehicles), but rather has the ability to detect any object and rank them by how anomalous they are. This presentation explains the challenges encountered, presents several approaches explored, and provides an overview of the final solution: In order to detect objects of possibly unkown classes we developed a unique pipeline containing multiple machine learning components, including a monocular depth estimation model, a segmentation stage, image embedding models and an anomaly detection model. As dataset, Digitale Schiene Deutschland provides us with OSDAR23, an open dataset that contains 45 scenes. Each scene contains images taken by several RGB cameras and infrared cameras, together with radar and lidar data. This dataset contains annotations for twenty classes of objects, which we use both for finetuning our model and for evaluating the final results. Besides, we were also granted access to a larger amount of unannotated data, which were used for self-supervised learning.