Detecting novelty

I’ve been thinking about what a company like Tesla should do about object detection and other computer vision tasks when you have much more data coming through your sensors than you can 1) collect and 2) label. You can of course look for all the things you know to look for. But what about the things you don’t know to look for? A paper about a potential solution to that problem:

Variational Autoencoder for End-to-End Control of Autonomous Driving with Novelty Detection and Training De-biasing

Abstract—This paper introduces a new method for end-to-end training of deep neural networks (DNNs) and evaluates it in the context of autonomous driving. DNN training has been shown to result in high accuracy for perception to action learning given sufficient training data. However, the trained models may fail without warning in situations with insufficient or biased training data. In this paper, we propose and evaluate a novel architecture for self-supervised learning of latent variables to detect the insufficiently trained situations. Our method also addresses training data imbalance, by learning a set of underlying latent variables that characterize the training data and evaluate potential biases. We show how these latent distributions can be leveraged to adapt and accelerate the training pipeline by training on only a fraction of the total dataset. We evaluate our approach on a challenging dataset for driving. The data is collected from a full-scale autonomous vehicle. Our method provides qualitative explanation for the latent variables learned in the model. Finally, we show how our model can be additionally trained as an end-to-end con- troller, directly outputting a steering control command for an autonomous vehicle.

1 Like

Another related paper:

Towards Corner Case Detection for Autonomous Driving

Abstract—The progress in autonomous driving is also due to the increased availability of vast amounts of training data for the underlying machine learning approaches. Machine learning systems are generally known to lack robustness, e.g., if the training data did rarely or not at all cover critical situations. The challenging task of corner case detection in video, which is also somehow related to unusual event or anomaly detection, aims at detecting these unusual situations, which could become critical, and to communicate this to the autonomous driving system (online use case). Such a system, however, could be also used in offline mode to screen vast amounts of data and select only the relevant situations for storing and (re)training machine learning algorithms. So far, the approaches for corner case detection have been limited to videos recorded from a fixed camera, mostly for security surveillance. In this paper, we provide a formal definition of a corner case and propose a system framework for both the online and the offline use case that can handle video signals from front cameras of a naturally moving vehicle and can output a corner case score.

The data stream:

  1. Sense — Information is received by a vehicle’s sensors (such as photons, sound waves, inertia, or steering angle).

  2. Filter — Software triggers determine what data is collected.

  3. Collect — Data is saved to memory and later uploaded.

  4. Label — Data that isn’t automatically labelled, and that is used in supervised learning, is reviewed by humans and given a label.