Tesla's large-scale fleet learning

Tesla has approximately 650,000 Hardware 2 and Hardware 3 cars on the road. Here are the five most important ways that I believe Tesla can leverage its fleet for machine learning:

  1. Automatic flagging of video clips that are rare, diverse, and high-entropy. The clips are manually labelled for use in fully supervised learning for computer vision tasks like object detection. Flagging occurs as a result of Autopilot disengagements, disagreements between human driving and the Autopilot planner when the car is fully manually driven (i.e. shadow mode), novelty detection, uncertainty estimation, manually designed triggers, and deep-learning based queries for specific objects (e.g. bears) or specific situations (e.g. construction zones, driving into the Sun).

  2. Weakly supervised learning for computer vision tasks. Human driving behaviour is used as a source of automatic labels for video clips. For example, with semantic segmentation of free space.

  3. Self-supervised learning for computer vision tasks. For example, for depth mapping.

  4. Self-supervised learning for prediction. The future automatically labels the past. Uploads can be triggered when a HW2/HW3 Tesla’s prediction is wrong.

  5. Imitation learning (and possibly reinforcement learning) for planning. Uploads can be triggered by the same conditions as video clip uploads for (1). With imitation learning, human driving behaviour automatically labels either a video clip or the computer vision system’s representation of the driving scene with the correct driving behaviour. (DeepMind recently reported that imitation learning alone produced a StarCraft agent superior to over 80% of human players. This is a powerful proof of concept for imitation learning.)

(1) makes more efficient/effective use of limited human labour. (2), (3), (4), and (5) don’t require any human labour for labelling and scale with fleet data. Andrej Karpathy is also trying to automate machine learning at Tesla as much as possible to minimize the engineer labour required.

These five forms of large-scale fleet learning are why I believe that, over the next few years, Tesla will make faster progress on autonomous driving than any other company.

Lidar is an ongoing debate. No matter what, robust and accurate computer vision is a must. Not only for redundancy, but also because there are certain tasks lidar can’t help with. For example, determining whether a traffic light is green, yellow, or red. Moreover, at any point Tesla can deploy a small fleet of test vehicles equipped with high-grade lidar. This would combine the benefits of lidar and Tesla’s large-scale fleet learning approach.

Did I miss anything important?

1 Like

This but as a Medium post:

1 Like