Notice that the visual fidelity of the simulator is not of paramount importance when learning to drive. Our simulated world is cartoon-like and nowhere near as artistically detailed as some of the latest driving video games on the market — it does not need to be photorealistic. Content fidelity matters much more than photorealism. However, simulating other agent behaviour effectively is important and remains a big challenge.
The real world is also significantly more diverse and complicated than what we are able to simulate. This is often referred to as the reality gap. In our Dreaming About Driving blog post, we showed that directly building a simulator from real world data can help to bridge the reality gap. But therein lies another issue, a simulator built purely on collected driving data will lack the very edge cases we wish to overcome.