Tsinghua-Tencent 100K is a training and testing dataset for traffic sign recognition using 100,000 high-resolution panoramic images from Tencent Street View:
Some interesting findings:
There are about 10,000 annotated images that include traffic signs.
About 6,600 of these images are used for training, and about 3,300 are used for testing.
Human performance has not been tested on the Tsinghua-Tencent 100K dataset, but it has been tested on the German Traffic Sign Recognition Benchmark (GTSRB) at 99.22% (CCR). Even in 2011, convolutional neural networks outperformed this score with an accuracy of 99.46%.
The GTSRB is a very different dataset from the Tsinghua-Tencent 100K (TTK) dataset. It’s hard to say which is likely to be harder for humans, even though GTSRB is clearly easier for neural networks. A lot of the images in the GTSRB are blurry or overexposed.
If we guess that human accuracy on TTK is 99.9% or 1 error in 1,000 examples, we can guess how much improvement is needed by neural networks to surpass humans. 92.82% accuracy is about 1 error in 14 examples. A 75x improvement would be 1 error in 1,042 examples, or 99.904% accuracy.
If we instead guess humans will have 99.5% accuracy on TTK, or make one error per 200 examples, then neural networks are only 15x away. A 15x improvement would mean 1 error per 208 examples, or 99.52% accuracy.
On this point, the most interesting fact to me is that 92.82% accuracy was achieved by training on only around 6,600 images. There is no reason a neural network used in an autonomous car couldn’t be trained on tens of millions of labelled images. A 3,000x increase in training data is well within reason.
I may be getting this part wrong, but I believe a bigger neural network could be used for traffic sign recognition as well. The neural network that achieved 92.92% accuracy on TTK is based on Faster R-CNN, which has around 100 million parameters. The winner of the 2017 ImageNet challenge, SENet-154, has 146 million parameters. ResNeXt-101 32x48d has 861 million parameters.