self training with noisy student improves imagenet classification

Self-training with noisy student improves imagenet classification, in: Proceedings of the IEEE/CVF Conference on Computer . This paper presents a unique study of transfer learning with large convolutional networks trained to predict hashtags on billions of social media images and shows improvements on several image classification and object detection tasks, and reports the highest ImageNet-1k single-crop, top-1 accuracy to date. We evaluate our EfficientNet-L2 models with and without Noisy Student against an FGSM attack. Do imagenet classifiers generalize to imagenet? We use the same architecture for the teacher and the student and do not perform iterative training. On ImageNet, we first train an EfficientNet model on labeled images and use it as a teacher to generate pseudo labels for 300M unlabeled images. However, during the learning of the student, we inject noise such as dropout, stochastic depth and data augmentation via RandAugment to the student so that the student generalizes better than the teacher. In our experiments, we use dropout[63], stochastic depth[29], data augmentation[14] to noise the student. There was a problem preparing your codespace, please try again. This model investigates a new method for incorporating unlabeled data into a supervised learning pipeline. Not only our method improves standard ImageNet accuracy, it also improves classification robustness on much harder test sets by large margins: ImageNet-A[25] top-1 accuracy from 16.6% to 74.2%, ImageNet-C[24] mean corruption error (mCE) from 45.7 to 31.2 and ImageNet-P[24] mean flip rate (mFR) from 27.8 to 16.1. A tag already exists with the provided branch name. In all previous experiments, the students capacity is as large as or larger than the capacity of the teacher model. Then, that teacher is used to label the unlabeled data. We iterate this process by putting back the student as the teacher. Infer labels on a much larger unlabeled dataset. (using extra training data). We investigate the importance of noising in two scenarios with different amounts of unlabeled data and different teacher model accuracies. supervised model from 97.9% accuracy to 98.6% accuracy. The inputs to the algorithm are both labeled and unlabeled images. Noisy Student Training is based on the self-training framework and trained with 4 simple steps: Train a classifier on labeled data (teacher). We then train a larger EfficientNet as a student model on the A common workaround is to use entropy minimization or ramp up the consistency loss. This result is also a new state-of-the-art and 1% better than the previous best method that used an order of magnitude more weakly labeled data [ 44, 71]. Scaling width and resolution by c leads to c2 times training time and scaling depth by c leads to c times training time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. Stochastic Depth is a simple yet ingenious idea to add noise to the model by bypassing the transformations through skip connections. Noisy Student self-training is an effective way to leverage unlabelled datasets and improving accuracy by adding noise to the student model while training so it learns beyond the teacher's knowledge. Noisy Student Training is a semi-supervised training method which achieves 88.4% top-1 accuracy on ImageNet Our largest model, EfficientNet-L2, needs to be trained for 3.5 days on a Cloud TPU v3 Pod, which has 2048 cores. Train a classifier on labeled data (teacher). Noisy StudentImageNetEfficientNet-L2state-of-the-art. Lastly, we trained another EfficientNet-L2 student by using the EfficientNet-L2 model as the teacher. Our study shows that using unlabeled data improves accuracy and general robustness. Our main results are shown in Table1. If nothing happens, download Xcode and try again. We have also observed that using hard pseudo labels can achieve as good results or slightly better results when a larger teacher is used. A number of studies, e.g. Summarization_self-training_with_noisy_student_improves_imagenet_classification. By clicking accept or continuing to use the site, you agree to the terms outlined in our. (or is it just me), Smithsonian Privacy Self-Training With Noisy Student Improves ImageNet Classification Abstract: We present a simple self-training method that achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. We obtain unlabeled images from the JFT dataset [26, 11], which has around 300M images. We vary the model size from EfficientNet-B0 to EfficientNet-B7[69] and use the same model as both the teacher and the student. The paradigm of pre-training on large supervised datasets and fine-tuning the weights on the target task is revisited, and a simple recipe that is called Big Transfer (BiT) is created, which achieves strong performance on over 20 datasets. https://arxiv.org/abs/1911.04252, Accompanying notebook and sources to "A Guide to Pseudolabelling: How to get a Kaggle medal with only one model" (Dec. 2020 PyData Boston-Cambridge Keynote), Deep learning has shown remarkable successes in image recognition in recent years[35, 66, 62, 23, 69]. mCE (mean corruption error) is the weighted average of error rate on different corruptions, with AlexNets error rate as a baseline. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger, Y. Huang, Y. Cheng, D. Chen, H. Lee, J. Ngiam, Q. V. Le, and Z. Chen, GPipe: efficient training of giant neural networks using pipeline parallelism, A. Iscen, G. Tolias, Y. Avrithis, and O. The top-1 accuracy is simply the average top-1 accuracy for all corruptions and all severity degrees. As a comparison, our method only requires 300M unlabeled images, which is perhaps more easy to collect. One might argue that the improvements from using noise can be resulted from preventing overfitting the pseudo labels on the unlabeled images. Self-training with Noisy Student improves ImageNet classification Original paper: https://arxiv.org/pdf/1911.04252.pdf Authors: Qizhe Xie, Eduard Hovy, Minh-Thang Luong, Quoc V. Le HOYA012 Introduction EfficientNet ImageNet SOTA EfficientNet . This way, the pseudo labels are as good as possible, and the noised student is forced to learn harder from the pseudo labels. Using Noisy Student (EfficientNet-L2) as the teacher leads to another 0.8% improvement on top of the improved results. Probably due to the same reason, at =16, EfficientNet-L2 achieves an accuracy of 1.1% under a stronger attack PGD with 10 iterations[43], which is far from the SOTA results. Chum, Label propagation for deep semi-supervised learning, D. P. Kingma, S. Mohamed, D. J. Rezende, and M. Welling, Semi-supervised learning with deep generative models, Semi-supervised classification with graph convolutional networks. We then train a larger EfficientNet as a student model on the combination of labeled and pseudo labeled images. This invariance constraint reduces the degrees of freedom in the model. Self-Training achieved the state-of-the-art in ImageNet classification within the framework of Noisy Student [1]. E. Arazo, D. Ortego, P. Albert, N. E. OConnor, and K. McGuinness, Pseudo-labeling and confirmation bias in deep semi-supervised learning, B. Athiwaratkun, M. Finzi, P. Izmailov, and A. G. Wilson, There are many consistent explanations of unlabeled data: why you should average, International Conference on Learning Representations, Advances in Neural Information Processing Systems, D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. Raffel, MixMatch: a holistic approach to semi-supervised learning, Combining labeled and unlabeled data with co-training, C. Bucilu, R. Caruana, and A. Niculescu-Mizil, Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, Y. Carmon, A. Raghunathan, L. Schmidt, P. Liang, and J. C. Duchi, Unlabeled data improves adversarial robustness, Semi-supervised learning (chapelle, o. et al., eds. Then by using the improved B7 model as the teacher, we trained an EfficientNet-L0 student model. Self-training with Noisy Student improves ImageNet classificationCVPR2020, Codehttps://github.com/google-research/noisystudent, Self-training, 1, 2Self-training, Self-trainingGoogleNoisy Student, Noisy Studentstudent modeldropout, stochastic depth andaugmentationteacher modelNoisy Noisy Student, Noisy Student, 1, JFT3ImageNetEfficientNet-B00.3130K130K, EfficientNetbaseline modelsEfficientNetresnet, EfficientNet-B7EfficientNet-L0L1L2, batchsize = 2048 51210242048EfficientNet-B4EfficientNet-L0l1L2350epoch700epoch, 2EfficientNet-B7EfficientNet-L0, 3EfficientNet-L0EfficientNet-L1L0, 4EfficientNet-L1EfficientNet-L2, student modelNoisy, noisystudent modelteacher modelNoisy, Noisy, Self-trainingaugmentationdropoutstochastic depth, Our largest model, EfficientNet-L2, needs to be trained for 3.5 days on a Cloud TPU v3 Pod, which has 2048 cores., 12/self-training-with-noisy-student-f33640edbab2, EfficientNet-L0EfficientNet-B7B7, EfficientNet-L1EfficientNet-L0, EfficientNetsEfficientNet-L1EfficientNet-L2EfficientNet-L2EfficientNet-B75. Self-Training with Noisy Student Improves ImageNet Classification [68, 24, 55, 22]. Here we study how to effectively use out-of-domain data. combination of labeled and pseudo labeled images. Their noise model is video specific and not relevant for image classification. Code for Noisy Student Training. Their framework is highly optimized for videos, e.g., prediction on which frame to use in a video, which is not as general as our work. The comparison is shown in Table 9. [76] also proposed to first only train on unlabeled images and then finetune their model on labeled images as the final stage. For simplicity, we experiment with using 1128,164,132,116,14 of the whole data by uniformly sampling images from the the unlabeled set though taking the images with highest confidence leads to better results. Work fast with our official CLI. We present Noisy Student Training, a semi-supervised learning approach that works well even when labeled data is abundant. We apply dropout to the final classification layer with a dropout rate of 0.5. The method, named self-training with Noisy Student, also benefits from the large capacity of EfficientNet family. , have shown that computer vision models lack robustness. . Finally, in the above, we say that the pseudo labels can be soft or hard. It has three main steps: train a teacher model on labeled images use the teacher to generate pseudo labels on unlabeled images For more information about the large architectures, please refer to Table7 in Appendix A.1. On ImageNet-C, it reduces mean corruption error (mCE) from 45.7 to 31.2. Self-Training With Noisy Student Improves ImageNet Classification Abstract: We present a simple self-training method that achieves 88.4% top-1 accuracy on ImageNet, which is 2.0% better than the state-of-the-art model that requires 3.5B weakly labeled Instagram images. International Conference on Machine Learning, Learning extraction patterns for subjective expressions, Proceedings of the 2003 conference on Empirical methods in natural language processing, A. Roy Chowdhury, P. Chakrabarty, A. Singh, S. Jin, H. Jiang, L. Cao, and E. G. Learned-Miller, Automatic adaptation of object detectors to new domains using self-training, T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen, Probability of error of some adaptive pattern-recognition machines, W. Shi, Y. Gong, C. Ding, Z. MaXiaoyu Tao, and N. Zheng, Transductive semi-supervised deep learning using min-max features, C. Simon-Gabriel, Y. Ollivier, L. Bottou, B. Schlkopf, and D. Lopez-Paz, First-order adversarial vulnerability of neural networks and input dimension, Very deep convolutional networks for large-scale image recognition, N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting. Use Git or checkout with SVN using the web URL. The main difference between our method and knowledge distillation is that knowledge distillation does not consider unlabeled data and does not aim to improve the student model. The accuracy is improved by about 10% in most settings. The learning rate starts at 0.128 for labeled batch size 2048 and decays by 0.97 every 2.4 epochs if trained for 350 epochs or every 4.8 epochs if trained for 700 epochs. ; 2006)[book reviews], Semi-supervised deep learning with memory, Proceedings of the European Conference on Computer Vision (ECCV), Xception: deep learning with depthwise separable convolutions, K. Clark, M. Luong, C. D. Manning, and Q. V. Le, Semi-supervised sequence modeling with cross-view training, E. D. Cubuk, B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le, AutoAugment: learning augmentation strategies from data, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, RandAugment: practical data augmentation with no separate search, Z. Dai, Z. Yang, F. Yang, W. W. Cohen, and R. R. Salakhutdinov, Good semi-supervised learning that requires a bad gan, T. Furlanello, Z. C. Lipton, M. Tschannen, L. Itti, and A. Anandkumar, A. Galloway, A. Golubeva, T. Tanay, M. Moussa, and G. W. Taylor, R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel, ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, J. Gilmer, L. Metz, F. Faghri, S. S. Schoenholz, M. Raghu, M. Wattenberg, and I. Goodfellow, I. J. Goodfellow, J. Shlens, and C. Szegedy, Explaining and harnessing adversarial examples, Semi-supervised learning by entropy minimization, Advances in neural information processing systems, K. Gu, B. Yang, J. Ngiam, Q.