Internal Covariate Shift and Batch Normalization– Part-2

Contains.
Batch Normalization
Differentiability of ‘Batch Normalization’
Discussion on Merits and Demerits of ‘Batch Normalization’
Reference:
[1]. Ioffe, S., & Szegedy, C. (2015, June). Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning (pp. 448-456). pmlr.
[2]. Awais, M., Iqbal, M. T. B., & Bae, S. H. (2020). Revisiting internal covariate shift for batch normalization. IEEE Transactions on Neural Networks and Learning Systems, 32(11), 5082-5092.
[3]. Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., & Bethge, M. (2020). Improving robustness against common corruptions by covariate shift adaptation. Advances in Neural Information Processing Systems, 33, 11539-11551.