Swish.pdf

The choice of activation functions in deep networks significantly affects training dynamics and task performance, with Rectified Linear Unit (ReLU) being the most successful and widely-used. However, alternatives to ReLU have been proposed, and a new activation function called Swish, defined as f(x) = x · sigmoid(x), has been introduced. Swish tends to work better than ReLU on deeper models across challenging datasets, improving top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2. Swish is smooth, non-monotonic, and has properties that make it easy to replace ReLUs in any neural network. Its simplicity and similarity to ReLU allow for easy implementation, and it can be trained with slightly lowered learning rates. Swish's properties, including being unbounded above, bounded below, non-monotonic, and smooth, are advantageous, and it consistently matches or outperforms ReLU on deep networks applied to various domains.