{"id":662,"date":"2019-03-25T17:34:52","date_gmt":"2019-03-25T17:34:52","guid":{"rendered":"https:\/\/tutorials.retopall.com\/?p=662"},"modified":"2019-03-26T18:28:43","modified_gmt":"2019-03-26T18:28:43","slug":"activation-functions-neural-networks","status":"publish","type":"post","link":"https:\/\/tutorials.retopall.com\/index.php\/2019\/03\/25\/activation-functions-neural-networks\/","title":{"rendered":"Activation Functions Neural Networks"},"content":{"rendered":"\n<p>Activations functions are widely used in almost every type of <a href=\"https:\/\/tutorials.retopall.com\/index.php\/2019\/02\/17\/neural-networks\/\">Neural Networks<\/a>. <a href=\"https:\/\/tutorials.retopall.com\/index.php\/2019\/03\/01\/feed-forward-neural-network\/\">Feed-forward neural network<\/a>, <a href=\"https:\/\/tutorials.retopall.com\/index.php\/2019\/02\/24\/convolutional-neural-networks\/\">Convolutional Neural Networks<\/a> and Recurrent Neural Networks use activation functions.<\/p>\n\n\n\n<p>There are a lot of different activation functions that could be used for different purposes.<\/p>\n\n\n\n<p>An activation function is used to separate active and inactive neurons depending on a rule or function.  Depending on their status they will modify their values or not.<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"601\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-7-1024x601.png\" alt=\"\" class=\"wp-image-674\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-7-1024x601.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-7-300x176.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-7-768x451.png 768w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>We will see a wide range of activation functions:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Identity<\/li><li>Binary Step<\/li><li>Logistic (Soft Step)<\/li><li>Hyperbolic Tangent<\/li><li>ReLU <\/li><li>ELU<\/li><li>SoftPlus<\/li><li>SoftMax<\/li><\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">IDENTITY<\/h4>\n\n\n\n<p>This activation function will not produce any change to the input value. The output will be the same.<\/p>\n\n\n\n<p>$$f(x)=x$$<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-3-1024x417.png\" alt=\"\" class=\"wp-image-663\" width=\"877\" height=\"357\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-3-1024x417.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-3-300x122.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-3-768x313.png 768w\" sizes=\"auto, (max-width: 877px) 100vw, 877px\" \/><\/figure><\/div>\n\n\n\n<p>The derivative of the identity is:<\/p>\n\n\n\n<p> $$f'(x)=1$$ <\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mce_28\">BINARY STEP<\/h4>\n\n\n\n<p>It&#8217;s a simple fragmented function.<\/p>\n\n\n\n<p>  $$f(x)= \\left\\{ \\begin{array}{lcc}<br>  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0 &amp; &nbsp; if &amp; x &lt; 0 \\\\<br>  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\\\\   1 &amp; &nbsp;if &nbsp;&amp; x \\geq 0  <br><br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\\end{array}<br>  \\right.$$ <\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"569\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-8-1024x569.png\" alt=\"\" class=\"wp-image-675\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-8-1024x569.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-8-300x167.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-8-768x427.png 768w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mce_28\">LOGISTIC<\/h4>\n\n\n\n<p>This function will be a step function but with an inferior and superior limit. This function is widely used for <strong>Feed-Forward Neural Networks<\/strong><\/p>\n\n\n\n<p>$$f(x)=\\frac{1}{1+e^{-x}}$$<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-4-1024x675.png\" alt=\"\" class=\"wp-image-664\" width=\"616\" height=\"405\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-4-1024x675.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-4-300x198.png 300w\" sizes=\"auto, (max-width: 616px) 100vw, 616px\" \/><\/figure><\/div>\n\n\n\n<p> The derivative of the logistic function is: <\/p>\n\n\n\n<p>$$f'(x)=f(x)(1-f(x))$$<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mce_28\">HYPERBOLIC TANGENT<\/h4>\n\n\n\n<p>It&#8217;s very similar to the logistic function but it will tend to -1 in the $-\\infty$ and to 1 in  $\\infty$ .<\/p>\n\n\n\n<p> $$f(x)=\\frac{2}{1+e^{-2x}}-1$$ <\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"569\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-4-1-1024x569.png\" alt=\"\" class=\"wp-image-665\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-4-1-1024x569.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-4-1-300x167.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-4-1-768x427.png 768w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p> The derivative of the hyperbolic tangent is:<\/p>\n\n\n\n<p>$$f'(x)=1-f(x)^2 $$<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mce_28\">ReLU (Rectifier Linear Unit)<\/h4>\n\n\n\n<p>This is used almost every network. In the negative X axis the result will be 0 and in the right positive X axis the value will be mantained<\/p>\n\n\n\n<p> $$f(x)= \\left\\{ \\begin{array}{lcc}<br>  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;0 &amp; &nbsp; if &amp; x &lt; 0 \\\\<br>  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\\\\   x &amp; &nbsp;if &nbsp;&amp; x \\geq 0  <br>  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  <br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\\end{array}<br>  &nbsp;\\right. $$<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter is-resized\"><img loading=\"lazy\" decoding=\"async\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-1-1-1024x601.png\" alt=\"\" class=\"wp-image-670\" width=\"819\" height=\"479\"\/><\/figure><\/div>\n\n\n\n<p>We have also one variant of this function called <strong>PReLU<\/strong>:<\/p>\n\n\n\n<p> $$f(x)= \\left\\{ \\begin{array}{lcc}<br>  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; \\alpha x  &amp; &nbsp; if &amp; x &lt; 0 \\\\<br>  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\\\\  x &amp; &nbsp;if &nbsp;&amp; x \\geq 0  <br><br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\\end{array}<br>  &nbsp;\\right.$$ <\/p>\n\n\n\n<p>Where $\\alpha$ is a parameter to change the slope of the function<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"569\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-2-1-1024x569.png\" alt=\"\" class=\"wp-image-678\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-2-1-1024x569.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-2-1-300x167.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-2-1-768x427.png 768w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mce_28\">ELU (Exponential Linear Unit)<\/h4>\n\n\n\n<p>Similar to ReLU but with an exponential curve in the negative X axis.<\/p>\n\n\n\n<p> $$f(x)= \\left\\{ \\begin{array}{lcc}<br>  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\\alpha(e^x-1) &amp; &nbsp; if &amp; x &lt; 0 \\\\<br>  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\\\\   x &amp; &nbsp;if &nbsp;&amp; x \\geq 0  <br><br>&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;\\end{array}<br>  &nbsp;\\right. $$ <\/p>\n\n\n\n<p>Where $\\alpha$ is a parameter to edit the curve. It&#8217;s recommended a value similar to 1<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"569\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-5-1024x569.png\" alt=\"\" class=\"wp-image-672\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-5-1024x569.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-5-300x167.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-5-768x427.png 768w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mce_28\">SOFTPLUS<\/h4>\n\n\n\n<p>In this function the Y value always increases.<\/p>\n\n\n\n<p>$$f(x)=\\log_e{(1+e^x)}$$<\/p>\n\n\n\n<figure class=\"wp-block-image\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"569\" src=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-6-1-1024x569.png\" alt=\"\" class=\"wp-image-673\" srcset=\"https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-6-1-1024x569.png 1024w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-6-1-300x167.png 300w, https:\/\/tutorials.retopall.com\/wp-content\/uploads\/2019\/03\/geogebra-export-6-1-768x427.png 768w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"mce_30\">SOFTMAX<\/h4>\n\n\n\n<p>Finally we have the softmax function that is mainly used for the output layers specially in the Convolutional Neural Network.<\/p>\n\n\n\n<p>It&#8217;s explained in the <a href=\"https:\/\/tutorials.retopall.com\/index.php\/2019\/02\/24\/convolutional-neural-networks\/\">Convolutional Neural Network<\/a> tutorial in the Output layer section.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p>To get more information about activation functions I recommend this tutorial: <a href=\"https:\/\/towardsdatascience.com\/activation-functions-neural-networks-1cbd9f8d91d6\">https:\/\/towardsdatascience.com\/activation-functions-neural-networks-1cbd9f8d91d6<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Activations functions are widely used in almost every type of Neural Networks. Feed-forward neural network, Convolutional Neural Networks and Recurrent Neural Networks use activation functions. [&#8230;]<\/p>\n","protected":false},"author":1,"featured_media":676,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[9,5,46],"tags":[64,103,107,105,104,108,53,63,52,102,106],"class_list":["post-662","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence","category-explanation","category-neural-networks","tag-activation-function","tag-active-neuron","tag-elu","tag-hyperbolic-function","tag-inactive-neuron","tag-logistic","tag-relu","tag-sigmoid","tag-softmax","tag-softplus","tag-tanh"],"_links":{"self":[{"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/posts\/662","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/comments?post=662"}],"version-history":[{"count":20,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/posts\/662\/revisions"}],"predecessor-version":[{"id":702,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/posts\/662\/revisions\/702"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/media\/676"}],"wp:attachment":[{"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/media?parent=662"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/categories?post=662"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tutorials.retopall.com\/index.php\/wp-json\/wp\/v2\/tags?post=662"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}