{"id":197,"date":"2018-01-09T10:20:00","date_gmt":"2018-01-09T09:20:00","guid":{"rendered":"https:\/\/kindsonthegenius.com\/blog\/2018\/01\/09\/what-is-an-activation-function-in-neural-networks\/"},"modified":"2020-08-22T11:05:20","modified_gmt":"2020-08-22T09:05:20","slug":"what-is-an-activation-function-in-neural-networks","status":"publish","type":"post","link":"https:\/\/kindsonthegenius.com\/blog\/what-is-an-activation-function-in-neural-networks\/","title":{"rendered":"What is an Activation Function in Neural Networks"},"content":{"rendered":"
\n
Activation Functions play a very important role in Neural Network so understanding them is key to getting a clearer understanding on how neural networks work.<\/p>\n

What is an Activation Function? <\/b>
An activation function in neural network is a function the that takes the input or the set of inputs to a node and produces an output that is within a particular range say  between 0 and 1(in the case of binary activation function). The input to an activation function could be a formula or even an output of another function.<\/p>\n

The Logistic Function<\/b>
The Logistic function a  type of sigmoid function that takes an input and produces an output between 0 and 1. The formula for the logistic function is given below:<\/p>\n

<\/a><\/div>\n

<\/p>\n

<\/a><\/div>\n

On problem with the logistic function is the vanishing gradient problem. This means that when a neurons activation approaches the limits of either 0 or 1, the gradient at that point gets very close to 0.
Another problem with the logistic function is that it is not zero-centered as well as the problem of slow convergence<\/p>\n

The Hyperbolic Tangent Function<\/b>
The Hyperbolic tangent is another type of sigmoid function that takes an input and produces an output between -1 and +1. Since the output of the Hyperbolic tangent function is 0-centere, it is preferred to the logistic function. But just like the logistic function, it also suffers from the vanishing gradient problem. The formula for the hyperbolic tangent function is given below<\/p>\n

<\/a><\/div>\n

<\/p>\n

<\/a><\/div>\n

The Rectified Linear Unit<\/b>
The ReLU is an activation function in neural networks defined as the positive part of its argument. The ReLU is defined as<\/p>\n

R(x) = max(0,x)<\/span><\/i><\/span><\/div>\n

 <\/span><\/i><\/span>
this means that the value is is zero when x is less than zero and linear with a slope of 1 when x is greater than zero. It was noted that it had a 6 x improvement over the the hyperbolic tangent function in a paper by Alex Krizhevsky on ImageNet Classification.
The formula for the rectified linear function is given as:<\/p>\n

<\/a><\/div>\n

Other variants of the ReLU function are the Leaky Rectified Linear Unit (Leaky ReLU), the Parametric ReLU and the Exponential Linear unit. I would recommend you do some personal research on these and other activation function.<\/p>\n

<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

Activation Functions play a very important role in Neural Network so understanding them is key to getting a clearer understanding on how neural networks work. … <\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_mi_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0},"categories":[11,16,14],"tags":[],"_links":{"self":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/197"}],"collection":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/comments?post=197"}],"version-history":[{"count":1,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/197\/revisions"}],"predecessor-version":[{"id":1468,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/posts\/197\/revisions\/1468"}],"wp:attachment":[{"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/media?parent=197"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/categories?post=197"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kindsonthegenius.com\/blog\/wp-json\/wp\/v2\/tags?post=197"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}