# Activation

Edit on Github

Activation functions determine what activation value neurons should get. Depending on your network's environment, choosing a suitable activation function can have a positive impact on the learning ability of the network.

### Methods

Name Graph Equation Derivative
LOGISTIC $f(x) = \frac{1}{1+e^{-x}}$ $f'(x) = f(x)(1 - f(x))$
TANH $f(x) = tanh(x) = \frac{2}{1+e^{-2x}} - 1$ $f'(x) = 1 - f(x)^2$
RELU $f(x) = \begin{cases} 0 & \text{if} & x \lt 0 \\ x & \text{if} & x \ge 0 \end{cases}$ $f'(x) = \begin{cases} 0 & \text{if} & x \lt 0 \\ 1 & \text{if} & x \ge 0 \end{cases}$
IDENTITY $f(x) = x$ $f'(x) = 1$
STEP $f(x) = \begin{cases} 0 & \text{if} & x \lt 0 \\ 1 & \text{if} & x \ge 0 \end{cases}$ $f'(x) = \begin{cases} 0 & \text{if} & x \neq 0 \\ ? & \text{if} & x = 0 \end{cases}$
SOFTSIGN $f(x) = \frac{x}{1+\left\lvert x \right\rvert}$ $f'(x) = \frac{x}{{(1+\left\lvert x \right\rvert)}^2}$
SINUSOID $f(x) = sin(x)$ $f'(x) = cos(x)$
GAUSSIAN $f(x) = e^{-x^2}$ $f'(x) = -2xe^{-x^2}$
BENT_IDENTITY $f(x) = \frac{\sqrt{x^2+1} - 1}{2} + x$ $f'(x) = \frac{ x }{2\sqrt{x^2+1}} + 1$
BIPOLAR $f(x) = \begin{cases} -1 & \text{if} & x \le 0 \\ 1 & \text{if} & x \gt 0 \end{cases}$ $f'(x) = 0$
BIPOLAR_SIGMOID $f(x) = \frac{2}{1+e^{-x}} - 1$ $f'(x) = \frac{(1 + f(x))(1 - f(x))}{2}$
HARD_TANH $f(x) = \text{max}(-1, \text{min}(1, x))$ $f'(x) = \begin{cases} 1 & \text{if} & x \gt -1 & \text{and} & x \lt 1 \\ 0 & \text{if} & x \le -1 & \text{or} & x \ge 1 \end{cases}$
ABSOLUTE1 $f(x) = \left\lvert x \right\rvert$ $f'(x) = \begin{cases} -1 & \text{if} & x \lt 0 \\ 1 & \text{if} & x \ge 0 \end{cases}$
SELU $f(x) = \lambda \begin{cases} x & \text{if} & x \gt 0 \\ \alpha e^x - \alpha & \text{if} & x \le 0 \end{cases}$ $f'(x) = \begin{cases} \lambda & \text{if} & x \gt 0 \\ \alpha e^x & \text{if} & x \le 0 \end{cases}$
INVERSE $f(x) = 1 - x$ $f'(x) = -1$

1 avoid using this activation function on a node with a selfconnection

### Usage

By default, a neuron uses a Logistic Sigmoid as its squashing/activation function. You can change that property the following way:

var A = new Node();
A.squash = methods.activation.<ACTIVATION_FUNCTION>;

// eg.
A.squash = methods.activation.SINUSOID;