Activation Functions in Deep Learning

Activation functions are crucial in neural networks as they introduce non-linearity into the model. Without these functions, neural networks would essentially behave like linear regression models, regardless of their complexity.

An activation function determines whether a neuron should be activated or not. It takes the input from the previous layer, applies a mathematical operation, and passes the result to the next layer. This process allows the network to learn complex patterns in the data.

1. Sigmoid Function

The sigmoid function maps any real-valued number into the range between 0 and 1.

2. Hyperbolic Tangent (tanh)

The tanh function maps real values to the range between -1 and 1. It is often preferred over the sigmoid function.

3. Rectified Linear Unit (ReLU)

ReLU is defined as f(x) = max(0, x). It has become the default activation function for many neural networks due to its simplicity and effectiveness.

4. Leaky ReLU

Leaky ReLU allows a small, non-zero gradient when the unit is not active, defined as f(x) = x if x > 0 else 0.01 * x.

5. Softmax Function

Softmax is typically used in multi-class classification problems. It outputs a probability distribution over multiple classes.

Sigmoid Function Implementation

import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

# Example usage
print(sigmoid(0))  # Output: 0.5
                

ReLU Function Implementation

def relu(x):
    return np.maximum(0, x)

# Example usage
print(relu(-5))  # Output: 0
                

• Use ReLU for hidden layers in deep networks.
• Use Sigmoid or Softmax for the output layer in binary classification.
• Monitor for vanishing gradients when using Sigmoid or tanh.
• Consider using Leaky ReLU to mitigate the dying ReLU problem.