Jawaharlal Nehru Technological University Hyderabad B.Tech IV NEURAL NETWORKS AND DEEP LEARNING Previous Question Paper (September - 2020 - R16)

Activation functions introduce non-linearity into artificial neural networks, allowing them to learn complex patterns beyond simple linear regression. The choice of activation function significantly impacts training stability and model performance.

1. Sigmoid (Logistic) Function

This function maps inputs to a range between 0 and 1. It is primarily used in the output layer for binary classification tasks. However, it suffers from the vanishing gradient problem, where gradients become extremely small for large positive or negative inputs, hindering deep network training.

2. Hyperbolic Tangent (tanh) Function

The tanh function maps inputs to a range between -1 and 1. It is zero-centered, which helps in faster convergence compared to the sigmoid function. Like sigmoid, it also faces the vanishing gradient problem in very deep networks.

3. Rectified Linear Unit (ReLU)

ReLU is the most widely used activation function for hidden layers. It outputs zero for negative inputs and the input itself for positive values. It is computationally efficient and mitigates the vanishing gradient problem. A limitation is the 'dying ReLU' problem, where neurons become inactive and output zero for all inputs if they fall into the negative range during training.

4. Leaky ReLU

Leaky ReLU addresses the dying ReLU problem by allowing a small, non-zero gradient when the input is negative. This ensures that neurons remain active and continue to learn even for negative inputs.

5. Softmax Function

Softmax is used exclusively in the output layer for multi-class classification problems. It converts a vector of raw scores into a probability distribution, where each value is between 0 and 1 and the sum of all values equals 1.

Suitability Summary:

• Hidden Layers: ReLU is the standard choice due to speed and gradient flow; Leaky ReLU is preferred if dying neurons are observed.
• Binary Classification: Sigmoid is standard for the output layer.
• Multi-class Classification: Softmax is standard for the output layer.
• Recurrent Neural Networks: Tanh is often preferred due to its zero-centered nature, which helps maintain stability in sequential data processing.

A Continuous Hopfield Network is a recurrent neural network model where the activation states are continuous variables, typically ranging between 0 and 1 (or -1 and 1), rather than binary. Unlike discrete models, the neurons in a continuous network update asynchronously using a sigmoid activation function, which allows the network to represent graded values.

Key characteristics include:

• Energy Function: The network possesses a Lyapunov energy function that is guaranteed to decrease or remain constant during state transitions, ensuring convergence to a local minimum.
• Dynamics: The state evolution is governed by differential equations of the form:
  duᵢ/dt = -uᵢ/τ + Σ(wᵢⱼvⱼ) + Iᵢ
  where vᵢ = f(uᵢ) is the sigmoid output.
• Stability: The network converges to stable fixed points that correspond to the local minima of the energy landscape.

To solve the Traveling Salesman Problem (TSP), the network is mapped to a 2D grid of neurons where rows represent cities and columns represent the order of visitation. The objective is to minimize the total path length while satisfying constraints.

The energy function is constructed to include penalty terms for constraints and a term for the path cost:

1. Constraint Terms: High energy penalties are added if more than one city is visited at a time, or if a city is visited more than once.
2. Optimization Term: A term proportional to the sum of distances between consecutive cities in the tour is added.
3. Minimization: As the network evolves, the continuous activations settle into a state that minimizes the total energy, effectively finding a valid tour with a short total distance.

While the continuous Hopfield network provides a heuristic approach to the NP-hard TSP, it is sensitive to parameter initialization and may converge to suboptimal local minima rather than the global optimum.