Which activation function passes positives and blocks negatives, aiding learning inside the network?

Which activation function passes positives and blocks negatives, aiding learning inside the network?

• A. ReLU
• B. Sigmoid
• C. Tanh
• D. Linear

Answer: (A)

ReLU works by outputting the input when it’s positive and producing zero when it’s negative, so it passes positives and blocks negatives. This simple rule—f(x) = max(0, x)—introduces a nonlinearity without squashing positive values, which helps the network learn more complex patterns while keeping gradients from vanishing as long as the input is positive. When x > 0, the derivative is 1, so gradients flow freely and learning speeds up. For negative inputs, the output is zero and the gradient is also zero, creating sparse activations that can improve efficiency and representation. In contrast, sigmoid and tanh compress large magnitudes toward their tails, causing vanishing gradients that slow learning, and a linear activation doesn’t introduce nonlinearity at all. Hence, ReLU best matches the behavior described.

ReLU works by outputting the input when it’s positive and producing zero when it’s negative, so it passes positives and blocks negatives. This simple rule—f(x) = max(0, x)—introduces a nonlinearity without squashing positive values, which helps the network learn more complex patterns while keeping gradients from vanishing as long as the input is positive. When x > 0, the derivative is 1, so gradients flow freely and learning speeds up. For negative inputs, the output is zero and the gradient is also zero, creating sparse activations that can improve efficiency and representation. In contrast, sigmoid and tanh compress large magnitudes toward their tails, causing vanishing gradients that slow learning, and a linear activation doesn’t introduce nonlinearity at all. Hence, ReLU best matches the behavior described.