06 / DEEP LEARNING ARCHIVE · BENNETT UNIVERSITY

Neural Networks

Jared Wilber & AIS Research Cohort · May 2024 · 12 min read

A VISUAL INTRODUCTION

Neural networks have revolutionized artificial intelligence and are the backbone of popular algorithms today, such as ChatGPT, Stable-Diffusion, and many others. In this visual introduction, we'll journey through the fundamentals of feed-forward neural networks, starting from their essential components, understanding their learning mechanisms, and even getting hands-on experience by interacting with one ourselves.

WHAT IS A NETWORK?

Neural networks are networks - that much is clear. But what is a “network”? A network is a structure consisting of interconnected computational nodes, or 'neurons', arranged in layers. These nodes perform mathematical operations on input data, learning some underlying patterns in the data, before producing some output based on those patterns.

FIGURE 01 · COMPUTATIONAL GRAPHFEED-FORWARD TOPOLOGY

*Input signals propagate forward through the mathematical kernelFORWARD GRAPH

01 / FOUNDATIONS

Building Blocks: Computational Graphs

A computational graph has an input node where data is fed into the graph, a function node where the input data is processed, and an output node where the result of the computation is produced.

As we can see, data flows in one direction, from inputs to output, through the graph. These simple directional flows form the foundational algebra of every deep learning network.

02 / PARAMETERS & SLIDERS

Weights: Scaling Input Signals

To give the network capacity to learn and adapt, each incoming connection wire is parameterized by a real-valued weight ( $w$ ). The function node calculates a weighted sum of inputs:

z = w_1 x_1 + w_2 x_2

Weights act like precision volume knobs: positive weights (Steel Teal) amplify aligned features, while negative weights (Terracotta) dampen contradictory inputs before they reach the nucleus.

FIGURE 02 · WEIGHT MODULATIONz = 1.60

Weight w₁:1.4

Weight w₂:-0.8

03 / LAYERS OF COMPUTATION

Activation Functions

As you can see, neural networks are not so complicated! They are just computational graphs, channeling inputs through successive layers of computation to generate outputs. This process of inference, whereby inputs are fed through the network to produce output predictions, is called the forward pass. Let's talk more about those layers of computation, the activation functions.

Activation functions are at the heart of artificial neurons in a neural network. These crucial components introduce non-linearity into the model, transforming the weighted inputs to generate an output. Simply put, an activation function decides how much signal to pass onto the next layer based on the input it receives. This idea of chaining many weighted signals together is what allows neural networks to learn very complex relationships.

The non-linear nature of these functions is essential for neural networks to learn from complex data. If we only used linear activation functions, no matter how many layers we stacked, the network would behave just like a single-layer perceptron because the composition of linear functions is still a linear function. Non-linear activation functions enable the network to learn complex patterns and solve intricate problems by adding layers of abstraction.

LIVE COMPUTATIONAL GRAPH & FEATURE SPACE SIMULATOR

Select Activation Function & Watch The Output Feature Space Change

FIGURE 03 · ACTIVATION FUNCTION SPACESMODE: SIGMOID

INPUT SENSORSCOMPUTATION OPERATOROUTPUT SPACE (ŷ)

Adjust Boundary Parameters in Feature Space:

Decision Boundary Angle / Tilt:35°

Decision Threshold / Bias Offset:0

Four Popular Activation Functions

There's a diverse range of activation functions utilized in neural networks, each with its unique benefits and applications. Here are four foundational activation functions:

Name	Mathematical Function	Description
Sigmoid (logistic)	$f(x) = \frac{1}{1 + e^{-x}}$	Squashes input into calibrated range $(0, 1)$ .
Hyperbolic Tangent (tanh)	$f(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$	Zero-centered output squashed into $(-1, 1)$ .
Rectified Linear Unit (ReLU)	$f(x) = \begin{cases} 0 & \text{if } x < 0 \\ x & \text{if } x \ge 0 \end{cases}$	Only keeps positive values; fast and prevents saturation.
Step Function (Perceptron)	$f(x) = \begin{cases} +1 & \text{if } x \ge 0 \\ -1 & \text{if } x < 0 \end{cases}$	Binary step: neuron fires ( $+1$ ) or remains silent ( $-1$ ).

The sigmoid (or logistic) function, which ranges from 0 to 1, is particularly useful in the output layer of binary classification models, representing the probability of a binary event. However, it can suffer from the vanishing gradients problem during backpropagation.

The hyperbolic tangent (tanh) function, which ranges from -1 to 1, provides a zero-centered output designed to make learning for the next layer easier. Yet, like sigmoid, it also faces vanishing gradients at extreme values.

The Rectified Linear Unit (ReLU) function is the ubiquitous choice in hidden layers due to computational efficiency and gradient propagation. It activates a node linearly if its input is positive; otherwise, it outputs zero.

Despite their simplicity, chaining these functions together in a neural network yields powerful geometric representations. The best choice depends on data distributions, computational constraints, and network depth.

04 / TOPOLOGY & LOOPING SIGNALS

Multi-Layer Perceptron (MLP)

We expand the network by stacking intermediate 'hidden' layers between input sensors and output classifiers. Each hidden neuron becomes a specialized feature detector.

Notice the continuous transmission wave running in an infinite loop on the canvas: signals propagate rhythmically across the bipartite synaptic mesh, progressively assembling low-level edges into high-level concepts.

FIGURE 04 · BIPARTITE SYNAPSE MESHLIVE SIGNAL LOOP

*Loop animation actively simulates rhythmic activation voltage propagation.

FIGURE 05 · LIVE NUMERICAL COMPUTEPrediction ŷ = 0.820

Input Sensor x₁:0.8

Input Sensor x₂:-0.4

05 / FORWARD PASS

The Forward Pass: Live Calculations

During the forward pass, numerical vectors enter at the left sensors. Each hidden neuron takes the linear combination of all inputs, applies the activation function, and exposes its internal activation.

Adjust the sensor sliders on the left: watch the internal activations inside h₁, h₂, and h₃ recalculate immediately, culminating in the terminal prediction $\hat{y}$ .

06 / HOW NETWORKS LEARN

Backpropagation: How Networks Learn

Now that we've grasped the concept of activation functions and their role in neural networks, it's time to understand how these networks learn. The magic behind this learning process is a technique known as backpropagation.

Backpropagation is an algorithm used during the training of neural networks. The goal of backpropagation is to update the weights so that the Neural Network makes better predictions. Specifically, backpropagation will calculate the gradient of the loss function with respect to the weights of the network, updating the weights layer-by-layer to minimize the network's prediction error.

Notice the reverse loop on the right: the error signal travels backward from the Loss node, flowing through the multivariable chain rule to update every weight parameter in the graph.

FIGURE 06 · BACKWARD FEEDBACK LOOP∂L/∂w GRADIENT ARCS

*Curved feedback loops convey continuous chain-rule error signals flowing backward to update weights.

07 / BACKPROPAGATION TRAINING LOOP SIMULATOR

The Full Training Loop in Motion

Every epoch in deep learning executes an unbroken circular dance: Forward Pass sends data through synaptic weights and activations, Loss Evaluation compares output predictions against target ground truth, and Backpropagation streams multivariable chain rule gradients in reverse to update every synapse before repeating the next training iteration.

Interact with the real-time simulation below: press Play to witness continuous gradient descent fitting the regression curve, or click Animate Epoch to trigger a step-by-step 4-phase walkthrough of gradient propagation.

INITIALIZING BACKPROPAGATION ENGINE

MATHEMATICAL FORMULATION: MULTIVARIABLE CHAIN RULE

To evaluate weight updates across successive layers, the chain rule decomposes the total derivative of the loss with respect to any arbitrary connection weight $w_{ij}^{(l)}$ :

\frac{\partial \mathcal{L}}{\partial w_{ij}^{(l)}} = \frac{\partial \mathcal{L}}{\partial a^{(L)}} \cdot \frac{\partial a^{(L)}}{\partial z^{(L)}} \cdot \prod_{k=l+1}^{L-1} \frac{\partial z^{(k)}}{\partial z^{(k-1)}} \cdot \frac{\partial z^{(l)}}{\partial w_{ij}^{(l)}}

Once these error derivatives are compiled across all training instances, the weights are updated via Stochastic Gradient Descent (SGD) parameterized by learning rate $\eta$ :

w_{\text{new}} = w_{\text{old}} - \eta \cdot \nabla_w \mathcal{L}(w)

PYTORCH REFERENCE IMPLEMENTATION

Here is how this feedforward computational graph and activation pipeline is declared in modern PyTorch:

import torch
import torch.nn as nn
import torch.optim as optim

# ── Define FeedForward Computational Graph ─────────────
class FeedForwardNetwork(nn.Module):
    def __init__(self, input_dim=2, hidden_dim=3, output_dim=1):
        super().__init__()
        # Layer 1: Input to Hidden
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu = nn.ReLU()

        # Layer 2: Hidden to Output
        self.fc2 = nn.Linear(hidden_dim, output_dim)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        # Affine combination + non-linear activation
        z1 = self.fc1(x)
        a1 = self.relu(z1)
        z2 = self.fc2(a1)
        y_hat = self.sigmoid(z2)
        return y_hat

# ── Train with Binary Cross-Entropy Loss & SGD ─────────
model = FeedForwardNetwork()
criterion = nn.BCELoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

08 / THE TWO PASSES

Forward, Then Backward: The Pass-by-Pass Walkthrough

Training is two sweeps over the same graph, run in opposite directions. On the forward pass each layer performs one affine step followed by one nonlinear step: $z^{(l)} = W^{(l)} a^{(l-1)} + b^{(l)}$ , then $a^{(l)} = f(z^{(l)})$ . The input $a^{(0)} = x$ starts the chain and the last activation produces the prediction $\hat{y}$ .

A scalar loss — say binary cross-entropy — folds that whole prediction down to one number, $L(\hat{y}, y)$ . That single number is what training minimises; every parameter is judged only by how much it moves it.

The backward pass is the chain rule applied right-to-left. Starting from $\partial L / \partial \hat{y}$ , each layer takes the gradient arriving from the layer above (the incoming gradient) and multiplies it by its own local derivative: the activation slope $f'(z)$ for the neuron, and the layer's input $a^{(l-1)}$ for the weights. Nothing is recomputed — the forward activations are cached and reused, which is why the forward pass must run first.

Every sample in a batch is pushed through with the same weights. Forward, each sample contributes its own output; backward, each contributes its own gradient, and those per-sample gradients are summed into one update. That is why a batch behaves like one shared model learning from many examples at once rather than many models — the weights are literally the same object on every forward and backward trip.

Stage	Forward pass	Backward pass (local × incoming)
Input L0	a⁰ = x (2×1)	∂L/∂x = (W¹)ᵀ δ¹
Hidden L1	z¹ = W¹a⁰ + b¹ → a¹ = ReLU(z¹) (3×1)	δ¹ = (W²)ᵀδ² ⊙ f′(z¹); ∂L/∂W¹ = δ¹(a⁰)ᵀ
Output L2	z² = W²a¹ + b² → ŷ = σ(z²) (1×1)	δ² = (∂L/∂ŷ) ⊙ σ′(z²); ∂L/∂W² = δ²(a¹)ᵀ
Loss	L(ŷ, y) (scalar)	seed: ∂L/∂ŷ from the loss

FIGURE 08 · FORWARD & BACKWARD SWEEP9s LOOP

*Ink pulse sweeps left→right as activations are computed; accent pulse sweeps right→left as the chain-rule gradient returns.

Explore additional visual essays in the collection:Explore Published Articles Grid →