Feedforward neural network

A feedforward (or “feed-forward”) neural network is the simplest kind of neural network. It consists of one or more layers of neurons, connected to all or a subset of the neurons from the layer before it. The layer is called “fully connected” if, for every layer , every neuron connects in layer connects to every neuron in layer . A fully connected feedforward network with at least two layers (including the output) is called a multi-layer perceptron.

We can express layer of a FFNN as

where:

• is the vector of activations for layer ;
• is an activation function;
• is the weight matrix for the connections between layer and layer ; and
• is the vector of bias vector for layer .

When the 02 Activation functions of the neuron is linear, the network as a whole becomes equivalent to a matrix operation. Hence, learning a weight matrix is equivalent to learning a feed-forward neural network with a linear activation function.