6 NETWORK ARCHITECTURES

The manner in which the neurons of a neural network are structured is intimately linked with the learning algorithm used to train the network. We may therefore speak of learn-ing algorithms (rules) used in the design of neural networks as belearn-ing structured. The classification of learning algorithms is considered in Section 8. In this section, we focus attention on network architectures (structures).

In general, we may identify three fundamentally different classes of network architectures:

(i) Single-Layer Feedforward Networks

In a layered neural network, the neurons are organized in the form of layers. In the sim-plest form of a layered network, we have an input layer of source nodes that projects directly onto an output layer of neurons (computation nodes), but not vice versa. In other words, this network is strictly of a feedforward type. It is illustrated in Fig. 15 for the case of four nodes in both the input and output layers. Such a network is called a single-layer network, with the designation “single-layer” referring to the output layer of computation nodes (neurons). We do not count the input layer of source nodes because no computation is performed there.

Section 6 Network Architectures 21

FIGURE 15 Feedforward network with a single layer of neurons.

Input layer of source

nodes

Output layer of neurons

(ii) Multilayer Feedforward Networks

The second class of a feedforward neural network distinguishes itself by the presence of one or more hidden layers, whose computation nodes are correspondingly called hidden neurons or hidden units; the term “hidden” refers to the fact that this part of the neural network is not seen directly from either the input or output of the network. The func-tion of hidden neurons is to intervene between the external input and the network out-put in some useful manner. By adding one or more hidden layers, the network is enabled to extract higher-order statistics from its input. In a rather loose sense, the network ac-quires a global perspective despite its local connectivity, due to the extra set of synap-tic connections and the extra dimension of neural interactions (Churchland and Sejnowski, 1992).

The source nodes in the input layer of the network supply respective elements of the activation pattern (input vector), which constitute the input signals applied to the neurons (computation nodes) in the second layer (i.e., the first hidden layer). The out-put signals of the second layer are used as inout-puts to the third layer, and so on for the rest of the network. Typically, the neurons in each layer of the network have as their inputs the output signals of the preceding layer only. The set of output signals of the neurons in the output (final) layer of the network constitutes the overall response of the net-work to the activation pattern supplied by the source nodes in the input (first) layer. The architectural graph in Fig. 16 illustrates the layout of a multilayer feedforward neural net-work for the case of a single hidden layer. For the sake of brevity, the netnet-work in Fig. 16 is referred to as a 10–4–2 network because it has 10 source nodes, 4 hidden neurons, and 2 output neurons. As another example, a feedforward network with m source nodes, h₁ neurons in the first hidden layer, h₂neurons in the second hidden layer, and q neurons in the output layer is referred to as an m–h₁–h₂–q network.

Input layer of source

nodes

Layer of hidden neurons

Layer of output neurons FIGURE 16 Fully connected

feedforward network with one hidden layer and one output layer.

The neural network in Fig. 16 is said to be fully connected in the sense that every node in each layer of the network is connected to every other node in the adjacent for-ward layer. If, however, some of the communication links (synaptic connections) are missing from the network, we say that the network is partially connected.

(iii) Recurrent Networks

A recurrent neural network distinguishes itself from a feedforward neural network in that it has at least one feedback loop. For example, a recurrent network may consist of a sin-gle layer of neurons with each neuron feeding its output signal back to the inputs of all the other neurons, as illustrated in the architectural graph in Fig. 17. In the structure de-picted in this figure, there are no self-feedback loops in the network; self-feedback refers to a situation where the output of a neuron is fed back into its own input. The recurrent network illustrated in Fig. 17 also has no hidden neurons.

In Fig. 18 we illustrate another class of recurrent networks with hidden neurons.

The feedback connections shown in Fig. 18 originate from the hidden neurons as well as from the output neurons.

The presence of feedback loops, be it in the recurrent structure of Fig. 17 or in that of Fig. 18, has a profound impact on the learning capability of the network and on its per-formance. Moreover, the feedback loops involve the use of particular branches com-posed of unit-time delay elements (denoted by z^-1), which result in a nonlinear dynamic behavior, assuming that the neural network contains nonlinear units.

Section 6 Network Architectures 23

FIGURE 17 Recurrent network with no self-feedback loops and no hidden neurons.

Unit-time delay operators z^1 z^1 z^1 z^1