All networks
of this work are fully connected except of the non-uniform growing neural networks
introduced in chapter 8.
Fully connected means that a neuron is connected to all neurons of the proceeding
layer. All networks except of the growing neural networks contain one layer
of hidden neurons. If no special optimization technique is used, the number
of hidden neurons is optimized by a gradient algorithm. Starting with 1 hidden
neuron this algorithm adds fully connected neurons to the hidden layer until
the error of prediction does not improve any more. For the hidden neurons, the
hyperbolic tangent was used as activation function, which has some advantages
referring to the convergence speed of learning in contrast to other nonlinear
functions [59].
The activation function of the output neurons is a linear function. The combination
of linear and nonlinear activation functions allows an effective modeling of
both, nonlinear and linear data sets.
In principle,
neural networks can model several responses simultaneously. Therefore, it is
possible either to use a neural network with as many output neurons as responses
to model or to use a separate neural model with one single output neuron for
each response. In congruence with Despagne et al. [8] and Moore et al. [60] several tests showed that for
the calibration and prediction single networks with one output are superior
in terms of lower errors of prediction. Thus, for all calibrations networks
with single outputs are used if not stated differently. For the optimization
of networks, like a variable selection, the choice of network type significantly
influences the results as single output networks select variables, which are
most predictive for one individual response whereas multi output networks select
the variables, which model the ensemble of responses best. This issue is further
discussed in section 10.1.
Ph. D. Thesis