2.8.1. Overfitting, Underfitting and Model Complexity

Neural
networks are often referred to as universal function approximators since theoretically
any continuous function can be approximated to a prescribed degree of accuracy
by increasing the number of neurons in the hidden layer of a feedforward backpropagation
network [75].
This can be proven by Kolmogrov's theorem stating that a neural network with
linear combinations of _{} monotonically increasing nonlinear functions
of only one variable is able to fit any continuous function of n variables [76].
Yet in reality, the objective of a multivariate calibration is not to approximate
a calibration data set with an ultimate accuracy, but to find a calibration
with the best possible generalizing ability [77].
The gap between the approximation of a calibration data set and the generalization
ability of a calibration becomes the more problematic the higher the number
of variables and the smaller the data set, which will be further explained in
the following sections.

The best measure for the generalizing ability is
the error of prediction of as many independent separate validation data as possible.
According to figure 2 the error of prediction is
composed of two main contributions, the remaining interference error and the
estimation error [39]. The interference error is the systematic
error (bias) due to unmodeled interference in the data, as the calibration model
is not complex enough to capture all the interferences of the relationship between
sensor responses and analytes. The estimation error is caused by modeling measured
random noise of various kinds. The optimal prediction is obtained, when the
remaining interference error and the estimation error balance each other (arrow
in figure 2). The effect of the prediction error
increasing due to a too simple model is called underfitting whereas the effect
of the increased prediction error due to a too complex model is called overfitting
or overtraining. In figure 3 it is shown that the
optimal complexity of the model highly depends on the size and quality of the
calibration data set. For data sets, which are noisy and limited in size, a
simple calibration model is needed to prevent the overfitting. Neural networks,
which are too complex (too big), are in danger of learning these data by heart
and consequently model noise of the data. For big data sets, which contain only
little noise, the best model is more complex resulting in an overall smaller
prediction error for the same functional relationship. Consequently, for each
data set an optimal model complexity has to be found [78]
whereby the complexity of the models is directly related with the number of
variables utilized by the model. The search of the optimal models is a very
difficult task in the field of the multivariate calibration and is further discussed
in section 2.8.2.

figure 2: Scheme for the error
of prediction as a function of the complexity of the calibration model.

An overfitting
can be detected, if the error of prediction of the independent validation data
is significantly higher than the error of prediction of the calibration data
whereby both data sets have to be within the same range of the response variables
(for example within the same concentration range) to prevent additional biases
due to extrapolation [79].
An underfitting manifests in high prediction errors for both data sets. Not
only neural networks are affected by the effects of underfitting and overfitting,
but also most modern multivariate calibration algorithms are subject to these
effects [39]. In the following section, the discussion of the construction of
optimal model complexities mainly refers to neural networks but can also be
generalized for various multivariate calibration methods in many topics.

figure 3: Scheme for the error
of prediction depending on the size and quality of the calibration data set,
which influence the estimation error.