2.4.6.   Conclusions

When comparing the advantages and disadvantages of the different subsampling algorithms bootstrapping and random subsampling are most suited for splitting the data into calibration, test and validation subsets. As the user definable ratio between the sizes of the different subset allows a high flexibility, the random subsampling procedure was used to split the data into calibration, test and monitor data sets in this work, whereas for most data sets a static external validation set was recorded and used. The monitor set for the early-stopping procedure of the neural networks (see section 2.7.3) was generated by a modified full crossvalidation procedure, which speeds up learning and which is described in detail in [28].

Besides of the averaging effect of the subsampling procedure, the comparison of the standard deviations between the predictions of the test data of the different subsets additionally allows an estimation of the robustness of the calibration method. A high standard deviation is an indication of the calibration being subject to the random partitioning of the data. If the quality of the calibration and prediction significantly depends on the perturbation of the data sub sets, the calibration method is not very robust.