4 Experiments on Well Logging Inversion
4.5 Higher-Order Feature Multi-Layer Neural Net (HOML)
In order to enhance the non-linear mapping of network, input features can be expanded to a larger space through the transformation of non-linear functions. In this study, we expand the input features using the higher-order function which from second-order function to 5th-order function. Using these functions we design 5 higher-order feature neural nets: HOML-1, HOML-2, HOML-3, HOML-4, and HOML-5, and are listed in Table 4-7. HOML-1 is actually the general MLP network with no higher-order features. All of these 5 network types are trained by gradient descent method, and the number of hidden nodes is 1.2 times of the number of input nodes. Figure 4-8 is an example that shows the HOML-2 that expands input features using second-order function.
Table 4-7. Five different HOML types.
Network type Features (include bias) Network size
HOML-1 1 + x 10-12-10
HOML-2 1 + x + x2 20-24-10
HOML-3 1 + x + x2 + x3 30-36-10
HOML-4 1 + x + x2 + x3 + x4 40-48-10 HOML-5 1 + x + x2 + x3 + x4 + x5 50-60-10
Fig. 4-8. HOML-2 with original and expanded features.
A. Comparison of Performance of Network that without Hidden Layer and that with One Hidden Layer
In this section, we use five higher-order feature neural nets with single layer:
HOSL-1, HOSL-2, HOSL-3, HOSL-4, and HOSL-5 that listed in Table 4-8. Figure 4-9 is an example that shows the HOSL-2 that expands input features using second-order function. We test the inverse ability of HOSL networks for well logging data and compare the performance with the HOML networks. Both HOSL and HOML networks use gradient descent method as learning rule. The learning rate is 0.6, the momentum parameter is 0.4, and the stopping error is 0.00001. The maximum iterations are set to 10,000. Repeat 10 trials for each experiment and the
number of training datasets is 30 for each trial. The dataset number to test the trained network for these 10 trials is 1, 5, 10, 12, 15, 17, 20, 23, 25, and 30 respectively. Each experimental result is the average of 10 trials and is shown in Table 4-9. The testing results show that higher-order feature neural nets without hidden layers can be used for well logging data inversion. However, from the results we found:
First, the average training time of each HOSL network is long since they failed to achieve the training goal and stop training at maximum iterations. However, HOSL-3, 4, and 5 have significant improvement in mean absolute error. The reason is that the networks become more non-linear with the usage of higher-order features.
Second, for each network type, the average training time of HOML is shorter than that of HOSL. This is because the HOML networks are more non-linear than HOSL networks with the usage of hidden layer and converged before reaching the maximal iterations. Figure 4-10 shows the comparison of MSE curve between HOML-5 and HOSL-5, the used testing dataset is number 30. HOSL-5 is failed to achieve the training goal and reaches the maximal iterations. The training time of HOML-5 is shorter, and the training goal is achieved at 1,189 iterations. From the above discussion we found that the network with one hidden layer performs better than that without hidden layer, so we use the one-hidden layer network to test the well logging data inversion.
Table 4-8. Five different HOSL types.
Network type Features (include bias) Network size
HOSL-1 1 + x 10-10
HOSL-2 1 + x + x2 20-10
HOSL-3 1 + x + x2 + x3 30-10
HOSL-4 1 + x + x2 + x3 + x4 40-10 HOSL-5 1 + x + x2 + x3 + x4 + x5 50-10
Table 4-9. Testing results of HOSL and HOML.
Network size Num. Of training
patterns Avg. of MAE Avg. training time (Sec.)
HOSL-1 10-10 600 0.023098 9,042
HOSL-2 20-10 600 0.022937 9,068
HOSL-3 30-10 600 0.009271 9,354
HOSL-4 40-10 600 0.009274 9,867
HOSL-5 50-10 600 0.009272 9,915
HOML-1 10-12-10 600 0.003033 1,882
HOML-2 20-24-10 600 0.002926 1,268
HOML-3 30-36-10 600 0.002879 1,121
HOML-4 40-48-10 600 0.002865 1,043
HOML-5 50-60-10 600 0.002860 1,231
Fig. 4-9. HOSL-2 with original and expanded features.
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000 0
1 2 3 4 5 6 7 8x 10-4
Iteration
MSE
MSE curve of HOSL-5 MSE curve of HOML-5
Fig. 4-10. Comparison of MSE curve between HOML-5 and HOSL-5.
B. Experiments of HOML with One Hidden Layer
We use the networks from HOML-1 to HOML-5 with one hidden layer. The learning rate is 0.6, the momentum parameter is 0.4, and the stopping error is 10-5. 31 trials are done for average performance for each network type. The testing results are shown in Table 4-10, and the average of mean absolute error of each network is plotted in Figure 4-11. From the experimental results, we make the following two discoveries:
First, the HOML-1 that without expanding the input features yields the largest average of mean absolute error. The average of mean absolute error is smaller with adding higher-order features on the input features (from HOML-2 to HOML-5). It is obvious that the performance of HOML is better than general MLP network. Figure 4-12 is an example that shows the testing result of HOML-5 on dataset number 23.
Second, the average training time of HOML-2 to HOML-5 that with higher-order features is shorter. In 31 trials of each HOML network, we plot the training time for each testing dataset in Figure 4-13. In order to emphasize the training time for HOML-1, a solid line connects the nodes. In Figure 4-13, most of the training times of HOML-1 are longest among 5 network types. The networks
have the shorter average training time (also shown in average training time column of Table 4-10). The reason is that the output of networks with higher-order features can fit the desired output more non-linear, the convergence becomes more quickly.
Table 4-10. Testing result of each type of HOML.
Network size Avg. of MAE Smallest MAE Avg. training time (sec.) HOML-1 10-12-10 0.003008 0.002065 2,105 HOML-2 20-24-10 0.002514 0.002149 1,804 HOML-3 30-36-10 0.002496 0.002040 1,483 HOML-4 40-48-10 0.002501 0.002075 1,633 HOML-5 50-60-10 0.002407 0.001912 1,663
0.002 0.0022 0.0024 0.0026 0.0028 0.003 0.0032
HONN-1 HONN-2 HONN-3 HONN-4 HONN-5 Network type
Average of MAE
Fig. 4-11. Average of mean absolute error in different HOML networks.
490 500 510 520 530 540 550 560 570 580 590
0 0.2 0.4 0.6 0.8
1 desired output
real output
200
Fig. 4-13. Training time for each dataset in 5 HOML networks.