利用神經網路解微分方程 - 政大學術集成

全文

(1)國立政治大學應用數學系碩士學位論文. 立. 政治大. ‧ 國. 學. 利用神經網路解微分方程. Neural Network Methods for Solving ‧. n. al. er. io. sit. y. Nat. Differential Equation Ch. engchi. i n U. v. 研究生：黃振維撰指導教授：符聖珍博士中華民國 108 年 7 月 DOI:10.6814/NCCU201900919.

(2) 謝在政大應數的這三年來，真的是要謝謝很多很多人，我才能順利的走到今天。首先要感謝符聖珍老師與蔡隆義老師，非常有耐心的重新教我基本概念與指導我. 政治大. 的論文，並且不厭其煩的陪我修正，才有今天的這篇論文的產生。也要感謝曾睿彬老師與李宣緯老師擔任我的口試委員，幫我補充了一些論文的不. 立. 足之處，使我的論文更加完整。. ‧ 國. 學. 感謝陪我三年的研究所同學們，以及一路上遇到的學長姐與學弟妹。有你們在，讓我在這三年的人生增添了不少色彩。. ‧. 最後感謝家人與女友，在我背後默默的支持我到現在，愛你們！. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. i. DOI:10.6814/NCCU201900919.

(3) 中文要本文是在敘述利用前饋人工神經網路的數值方法去近似微分方程的解，其中分別利用邊界條件或是初始條件去造出試驗函數去讓神經網路去近似，或是試驗函數不. 政治大. 隱含初始條件或邊界條件，直接把初始條件與邊界條件當作神經網路的目標函數的優化條件，利用 SGD 和 ADAM 優化器去更新神經網路參數，再分別做比較。. 立. 其中在常微分方程分別去試驗了邊界值問題、特徵值問題、初始值問題、生態系. ‧ 國. 學. 統、及三種經典的偏微分方程，依照不同的方法去滿足不同的條件，進一步的去降低數值解的誤差。. ‧. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. ii. DOI:10.6814/NCCU201900919.

(4) Abstract This paper descirbes how to use the feed forward artificial neural network method to find the approximate solution of differential equations. Two types of. 政治大. the trial funcitons are used, and the objective function is minimized by SGD and ADAM methods respectively.. 立. We test the boundary value problem, eigenvalue problem, initial value problem,. ‧ 國. 學. two types of the ecological systems, and three classical types of the partial differential equations. We illustrate some examples and give some comparison. ‧. results in Chapter 4.. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. iii. DOI:10.6814/NCCU201900919.

(5) Contents 謝. i. 中文要. 立. Abstract. ‧ 國. iv vi. sit. y. vii. n. al. 1. er. io. 1 Introduction. Nat. List of Figures. iii. ‧. List of Tables. ii. 學. Contents. 政治大. Ch. 2 Feed Forward Artificial Neural Network. . e. n . .g. c . h . .i .. i n U. v. 3. 2.1. Architecture . . . . . . . .. . . . . . . . . . . . . . . . . . . .. 3. 2.2. Mathematical Model of Artificial Neural Network . . . . . . . . . . . . . . . .. 4. 2.3. Activation Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 4. 2.4. Optimizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 5. 2.4.1. Stochastic Gradient Decent(SGD) . . . . . . . . . . . . . . . . . . . .. 5. 2.4.2. Adaptive Moment Estimation(ADAM) . . . . . . . . . . . . . . . . .. 5. 3 Objective Functions and Algorithm. 7. 3.1. Trial function method of type1 . . . . . . . . . . . . . . . . . . . . . . . . . .. 7. 3.2. Trial function method of type2 . . . . . . . . . . . . . . . . . . . . . . . . . .. 8. 3.3. Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 8. DOI:10.6814/NCCU201900919.

(6) 4 Using Neural Network Method to Solve Differential Equations 4.1. 9. Boundary Value Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 9. 4.1.1. Construction of the trail function and objective function . . . . . . . .. 10. 4.1.2. Select the number of hidden layers . . . . . . . . . . . . . . . . . . . .. 13. 4.1.3. Select the optimizers of the neural network . . . . . . . . . . . . . . .. 14. 4.2. The Eigenvalue Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 14. 4.3. Initial Value Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 21. 4.3.1. Lane-Emden Equation . . . . . . . . . . . . . . . . . . . . . . . . . .. 21. Systems of Differential Equations . . . . . . . . . . . . . . . . . . . . . . . .. 26. 4.4.1. Rabbit versus Sheep Problem . . . . . . . . . . . . . . . . . . . . . .. 26. 4.4.2. Lokta-Volterra Model . . . . . . . . . . . . . . . . . . . . . . . . . . .. 29. 立. 33. 4.5.1. Laplace’s Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 33. 4.5.2. Heat Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 36. 4.5.3. Wave Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 39. ‧. 43. io. sit. y. Nat. Bibliography. 學. Partial Differential Equations . . . . . . . . . . . . . . . . . . . . . . . . . . .. n. al. er. 4.5. 政治大. ‧ 國. 4.4. Ch. engchi. i n U. v. DOI:10.6814/NCCU201900919.

(7) List of Tables 4.1. The exact solution and yT1 . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 11. 4.2. Comparison between yT1 and yT2 . . . . . . . . . . . . . . . . . . . . . . . . .. 12. Solutions of different number of hidden layers . . . . . . . . . . . . . . . . . .. 13. 4.3. 16. 4.6. Comparison between the nurerical solution yn and yT2 . . . . . . . . . . . . . .. 17. 4.7. Comparison of two numerical solution and yT2 and yT2 . . . . . . . . . . . . .. 18. 4.8. Comparison between exact solution and uT1 . . . . . . . . . . . . . . . . . . .. 22. 4.9. Comparison between old uT1 and new unT1. 24 25. ‧ 國. 4.5. 學. yT1 and yT2 of eigenvalue problem . . . . . . . . . . . . . . . . . . . . . . . .. y. 14. sit. 4.4. 政治大 Comparison of the solutions with differnet optimizers . . . . . . . . . . . . . . 立 (1). (2). ‧. Nat. . . . . . . . . . . . . . . . . . . .. io. er. 4.10 Comparision among uT1 , unT1 and uT2 . . . . . . . . . . . . . . . . . . . . . . 4.11 Comparison between the exact solution and xT2 , yT2 . . . . . . . . . . . . . .. n. al. Ch. 4.12 Comparison between the uE and uT1 . . .. v . . . n. i. . U. 28. . . . . . . . . . . . . . .. 35. . . . . . . . . . . . . . . . . . . .. 38. 4.14 Comparison between uE and unT1 . . . . . . . . . . . . . . . . . . . . . . . .. 41. 4.13. i e Comparison between uE and uTn.g. c . h . . . 1. DOI:10.6814/NCCU201900919.

(8) 2.1. The architecture of feed forward neural network . . . . . . . . . . . . . . . . .. 3. 4.1. The exact solution and yT1 . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 11. 4.2. The exact solution and yT1. 政y . 治 and . . . . .大 . . . . . . . . . . . . . . . . . .. 12. 4.3. The exact solution and yT1 and yT2 . . . . . . . . . . . . . . . . . . . . . . . .. 16. 4.4. The relations between slopes y ′ (0) and boundary values y(1) . . . . . . . . . .. 17. 4.5. Two approximate solutions of yT2 and yT2 . . . . . . . . . . . . . . . . . . . .. 19. 4.6. The relation of λ and y ′ (0) . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 4.7. ‧. 19. The relations between λ and y ′ (0) . . . . . . . . . . . . . . . . . . . . . . . .. 20. 4.8. y. List of Figures. The exact solution and uT1 . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 22. 4.9. The exact solution and uT1 and unT1 . . . . . . . . . . . . . . . . . . . . . . .. 24. (1). (2). Nat. al. er. io. sit. 學. ‧ 國. 立. T2. n. v i n Ch The architecture of the reconstruct i U .. e n gneural c hnetwork. 4.10 The exact solution and uT1 , unT1 and uT2. . . . . . . . . . . . . . . . . . . . .. 25. . . . . . . . . . . . . . .. 27. 4.12 Four neural network solutions and numerical solution with fixed points . . . .. 28. 4.13 The numerical solution and (xT2 , yT2 ) . . . . . . . . . . . . . . . . . . . . . .. 30. 4.14 The numerical solution and xT2 ,yT2 with t . . . . . . . . . . . . . . . . . . . .. 31. 4.15 Numerical solution and (xT2 , yT2 ) . . . . . . . . . . . . . . . . . . . . . . . .. 32. 4.16 Numerical solution and xT2 , yT2 with t . . . . . . . . . . . . . . . . . . . . . .. 32. 4.17 The exact solution (4.30) . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 34. 4.18 The graph of uT1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 35. 4.19 The error of uT1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 36. 4.20 The exact solution of (4.34) . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 37. 4.21 The graph of uT1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 38. 4.22 The error of the uT1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 39. 4.11. DOI:10.6814/NCCU201900919.

(9) 4.23 The exact solution of (4.38) . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 40. 4.24 The graph of uT1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 41. 4.25 The graph of unT1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 42. 4.26 The error of the unT1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 42. 立. 政治大. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. DOI:10.6814/NCCU201900919.

(10) Chapter 1 Introduction 政治大. Differential equation is a mathematical model which usually descirbes a phenomenon of. 立. real systems in the world, such as in biology, physics, chemistry, engineering, economics and. ‧ 國. 學. so on. In order to observe the behavior of the event, how to find the solution of differential equations is very important. However, the exact solutions is not easy to get, especially for the. ‧. nonlinear problem. The numerical methods are very popular to be used in solving differential equations, such as finite difference method, finite element method, ...etc [3]. But they need. y. Nat. sit. some discretizations with equal mesh. Some available approximate methods can be found in. er. io. the literatures, such as the Tayler series method, variational iteration method, ...etc [6].. al. n. v i n C h the neural network equaitons.We note that the solution via e n g c h i U method is differentiable, mesh free, In recent years, there are many authors using neural network methods to solve differential. and easy to be used in subsequet calculations [14]. In this paper, we consider the neural network method to solve some types of the differential equations, including the boundary value problem, initial value problem, two ecological systems, and some three types of the partial differential equations. We illustrate some examples and give the comparesion result with the analytic solutions or numerical solutions. We use the feed forward artificial neural network method to find the approximate solution of differential equations. Two types of the trial funcitons are used, and the objective function is minimized by SGD and ADAM methods respectively. We use differnent strategies from other studies [14] to improve the accuracy of the approximate solutions. In Remark 4.1, 4.2 and 4.3, we provide some extra conditions, such as slope or period or symmetry of the solutions, to make the result better.. 1. DOI:10.6814/NCCU201900919.

(11) The content of this paper is orginazed as follows.. In chapter 2, we introduce the. architectures of the feed forward neural network, including the activation funcion, and some optimizers, SGD and ADAM. In chapter 3, two types of the trial function and the objective function are defined. In chapter 4, we consider several examples in boundary value problem, eigenvalue problem, initial value problem, two types of the ecological systems, and three classical types of the partial differential equations.. 立. 政治大. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. 2. DOI:10.6814/NCCU201900919.

(12) Chapter 2 Feed Forward Artificial Neural Network 政治大. In this chapter, we first introduce the fundamental feed forward artificial neural network,. 立. including architecture, activation function. Next we give algorithms for SGD and ADAM.. ‧ 國. 學. 2.1 Architecture. ‧. Feed forward artificial neural network is a simple type of interconnected neural network.. y. Nat. sit. In this neural network the information from neurons to neurons moves forward in only one. n. al. er. io. direction. The data flow from the input layer to the hidden layers, and then to the output layer, as in the following figure.. Ch. engchi. i n U. v. Figure 2.1: The architecture of feed forward neural network 3. DOI:10.6814/NCCU201900919.

(13) In this paper, we define the frame of the neural network as: [I, H1 , H2 , H3 ...Hn , O].. (2.1). Where I is the number of neurons in the input layer, Hi is the number of neruons in the ith hidden layers where1 ≤ i ≤ n, n is the number of the hidden layers, O is the number of the neurons in the output layer.. 治 2.2 Mathematical Model Neural Network 政of Artificial N (x) =. H ∑. 學. vi σ(zi ) ,. i=1. y. wij xj + bi ,. j=1. al. (2.2b). er. io. sit. Nat. zi =. n ∑. (2.2a). ‧. where. ‧ 國. 大立 The output in the neural network can be written in the following form [14]:. n. where wij is the weight form the neuron j in the input layer to the neuron i in the hidden layer,. Ch. i n U. v. and bi is the bias in the neuron of the hidden layer, and vi is the weight from the neuron in the. engchi. hidden layer to the output layer. The weight of v , w and bias b are random numbers, and xj is the input data, H is the number of the neurons in the hidden layer, and n is the number of the neurons in the input layer. In gerneral, the N is a nonconvex function of the x , v , w and b.. 2.3 Activation Function We define the activation function as: σ(x) =. 1 , x∈R. 1 + e−x. (2.3). 4. DOI:10.6814/NCCU201900919.

(14) 2.4 Optimizers In this section we introduce two types of optimizer, SGD and ADAM respectly.. 2.4.1 Stochastic Gradient Decent(SGD) Stochastic Gradient Decent [8] is a common optimizer in neural network, we always use it to update the weights and bias. Ginen a function F as in (2.2). We initialize the random (1). (1). (1). parameters vi ,wij and bi , and t = 1, 2, ...T is the number of iteration, then we give the updating formulas as following: , 政治− α ∂F ∂v大 (t). (t−1). vi = vi. 立. (t). i. −α. ∂F , ∂wij. (t−1). −α. ∂F . ∂bi. (2.4). ‧. ‧ 國. (t). bi = bi. 學. (t−1). wij = wij. Where the α is the learning rate which is a small real number.. n. Ch. er. io. al. sit. y. Nat. 2.4.2 Adaptive Moment Estimation(ADAM). i n U. v. Adaptive Moment Estimation [10] is a generalized optimizer. We initialize the following parameters:. engchi. (1). (1). vM = 0 , vR = 0 , (1). (1). wM = 0 , w R = 0 , (1). (1). bM = 0 , b R = 0 ,. (2.5). then we difine the first moment estimates: (t). ∂F , ∂vi. vM = β1 vM. (t−1). + (1 − β1 ). (t). (t−1). + (1 − β1 ). ∂F , ∂wij. (t−1). + (1 − β1 ). ∂F , ∂bi. wM = β1 wM (t). bM = β1 bM. (2.6). 5. DOI:10.6814/NCCU201900919.

(15) where β1 ∈ (0, 1), β1 means the importance of the previous values. In this paper, we choose the β1 = 0.9. Next we compute the second moment estimates: (t). (t−1). (t) wR. =. ∂F 2 + (1 − β2 )( ) , ∂wij. (t−1) β2 w R. (t). ∂F 2 ) , ∂vi. + (1 − β2 )(. vR = β2 vR. (t−1). + (1 − β2 )(. bR = β 2 bR. ∂F 2 ) . ∂bi. (2.7). We always choose the β2 = 0.99. Next we compute:. 政v ,治 v v = 大 1−β 1−β (t) M. (t) vm =. 立. (t) R. (t) r. t 1. (t). wM wR , wr(t) = , = t 1 − β1 1 − β2t (t). 學. (t). bR bM , b(t) , r = t 1 − β1 1 − β2t. b(t) m =. n. =. (t−1) vi. Ch. vm. − α√. (t−1) vr. engchi U. sit er. io. (t−1). al. (2.8). y. Nat. and finally we update the paramerers: (t) vi. ‧. ‧ 國. (t). (t) wm. ,. t 2. v + n iϵ. ,. (t−1). (t) wij. =. (t−1) wij. wm. − α√. ,. (t−1). wr. +ϵ. (t−1). (t). (t−1). bi = bi. bm. − α√. (t−1) br. .. (2.9). +ϵ. Because at first the denominator may be zero, so we add an small positive ϵ to ensure the denominator not be zero, and we choose the ϵ = 10−6 .. 6. DOI:10.6814/NCCU201900919.

(16) Chapter 3 Objective Functions and Algorithm 政治大 Trial function立 method of type1. 學. ‧ 國. 3.1. We consider a boundary value problem:. y(a) = α , y(b) = β .. n. al. (3.1b). er. io. sit. y. Nat. with. (3.1a). ‧. y ′′ = f (x, y, y ′ ) ,. i n U. v. We define the trial function of type1 as the following form [14]:. Ch. engchi. yT1 (x, p) = A(x) + B(x) × N (x, p) ,. (3.2). where p denote the parameters of weights v,w and bias b in the neural network, xi is the trianing data ∈ [a, b], and the A(x) and B(x) are choosen to make the trial function satisfying the boundary conditions, and the N (x, p) is the output function. Then we start to train the neural network. Through the optimizer, we minimize the objective function: n 2 ∑ dyT1 (xi , p) d2 yT1 (xi , p) − f (xi , yT1 , )} . E1 = { dx2 dx i=0. (3.3). The tiral functions in ordinary and partial differential equation can be similarly defined as above.. 7. DOI:10.6814/NCCU201900919.

(17) 3.2 Trial function method of type2 In contrast, we may not use the boundary or initial condition in the trial function [4]. That means the trial function of type2 is defined as an output function in the neural network. That is: yT2 (x, p) = N (x, p) .. (3.4). We define the corresponding objective function: n ∑ d2 yT2 (xi , p) dyT2 (xi , p) 2 E2 = )] + [yT2 (a, p) − α]2 + [yT2 (b, p) − β]2 . (3.5) [ − f (xi , yT2 , 2 dx dx i=0. 政治大. For all problems in chapter 4, the trial function and objective function can be similarly. 立. defined as above.. ‧ 國. 學. 3.3 Algorithm. ‧. The processes for solving differiential equation by the neural network can be divided into. sit. y. Nat. following steps :. al. n. neurons.. er. io. Step1: Choose the numbers of the layers and its neurons, and initialize the parameters of the. Ch. Step2: Define the activation function.. engchi. i n U. v. Step3: Choose the training-data in the domain. Step4: Choose the type of the trial function.. Step5: Construct the corresponding objective(error) function. Step6: Choose the optimizer and the learning rate to minimize the objective function. Step7: Use the optimal weights to get the approximate solution.. 8. DOI:10.6814/NCCU201900919.

(18) Chapter 4 Using Neural Network Method to Solve Differential Equations 政治大立 ‧ 國. 學. In this paper, we use python to construct the neural network environment, and we use ADAM optimizer to update the weights and bias.. ‧ sit. y. Nat. 4.1 Boundary Value Problem. al. n. of falling object [14]:. er. io. Consider a simple two-point boundary value problem as (3.1), which describe the motion. i n dy dCy h+ 0.15( dtc) −h 9.82 i U= 0 dt2 e n g 2. v. ,. (4.1a). with the following boundary conditions: y(0) = 0, y(12) = 600, 0 ≤ t ≤ 12 .. (4.1b). We know that the exact solution is given by: yE (t) = −22.36 + 65.467t + 222.36e−0.15t .. (4.2). We are going to find the approximate solutions by neural network method.. 9. DOI:10.6814/NCCU201900919.

(19) 4.1.1 Construction of the trail function and objective function We define the trial function of type1 as: yT1 (t, p) = 50t + t(12 − t) × N (t, p) ,. (4.3). and this trial function satisfies the boundary condition: yT (0, p) = 50 × 0 + 0 × (12 − 0) × N (0, p) = 0 , yT (12, p) = 50 × 12 + 12 × (12 − 12) × N (12, p) = 600 .. 政治大. Our objective function is defined by as:. 立∑ n. ‧ 國. d2 yT1 (ti , p) dyT1 (ti , p) 2 − f (ti , )} , 2 dt dt. f (t,. y. Nat. dy dy ) = −0.15( ) + 9.82 . dt dt. (4.4). ‧. where. i. {. 學. E1 =. n. al. er. io. is:. sit. We choose p in the random number ∈ [−0.5, 0.5] of the weight and bias, and the input data. i n U. ti = 0.12i, 0 ≤ i ≤ 100 .. Ch. engchi. v. The frame of the neural network is given by[1, 20, 20, 20, 1] as in (2.1). We train ten thousand times and get the yT1 in Table 4.1 and Figure 4.1:. 10. DOI:10.6814/NCCU201900919.

(20) t 0 1 2 3 4 5 6 7 8 9 10 11 12. yE 0 34.49402564 73.30233955 115.82399579 161.5417554 210.01042667 260.84682954 313.72115789 368.34954496 424.48766436 481.92522241 540.48121768 599.99986078. yT1 0 36.87767655 76.23103212 118.13553794 162.47970608 209.17609822 258.17708544 309.45867857 363.01177675 418.83658755 476.93823074 537.32368398 600. 政治大 Table 4.1: The exact solution and y 立. T1. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.1: The exact solution and yT1 Next we are going to find the approximate solutions by using trial funtion method of type2, we construct the trial function of type2 as(3.4): yT2 (t, p) = N (t, p) ,. (4.5). 11. DOI:10.6814/NCCU201900919.

(21) and construct the objective function as follows: [4]: n ∑ d2 yT2 (ti , p) dyT2 (ti , p) 2 )] + [yT2 (0, p) − 0]2 + [yT2 (12, p) − 600]2 . (4.6) E2 = [ − f (ti , 2 dt dt i=0. We train this network ten thousand times and get yT 2 .The comparison among yT 1 , yT 2 and exact solution are given in the Table 4.2 and the Figure 4.2. yE 0 34.49402564 73.30233955 115.82399579 161.5417554 210.01042667 260.84682954 313.72115789 368.34954496 424.48766436 481.92522241 540.48121768 599.99986078. 政治大. yT2 0.00105545 34.53372788 73.35689183 115.89744851 161.63417186 210.13845717 260.98877798 313.86614368 368.49914787 424.62671499 482.02903746 540.5465305 600.02594204. ‧. ‧ 國. 立. yT1 0 36.87767655 76.23103212 118.13553794 162.47970608 209.17609822 258.17708544 309.45867857 363.01177675 418.83658755 476.93823074 537.32368398 600. 學. t 0 1 2 3 4 5 6 7 8 9 10 11 12. sit. y. Nat. n. al. er. io. Table 4.2: Comparison between yT1 and yT2. Ch. engchi. i n U. v. Figure 4.2: The exact solution and yT1 and yT2 12. DOI:10.6814/NCCU201900919.

(22) In fact, there are too many factors affecting the neural network. They include some question: how many neurons are used, the choice of optimezers and corresponding learning rate, the choice of activation funcitons, the number of times of learning. Many problems can be researched further. Hereafter, we choose two factors to see what happens in selecting the number of hidden layers and selecting the optimizers of the neural network.. 4.1.2 Select the number of hidden layers In this section, we pick three different hidden layers to compare accuracy of this problem. We use the trial function method of type 2, and pick the frame [1, 20, 1], [1, 20, 20, 20, 1] and. 政治大. [1, 20, 20, 20, 20, 20, 1] respectively. ADAM optimizer is used, the comparison result is given. engchi. y. ‧ 國. H5 0.52811748 41.47485532 87.63065478 137.44697973 189.51250584 243.12585518 297.2290195 351.09760511 404.62861316 457.42985372 508.48999901 557.87430914 604.89085105. ‧. n. Ch. H3 0.00105545 34.53372788 73.35689183 115.89744851 161.63417186 210.13845717 260.98877798 313.86614368 368.49914787 424.62671499 482.02903746 540.5465305 600.02594204. 學. io. al. H1 0.01143187 44.2236925 91.17735702 140.28145607 190.98798678 242.84318352 295.41405476 348.14807984 400.56065236 452.36956755 503.14994806 552.14305011 599.19396611. sit. 立. yE 0 34.49402564 73.30233955 115.82399579 161.5417554 210.01042667 260.84682954 313.72115789 368.34954496 424.48766436 481.92522241 540.48121768 599.99986078. Nat. t 0 1 2 3 4 5 6 7 8 9 10 11 12. er. in Table 4.3:. i n U. v. Table 4.3: Solutions of different number of hidden layers From the above result, we observe the error,. error =. 12 ∑. |yE (ti ) − yT2 (ti )| .. (4.7). i=0. In this problem, we find the total error in 1 hidden layer is 277.13217523, the total error in 3 hidden layers is 1.16000653, the total error in 5 hidden layers is 296.37517834. In fact, the total error of objective function in 1 hidden layer converges to about 7 when updating about 8000 times. And the total error of objective function in 3 hidden layers converges 13. DOI:10.6814/NCCU201900919.

(23) to about 0.01 when updating about 9900 times. But the total error of objective function in 5 hidden layers oscillates when updating about 9900 times. To avoid the situation, we can change the learning rate from 0.01 to 0.001 after 10000 times, we find the total error of objective function converges to 0.002 in 19800 times. According to the rule of thumb and time costs, we shall select 3 hidden layers in the following sections.. 4.1.3 Select the optimizers of the neural network Take the frame [1, 20, 20, 20, 1] of the neural network and use trial function method of type. 政治大. 2, then we trian the model by two optimizers SGD and ADAM respectively, the result is given. io. n. Ch. engchi. ADAM 0.00105545 34.53372788 73.35689183 115.89744851 161.63417186 210.13845717 260.98877798 313.86614368 368.49914787 424.62671499 482.02903746 540.5465305 600.02594204. y. sit. Nat. al. SGD 0.13765461 37.66888042 79.34697674 124.0398045 170.97323445 220.04968199 271.08223258 323.68005656 377.39330591 431.91223348 486.90608549 542.19095898 600.02594204. ‧. ‧ 國. 立. yE 0 34.49402564 73.30233955 115.82399579 161.5417554 210.01042667 260.84682954 313.72115789 368.34954496 424.48766436 481.92522241 540.48121768 599.99986078. 學. t 0 1 2 3 4 5 6 7 8 9 10 11 12. er. in Table 4.4:. i n U. v. Table 4.4: Comparison of the solutions with differnet optimizers By (4.7), the total error in SGD optimizer is 82.89082303, and the total error in ADAM optimizer is 1.16000653. In this equation, ADAM is better then SGD.. 4.2 The Eigenvalue Problem We consider an eigenvalue problem [13] y ′′ + λey = 0, 0 ≤ x ≤ 1 ,. (4.8a). 14. DOI:10.6814/NCCU201900919.

(24) with the boundary condition: y(0) = 0, y(1) = 1.23125 .. (4.8b). When λ = −2, the exact solution of (4.8) is obtained in [13]: yE (x) = −2ln(cosx) .. (4.9). We choose the frame [1, 20, 20, 20, 1] of the neural network, and ADAM optimizer, and the input data is: xi = 0.01i, 0 ≤ i ≤ 100 .. T1. yT 1 = 1.23125x + x(x − 1) × N (x, p) ,. (4.10). ‧ y. sit. io. E1 =. n ∑ d2 yT1 (xi , p) − f (xi, yT1 (xi , p))}2 , { 2 dx i=0. n. al. er. Nat. and the corresponding objective function is:. where. and the yT 2 respectively. First,. 學. ‧ 國. 政治大 In the following we shall compute the approximate solutions, y 立 we define the trial funciton of type1:. Ch. engchi. i n U. (4.11). v. f (x, y(x)) = −λey . Next we define the trial funciton of type2: yT2 = N (x, p) ,. (4.12). and the corresponding objective function is: n ∑ d2 yT2 (xi , p) E2 = − f (xi, yT2 (xi , p))]2 + [yT2 (0, p) − 0]2 + [yT2 (1, p) − 1.23125]2 . (4.13) [ 2 dx i=0. After training ten thousand times, we obtain the following results given in Table 4.5 and Figure 4.3:. 15. DOI:10.6814/NCCU201900919.

(25) x 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0. yE 0 0.01001671 0.04026955 0.09138331 0.16445804 0.26116848 0.38393034 0.53617152 0.72278149 0.95088489 1.23125294. yT1 0 0.03030576 0.07362532 0.12992535 0.19962714 0.28401546 0.38597206 0.5113514 0.67159853 0.88940052 1.23125. yT2 -0.000203253706606 0.00980813880083 0.0401180320842 0.0913012420819 0.164386137519 0.261066376507 0.383816870326 0.536088811958 0.722733687479 0.950824422803 1.23119439266. 政治大. Table 4.5: yT1 and yT2 of eigenvalue problem. 立. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.3: The exact solution and yT1 and yT2 We observe that the total error of yT1 is 0.28973505, and the total error of yT2 is 0.00118241, so yT2 is better than yT1 in this problem. Now we consider another problem with different boundary conditions from (4.8b) [13]: y ′′ + λey = 0 ,. (4.14a). 16. DOI:10.6814/NCCU201900919.

(26) and y(0) = 0, y(1) = 0 .. (4.14b). We set λ = 1, and do the similar process as above and get the result in Table 4.6. Here we use the solver in python scipy [9] to get the numerical solution :. ‧ 國. yn 0 0.049846692471314805 0.08918974460767908 0.11760883045125782 0.13478993851047935 0.14053888143256188 0.13478993851047935 0.11760883045125779 0.08918974460767906 0.04984669247131479 -1.8973538018496328e-19. 立. yT2 -2.0475e-07 0.04984692 0.08918724 0.11760568 0.13478909 0.14053862 0.13478676 0.11760303 0.08918515 0.04984389 -4.77829321e-06. 政治大. 學. x 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0. ‧. Table 4.6: Comparison between the nurerical solution yn and yT2. Nat. sit. y. From the theoretical result in [2], we see that there are two solutions of problem (4.14). We. n. al. er. io. shall use shooting method to obtain the following relation between y ′ (0) and y(1) in Figure 4.4:. Ch. engchi. i n U. v. Figure 4.4: The relations between slopes y ′ (0) and boundary values y(1). 17. DOI:10.6814/NCCU201900919.

(27) And from the Figure 4.4, we find two slopes of the solution satisfying y(1) = 0, that is: y1′ (0) = 0.54935272874400598 ,. (1). y2′ (0) = 10.846900129979208 .. (2). and. So we can use the y ′ (0) as an extra condition in the objective function:. n ∑ d2 yT2 (xi , p) E2 = [ −f (xi, yT2 (xi , p))]2 +[yT2 (0, p)−0]2 +[yT2 (1, p)−0]2 +[yT′ 2 (0, p)−y ′ (0)]2 . 2 dx i=0 (4.15). 立. 政治大. Then we pick the frame [1, 20, 20, 20, 1] in the neural network and use ADAM optimizer. (1). yT2 in Table 4.7 and Figure 4.5:. n. Ch. engchi. (2). y. yn 0 1.07727343 2.12239256 3.0773954 3.80615224 4.09146745 3.80615221 3.07739536 2.12239253 1.07727341 -2.42716680e-11. sit. io. al. (1). yT2 -6.73350614e-06 0.04983686 0.08918295 0.11761002 0.13479295 0.14053508 0.13477962 0.11760196 0.08919419 0.04985561 2.36774456e-06. er. (1). yn 0 0.049846692471314805 0.08918974460767908 0.11760883045125782 0.13478993851047935 0.14053888143256188 0.13478993851047935 0.11760883045125779 0.08918974460767906 0.04984669247131479 -1.8973538018496328e-19. Nat. x 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0. ‧. ‧ 國. (2). 學. After training ten thousand times, we obtain the following two approximate solution yT2 and. i n U. v. (1). (2). yT2 -0.0003091 1.07679451 2.12178255 3.0766249 3.80540402 4.09104248 3.80625067 3.07788276 2.12332098 1.07838873 0.00138331 (2). Table 4.7: Comparison of two numerical solution and yT2 and yT2. 18. DOI:10.6814/NCCU201900919.

(28) 立. 政治大. (1). (2). Figure 4.5: Two approximate solutions of yT2 and yT2. ‧ 國. 學. On the other hand, we get the following relationship between y ′ (0) and λ in Figure 4.6:. ‧. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.6: The relation of λ and y ′ (0) We find that there exists a λ∗ ≈ 3.51417 such that the equation (4.14) has two solutions as λ < λ∗ , has only one solutions as λ = λ∗ , has no solution as λ > λ∗. 19. DOI:10.6814/NCCU201900919.

(29) Finally we consider another type of the eigenvalue problem [2] y. y ′′ + λe[ 1+αy ] = 0 ,. (4.15a). y(0) = 0, y(1) = 0 ,. (4.15b). and. where α and λ are two parameters. When α = 0, it is the equation (4.11). We are going to find the relations between λ and y ′ (0) for α = 0.1 and α = 1 respectly. By shooting method, we get the result in Figure 4.7:. 立. 政治大. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.7: The relations between λ and y ′ (0) As α = 0.1, we find that there exists a λ∗ ≈ 3.20088 such that the equation (4.15) has two solutions as λ < λ∗ , has only one solutions as λ = λ∗ , has no solution as λ > λ∗ . As α = 1, we find that there exists a λ∗ ≈ 1.75745 such that the equation (4.15) has two solutions as λ < λ∗ , has only one solutions as λ = λ∗ , has no solution as λ > λ∗ . Similarly, we can use the argument as above to find the approximate solutions.. 20. DOI:10.6814/NCCU201900919.

(30) 4.3 Initial Value Problem We consider the initial value problem as following: y ′′ = f (t, y, y ′ ) ,. (4.16a). y(a) = α , y ′ (a) = β .. (4.16b). with the initial conditions. We shall study the particular example on Lnae-Emden equation by neural network method.. 4.3.1. 政治大 Lane-Emden Equation 立. ‧ 國. 學. Lane-Emden equation [7] is a complicated problem which describes a variety of phenomena in physics and astrophysics. We consider the following simple equation:. ‧. (4.17a). with the initial condition:. io. u(0) = 1, u′ (0) = 0 .. n. al. Ch. The exact solution is given by [7]:. engchi U. er. sit. y. Nat. 2 u′′ (t) + u′ (t) = 2(2t2 + 3)u(t), 0 < t ≤ 1 , t. (4.17b). v ni. 2. u(t) = et .. (4.18). We define the trial function of type1: uT (t) = (1 + t2 ) + t2 × N (t, p) ,. (4.19). and the corresponding objective function: n ∑ duT1 (ti , p) 2 d2 uT1 (ti , p) − f (ti, uT1 (ti , p), )} , E1 = { 2 dt dt i=0. (4.20a). 2 f (t, u(t), u′ (t)) = − u′ (t) + 2(2t2 + 3)u(t) . t. (4.20b). where. 21. DOI:10.6814/NCCU201900919.

(31) We use the frame [1, 20, 20, 20, 1] of the neural network and ADAM optimizer. We pick the input data ti ∈ [0.01, 1] as below: t0 = 0.01, ti = t0 + 0.0099i, 1 ≤ i ≤ 100 . After training one thousand times, we obtain the following results given in Table 4.8 and Figure 4.8: uT1 1 1.02318904 1.09315011 1.21047736 1.37576557 1.58960547 1.85257897 2.16525421 2.52818075 2.94188478 3.40686454. 政治大. Nat. y. ‧. ‧ 國. 立. uE 1 1.01005017 1.04081077 1.09417428 1.17351087 1.28402542 1.43332941 1.63231622 1.89648088 2.24790799 2.71828183. 學. t 0 1 2 3 4 5 6 7 8 9 10. n. al. er. io. sit. Table 4.8: Comparison between exact solution and uT1. Ch. engchi. i n U. v. Figure 4.8: The exact solution and uT1. 22. DOI:10.6814/NCCU201900919.

(32) We see that the error of approximate solution is not small enough, so we make another condition to improve the accuracy of the solution. After putting t −→ 0 in equation (4.17a), and by L’Hôpital’s rule and initial condition, we obtain: 2u′′ (t) 2u′ (t) = lim = 2u′′ (0) , t→0 t→0 t 1. lim than we get. u′′ (0) = 2 . So in this problem, we have three conditions:. 政治大. u(0) = 1, u′ (0) = 0, u′′ (0) = 2 ,. 立. ‧ 國. 學. hence we define a new trial function satisfying these condition, that is: unT1 (t) = (1 + t2 ) + t3 × N (t, p) ,. ‧ sit. y. Nat. and the corresponding objective function is given by:. n. al. er. n ∑ dunT1 (ti , p) 2 d2 unT1 (ti , p) − f (ti, u (t , p), )} , { nT i 1 2 dt dt i=0. io. En 1 =. (4.21). where f is given by (4.20b).. Ch. engchi. i n U. (4.22). v. After trianing one thousand times, we obtain the following results given in Table 4.9 and Figure 4.9:. 23. DOI:10.6814/NCCU201900919.

(33) t 0 1 2 3 4 5 6 7 8 9 10. uE 1 1.01005017 1.04081077 1.09417428 1.17351087 1.28402542 1.43332941 1.63231622 1.89648088 2.24790799 2.71828183. uT1 1 1.02318904 1.09315011 1.21047736 1.37576557 1.58960547 1.85257897 2.16525421 2.52818075 2.94188478 3.40686454. unT1 1 1.01105706 1.04851669 1.1189488 1.22910843 1.38593668 1.59655931 1.86828299 2.20858899 2.62512426 3.12569002. 政治大. Table 4.9: Comparison between old uT1 and new unT1. 立. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.9: The exact solution and uT1 and unT1 Form the Figure 4.9, we see that the result is still not good enough, so we try to use the trial function method of type2. We define the trial function: uT2 (t, p) = N (t, p) ,. 24. DOI:10.6814/NCCU201900919.

(34) and the corresponding objective function: n ∑ d2 uT2 (ti , p) duT2 (ti , p) 2 )] + [uT2 (0, p) − 1]2 + [u′T2 (0, p) − 0]2 , (4.23) [ − f (ti, uT2 (ti , p), 2 du du i=0. where f is given by (4.20b). After trianing one thousand times, we get uT2 , and we compare it with uT1 and unT1 . The result is given in Table 4.10 and Figure 4.10: uE 1 1.01005017 1.04081077 1.09417428 1.17351087 1.28402542 1.43332941 1.63231622 1.89648088 2.24790799 2.71828183. unT1 1 1.01105706 1.04851669 1.1189488 1.22910843 1.38593668 1.59655931 1.86828299 2.20858899 2.62512426 3.12569002. uT2 0.99793039 1.00794369 1.03868652 1.09220259 1.1716522 1.28201534 1.43088164 1.62936526 1.89313269 2.24393207 2.7132886. 政治大. ‧. ‧ 國. 立. uT1 1 1.02318904 1.09315011 1.21047736 1.37576557 1.58960547 1.85257897 2.16525421 2.52818075 2.94188478 3.40686454. 學. t 0 1 2 3 4 5 6 7 8 9 10. n. al. er. io. sit. y. Nat. Table 4.10: Comparision among uT1 , unT1 and uT2. Ch. engchi. i n U. v. Figure 4.10: The exact solution and uT1 , unT1 and uT2 Remark 4.1 : When we add one extra condition in trial function of type1, we can improve 25. DOI:10.6814/NCCU201900919.

(35) the accuracy of the solution.. We also see that uT2 is more accurate than unT1 in Figure 4.10.. 4.4 Systems of Differential Equations In this section, we consider two types of the ecological systems: Rabbit versus Sheep problem and Lokta-Volterra model.. 4.4.1 Rabbit versus Sheep Problem. 治政大food source. Consider two populations competing for the same 立 population of rabbits and sheeps respectively [12].. Let x and y be the. ‧ 國. 學 ‧. y ′ = g(x, y) = y(2 − x − y) .. io. sit. Nat. x′ = f (x, y) = x(3 − x − 2y) ,. y. Each species follows logistic growth plus a competition term. (4.24a). n. al. er. Then we have four fixed points, (0, 0), (1, 1), (0, 2) and (3, 0). And we can find the Jacobian matrix J(x, y):. Ch . engchi J(x, y) =  3 − 2x − 2y −y. i n U. v. −2x. 2 − x − 2y.  .. We can find the eigenvalues of the matrix J. We find the eigenvalues of J(0, 2) are −1 and −2, and the eigenvalues of J(3, 0) are −3 and −1. Since the eigenvalues are real negative, so the fixed points (0, 2) and (3, 0) are stable by theory [1]. The eigenvalues of J(0, 0) are 3 and 2, √ and the eigenvalue of J(1, 1) is −1 ± 2. Since one of the eigenvales is positive, so the fixed points (0, 0) and (1, 1) are unstable. In this section, we are going to observe the solution’s behavior of the system starting from some points. First we consider the point: x(0) = 3 , y(0) = 1 .. (4.24b). 26. DOI:10.6814/NCCU201900919.

(36) We define the trial function of type2: xT2 (ti , p) = N1 (ti , p) , yT2 (ti , p) = N2 (ti , p) , and the corresponding objective function: n ∑ dxT2 (ti , p) E2 = [ − f (xT2 (ti ), yT2 (ti ))]2 + [xT2 (0, p) − 3]2 dt i=0. dyT2 (ti , p) − g(xT2 (ti ), yT2 (ti ))]2 + [yT2 (0, p) − 1]2 . dt. (4.25) 政治大 Where the neural network architecture 立 N (x, p) and N (x, p) are given in the following Figure +[. 1. ‧. ‧ 國. 學. 4.11:. 2. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.11: The architecture of the reconstruct neural network We use the same frame [1, 20, 20, 20, 1] in neural networks N1 and N2 , and choose the input data. By : ti = 0.03i, 1 ≤ i ≤ 100 . After training ten thousand times, we get the result given in Table 4.11:. 27. DOI:10.6814/NCCU201900919.

(37) t 0 0.5 1 1.5 2 2.5 3. xn 3 2.02044466 1.99687118 2.12521159 2.29197316 2.45936698 2.60823506. xT 2 2.99997275 2.02066594 1.99689965 2.126161 2.29435994 2.46093313 2.6083037. yn 1 0.59070885 0.45899089 0.36371409 0.27930029 0.20507731 0.14385463. yT2 0.99985524 0.59088628 0.45880094 0.36240943 0.27822095 0.20499946 0.14389316. Table 4.11: Comparison between the exact solution and xT2 , yT2 Similarly, we do the same computation as above at the other points:. ‧. x(0) = 1 , y(0) = 0.5 .. (4.24d). Nat. sit. y. (4.24e). In conclusion, we obtain the following result in Figure 4.12:. io. n. al. er. and. x(0) = 0.5 , y(0) = 1 ,. (4.24c). 學. and. ‧ 國. 立. 治政 x(0) = 3 , y(0) = 2大 ,. Ch. engchi. i n U. v. Figure 4.12: Four neural network solutions and numerical solution with fixed points. 28. DOI:10.6814/NCCU201900919.

(38) We see that the approximate solution starting at (0.5, 1) approaches the stable point (0, 2). The approximate solution starting at (1, 0.5), (3, 1) and (3, 2) approach the stable point (3, 0). It is in accord with the theoretical result.. 4.4.2 Lokta-Volterra Model Lokta-Voterra model [5] is frequently used to describe the dynamics of biological systems in which two species, predator and prey, interact. We consider the following model: x′ = f (x, y) = 3x(1 − y) ,. with the initial condition :. 立. 政治大. y ′ = g(x, y) = y(x − 1) ,. (4.26a). ‧ 國. 學. x(0) = 1.2 , y(0) = 1.1 , 0 ≤ t ≤ 10 ,. (4.26b). ‧. where x(t) is the population the of prey and y(t) is the population of the predator. x′ (t) and y(t). y. Nat. sit. represent the instantaneous growth rates of the two populations, and t represents time. We can. n. al. er. io. obtain the following relations:. Ch. i n U. v. 1 (x − lnx) = lny − y + C , 3. engchi. (4.27). where the C is: C ≈ 1.3439159679310235 . Note that the equation (4.26) shows that it is the periodic orbit in x − y plane. In the following we shall use the neural network method to solve the equation (4.26). We define the trial function of type2: xT2 (ti , p) = N1 (ti , p) , yT2 (ti , p) = N2 (ti , p) ,. 29. DOI:10.6814/NCCU201900919.

(39) and the corresponding objective function: n ∑ dxT2 (ti , p) − f (xT2 (ti ), yT2 (ti ))]2 + [xT2 (0, p) − 1.2]2 E2 = [ dt i=0. +[. dyT2 (ti , p) − g(xT2 (ti ), yT2 (ti ))]2 + [yT2 (0, p) − 1.1]2 . dt. (4.28). We use the same frame [1, 20, 20, 20, 1] in neural networks N1 and N2 . And the input data is: ti = 0.1i, 1 ≤ i ≤ 100 .. 政治大. After training ten thousand times, we get the result is given in Figure 4.13 and Figure 4.14:. 立. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.13: The numerical solution and (xT2 , yT2 ). 30. DOI:10.6814/NCCU201900919.

(40) 立. 政治大. ‧ 國. 學. Figure 4.14: The numerical solution and xT2 ,yT2 with t. ‧. We see that it is a periodic solution with peroid T ≈ 3.6275 by [11]. Hence the error of. Nat. al. er. io. sit. the solution, so we add an extra condition in objective function.. y. approximate solution is not small enough from Figure 4.14. In order to improve the accuracy of. v. n. E2∗ = E2 + ([xT2 (3.6275, p) − 1.2]2 + [yT2 (3.6275, p) − 1.1]2 +. Ch. engchi. i n U. [xT2 (7.2551, p) − 1.2]2 + [yT2 (7.2551, p) − 1.1]2 + [xT2 (10.8827, p) − 1.2]2 + [yT2 (10.8827, p) − 1.1]2 ) .. (4.29). We use [1, H1 , H2 , ..., H10 , 1] with Hi = 100, 1 ≤ i ≤ 10 in neural networks N1 and N2 . and the input data is: ti = 0.11i, 1 ≤ i ≤ 100 . After training ten thousand times, and the result is given as following in Figure 4.15 and Figure 4.16:. 31. DOI:10.6814/NCCU201900919.

(41) 立. 政治大. Figure 4.15: Numerical solution and (xT2 , yT2 ). ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.16: Numerical solution and xT2 , yT2 with t Remark 4.2 : After adding the peroid condition in the objective function, we obtain a good approximate solutions.. 32. DOI:10.6814/NCCU201900919.

(42) 4.5 Partial Differential Equations In this section, we shall consider three different types of partial differential equations in a bounded domain [14]. We use the neural network method to solve those problem.. 4.5.1 Laplace’s Equation Laplace’s equation is the simplest example of elliptic differential equation, which is important in physics and fluid dynamics. We shall consider the boundary value problem in a bounded domain: uxx + utt = 0 , 0 ≤ x ≤ 1 , 0 ≤ t ≤ 1 , with the boundary conditons:. 立. (4.30a). 政治大. ‧ 國. 學. u(x, 0) = 0 , u(x, 1) = sin(πx) , 0 ≤ x ≤ 1 , u(0, t) = 0 , u(1, t) = 0 , 0 ≤ t ≤ 1 .. ‧. (4.30b). sinhπt × sinπx . sinhπ. er. io. al. uE (x, t) =. sit. y. Nat. By using the method of separation of variables, we obtain the exact solution: (4.31). n. v i n C is given below in Figure The graph of the exact solution h e n g c h i U 4.17:. 33. DOI:10.6814/NCCU201900919.

(43) 立. 政治大. ‧ 國. 學. Figure 4.17: The exact solution (4.30). Hereafter, we use the trial function of type1:. ‧. io. n. er. and the objective function is:. al. (4.32). sit. y. Nat. uT1 (x, t, p) = t × (sin(πx)) + x × (1 − x) × t(1 − t) × N (x, t, p) ,. Ch. i n U. v. m ∑ n ∑ ∂ 2 uT1 (xi , tj ) ∂ 2 uT1 (xi , tj ) 2 E1 = { + } . 2 2 ∂t ∂x j=0 i=0. engchi. (4.33). Where n and m are the number of the input data in x and t respectively, we use the frame [2, 20, 20, 20, 1] of the neural network and ADAM optimizer. We pick the input data xi ∈ [0, 1] and tj ∈ [0, 1] as below: xi = 0.1i , 0 ≤ i ≤ 10 , tj = 0.1j , 0 ≤ j ≤ 10 . After training one thousand times, we obtain the following results given in Table 4.12:. 34. DOI:10.6814/NCCU201900919.

(44) x 0 0.5 0.5 0.4 0.8 1.0. t 0 0.1 0.4 0.6 1.0 0.3. uE 0 0.027652588 0.139797772 0.264934229 0.58778525 1.15408851401e-17. uT1 0 0.04178677 0.16957856 0.25728787 0.58778525 3.67394040e-17. Table 4.12: Comparison between the uE and uT1. 立. 政治大. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.18: The graph of uT1 We can observe that the error |uE (x, t) − uT1 (x, t)| in the region is shown in Figure 4.19:. 35. DOI:10.6814/NCCU201900919.

(45) 政治大. 立. ‧ 國. 學. Figure 4.19: The error of uT1. We see that the maximum error of |uE (x, t)−uT1 (x, t)| is 0.192 at(0.5, 0.7), and the accuacy. ‧. of approximate solution near the boundary is better than in the central part of the domain.. sit. y. Nat. io. n. al. er. 4.5.2 Heat Equation. v. Heat equation describes the heat distribution in the metal rod. We shall solve an initial. Ch. boundary value problem in a region: ut =. engchi. i n U. 1 uxx , 0 < x < 1 , t > 0 , π2. (4.34a). with initial and boundary conditons: u(x, 0) = sin(πx) , 0 < x < 1 , u(0, t) = 0 , u(1, t) = 0 , t > 0 .. (4.34b). By using the method of separation of variables, we obtain the exact solution: uE (x, t) = sin(πx)e−t .. (4.35). 36. DOI:10.6814/NCCU201900919.

(46) The graph of the exact solution is given in Figure 4.20:. 政治大. 學. Figure 4.20: The exact solution of (4.34). ‧. ‧ 國. 立. We define the trial functions of type1:. sit. y. Nat. n. al. Ch. and the corresponding objective function:. E1 =. (4.36). er. io. uT1 (x, t, p) = sin(πx) + x × (1 − x) × t × N (x, t, p) ,. m ∑ n ∑ j=0 i=0. {π. engchi. 2 ∂uT1 (xi , tj ). ∂t. i n U. v. ∂ 2 uT1 (xi , tj ) 2 − } . ∂x2. (4.37). Where n and m are the number of the input data in x and t respectly, we use the frame [2, 20, 20, 20, 1] of the neural network and ADAM optimizer. We pick the input data xi ∈ [0, 1] and tj ∈ [0, 1] as below: xi = 0.1i , 0 ≤ i ≤ 10 , tj = 0.1j , 0 ≤ j ≤ 10 . After training one thousand times, we obtain the following results given in Table 4.13:. 37. DOI:10.6814/NCCU201900919.

(47) x 0 0.5 0.5 0.4 0.8 1.0. t 0 0.1 0.4 0.6 1.0 0.3. uE 0 0.904837418036 0.670320046 0.521950882 0.216234110142 9.07240662708e-17. uT1 0 0.91394536 0.65539701 0.45536706 0.03419138 1.22464680e-16. Table 4.13: Comparison between uE and uT1. 立. 政治大. ‧. ‧ 國. 學. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.21: The graph of uT1 We can observe that the error |uE (x, t) − uT1 (x, t)| in the region is shown in Figure 4.22:. 38. DOI:10.6814/NCCU201900919.

(48) 立. 政治大. ‧ 國. 學. Figure 4.22: The error of the uT1. From Figure (4.22), the maximum error of the approximate solution is 0.232 at (0.5, 1).. ‧ sit. y. Nat. 4.5.3 Wave Equation. n. al. er. io. We shall consider an initial boundary value problem in a bounded domain. Ch. u =u. ,. e nttg c xxh i. i n U. v. (4.38a). with the initial and boundary condition: u(x, 0) = sin(πx) ,. ∂u(x, 0) =0, 0≤x≤1 , ∂t. u(0, t) = u(1, t) = 0 , 0 ≤ t ≤ 1 .. (4.38b). By using the method of separation of variables, we obtain the exact solution: u(x, t) = sin(πx)cos(πt) .. (4.39). The graph of the exact solution is given below in Figure 4.23:. 39. DOI:10.6814/NCCU201900919.

(49) Figure 4.23: The exact solution of (4.38). 學. ‧ 國. 立. 政治大. We define the trial functions of type1:. ‧. io. n. er. and the corresponding objective function:. al. Ch. (4.40). sit. y. Nat. uT (x, t, p) = (1 − t2 ) × sin(πx) + x × (1 − x) × t2 × N (x, t, p) ,. i n U. v. m ∑ n ∑ ∂ 2 uT1 (xi , tj ) ∂ 2 uT1 (xi , tj ) 2 E1 = { − } . 2 2 ∂t ∂x j=0 i=0. engchi. (4.41). Where n and m are the number of the input data in x and t respectly, we use the frame [2, 20, 20, 20, 1] of the neural network and ADAM optimizer. We pick the input data xi ∈ [0, 1] and tj ∈ [0, 1] as below: xi = 0.1i , 0 ≤ i ≤ 10 , tj = 0.1j , 0 ≤ j ≤ 10 . After training one thousand times, we obtain the following results given in Figrue 4.24:. 40. DOI:10.6814/NCCU201900919.

(50) 立. 政治大. ‧ 國. 學. Figure 4.24: The graph of uT1. ‧. Because the accuracy of uT1 is not good, so we add a symmetry condition, u(x, 21 ) = 0 and u(x, 1) = −sin(πx) in the trial function of type1:. y. Nat. n. al. er. io. sit. unT1 (x, t, p) = (1−6t2 +4t3 )×(sin(πx))+x×(1−x)×t2 ×(1−t)×(2−t)×N (x, t, p) . (4.42) and we obtain the result in Table 4.14 and Figure 4.25: x 0 0.5 0.5 0.4 0.8 1.0. t 0 0.1 0.4 0.6 1.0 0.3. Ch. engchi. uE 0 0.951056516295 0.309016994375 -0.293892626146 -0.587785252292 7.19829327806e-17. i n U. v. unT1 0 0.95006481 0.30822565 -0.29104359 -0.58778525 6.95599382e-17. Table 4.14: Comparison between uE and unT1. 41. DOI:10.6814/NCCU201900919.

(51) 立. 政治大. ‧ 國. 學. Figure 4.25: The graph of unT1. We can observe that the error |uE (x, t) − unT1 (x, t)| in the region is shown in Figure 4.26:. ‧. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. Figure 4.26: The error of the unT1 From Figure (4.26), the maximum error of the approximate solution is 0.0509 at (0.5, 0.8). Remark 4.3 : When we add extra conditions from the porperty such as symmetry, we can improve the accuracy of the approximate solution. 42. DOI:10.6814/NCCU201900919.

(52) Bibliography [1] Ravi P Agarwal and Donal O’Regan. An introduction to ordinary differential equations. Springer Science & Business Media, 2008.. 政治大 volume 83. Springer Science & Business Media, 2013. 立. [2] Jerrold Bebernes and David Eberly. Mathematical problems from combustion theory,. ‧ 國. 學. [3] Richard L Burden and J Douglas Faires. Numerical analysis(7th). Brooks/Cole, 2001. [4] Matt Curnan, Siddharth Deshpande, Hari Thirumalai, Zhaofeng Chen, John Michael, et al.. ‧. Solving odes with a neural network and autograd. https://kitchingroup.cheme.cmu.edu/. sit. y. Nat. blog/2017/11/28/Solving-ODEs-with-a-neural-network-and-autograd/.. n. al. er. io. [5] Vivek Dua. An artificial neural network approximation based decomposition approach for. v. parameter estimation of system of ordinary differential equations. Computers & chemical. Ch. engchi. engineering, 35(3):545–553, 2011.. i n U. [6] Ji-Huan He. Variational iteration method for autonomous ordinary differential systems. Applied Mathematics and Computation, 114(2-3):115–123, 2000. [7] Hamid A Jalab, Rabha W Ibrahim, Shayma A Murad, Amera I Melhum, and Samir B Hadid.. Numerical solution of lane-emden equation using neural network.. In AIP. Conference Proceedings, volume 1482, pages 414–418. AIP, 2012. [8] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [9] Hans Petter Langtangen and Hans Petter Langtangen. A primer on scientific programming with Python, volume 6. Springer, 2011.. 43. DOI:10.6814/NCCU201900919.

(53) [10] Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237, 2019. [11] Shagi-Di Shih et al. The period of a lotka-volterra system1. Taiwanese Journal of Mathematics, 1(4):451–470, 1997. [12] Steven H Strogatz. Nonlinear Dynamics and Chaos with Student Solutions Manual: With Applications to Physics, Biology, Chemistry, and Engineering. CRC Press, 2018. [13] Luma NM Tawfiq and Othman M Salih. Design feed forward neural network to solve eigenvalue problems with dirishlit boundary conditions. Int. J. Modern Math. Sci, 11(2):. 政治大 [14] Neha Yadav, Anupam 立 Yadav, Manoj Kumar, et al. An introduction to neural network 58–68, 2014.. ‧. ‧ 國. 學. methods for differential equations. Springer, 2015.. n. er. io. sit. y. Nat. al. Ch. engchi. i n U. v. 44. DOI:10.6814/NCCU201900919.

(54)