應用在圖形辨識的自發性比例式記憶體細胞非線性網路的穩定度分析

全文

(1)»ñø;. é

(2) .é@~X }ÿ¡Z Tà3%ïIÝsPf»PB7Þi& aPçÝ%5 Stability Analysis of Autonomous Ratio-Memory Cellular Nonlinear Network for Pattern Recognition. @~ß : |8K ¼0>0 : Ò¥$ ºÓ»×yOÚ`.

(3) Tà3%ïIÝsPf»PB7Þi& aPçÝ%5 Stability Analysis of Autonomous Ratio-Memory Cellular Nonlinear Network for Pattern Recognition. @~ß : |8K Student : Su-Yung Tsai ¼0>0 : Ò¥$ Advisor : Chung-Yu Wu »ñø;. é^.o é

(4) . é@~X }ÿ¡Z A Dissertation Submitted to Department of Electronics Engineering and Institute of Electronics College of Electrical and Computer Engineering National Chiao-Tung University in partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Electronics Engineering July 2011 Hsin-Chu, Taiwan, Republic of China. ºÓ»×yOÚ`.

(5) Tà3%ïIÝsPf»PB7Þi& aPçÝ%5 .ß : |8K é^.o. ¼0>0 : Ò¥$. »ñø;. é

(6) . é@~X `. ß\|

(7) §Ey×°à¨*ÝóÅ ]°)QP°bßÝÅ ® Þ»AÅ 5vïI×°@~ßõ8*\±è²|ÐÝ]P/7 VÆ¼ß\">

(8) §Å ®ÞÝæ.Ûãyß\à#B7ã n Ý

(9) §Å ïI&ÆèsPf»PB7Þi&aPç (|ìÌ f»PB7ç) ¼%®ÐÝB7f»PB7çÎ×Í&aP Ù¸ÝNÍÞi!ÝÏ&ÞiBh=#9°Bh=#ÝxEy3NÍ ÞiîÝvfíáÇ`Gr×R!`

(10) §ßvÚçnùÌb9ËË©²: NÍÞ iÍ!ÝÏ&ÞiBh=#|CvfíáÇ`Gr×R!`

(11) §9ËË© ²¸ÿf»PB7çÊ)|vfPølé¼%® EyÅ ïIP%VÂCõVÚS Ey%5Î¥Ý¯× ÝÅ FÞºµÍX3ÝõVÚS ¼[eÕXETÝ%VÂ¢ãA-û ÙG§Í¡Zè×ÍíáÅ Ý1FÝõVÚS ¬v¢ãA-ûÙG i.

(12) §Í¡Zè×Í%°¼xhõVÚS é\ÿaCÒ÷éÆÝÌF¼:0ÕóÂ5°Ýt` M SRóÂ%ÎÁÍ¥Ý¯tÝóÂ%²¨²×ÍÈÞµÎý0Ý B7F×°¬&ATçÝB7FD39°ý0ÝB7Fºª±%Ýï I£Í¡Z¸àU÷ÿPC©Pè 5°¼"D3ëËµìÝ×îf» PB7çÝZóÂ5Ýt` M9ëËµ5½ÎXbÞi

(13) 3a P ×ÐÞi á÷õ ËÐÞi á÷õ N×ËµbÍETÝU÷ ÿPÝ\&f9ëËµÝ©Pè 5°&Æ.0ßB-¦ÇÝ2 P¼ª±ý0B7FÝóê h²ãh©Pè 5°.¡3×Í fì¸àZóÂ5Ý f»PB7çÝ%Víyvf=PÝf»PB7çÝ%Ví 4QÍ¡Z¬Î.0ÞîÝf»PB7çÝ©Pè 5&ÆÈv« Ý'æJD3Í¡Z|×îCÞîÝf»PB7çÝ»¼Y¹Í ¡ZÝ']P. ii.

(14) Stability Analysis of Autonomous Ratio-Memory Cellular Nonlinear Network for Pattern Recognition Student : Su-Yung Tsai. Advisor : Chung-Yu Wu. Department of Electronics Engineering and Institute of Electronics National Chiao-Tung University Abstract Humans brains can resolve many complex image tasks such as pattern recognition and segmentation which are still difficult for digital image processing algorithms to have a satisfying result. Some researchers believe that the neocortex recalls patterns autoassociatively. They pointed out the reason for the brain to efficiently resolve these image tasks is that the brain retrieves the answer from memory. We propose the ARMCNN (Autonomous Ratio-Memory Cellular Nonlinear Network) structure to implement the associative memory for pattern recognition. The ARMCNN is a nonlinear system with each cell locally coupled to its neighbors. This locally connected structure processes the input real-time analog signals at each cell simultaneously. Human retinas also have these two characterictics : the locally connected structure and the ability to process the input real-time analog signal. These two characterictics make ARMCNNs suitable to implement in analog VLSI. iii.

(15) For pattern recognition, stability analysis is essential to understand the steady state values and the domain of attraction. The initial image point will converge to its corresponing steady state value depending on which corresponding domain of attraction the initial point belong. This thesis aims to provide a conservative domain of attraction for the input image by Lyapunov stability analysis. From this Lyapunov stability analysis, a graphical method is proposed to construct the domain of attraction. From the view point of computer simulation and discrete-time circuit implementation, it is crucial to obtain the maximum time step in the numerical integration algorithm without causing numerical instability. Besides the numerical instability, another issue is the spurious memory points. Some non-binary equilibrium points may exist. These spurious memory points lower the recognition rate. This thesis uses the diffusion model and the eigenspace decomposition method to examine the maximum forward Euler time step for one-dimensional ARMCNN in three cases: all neurons in linear regions, one end neuron entering the saturation region, and two end neurons entering the saturation region. Each case has a distinct boundary condition for its corresponding diffusion model. From the eigenspace decomposition in these three cases, an analytic neuron gain is derived to lower the amount of spurious memory points. In addition, the eigenspace decomposition implied a sufficient condition to guarantee that the forward Euler ARMCNN has the same steady state output as the continuous time ARMCNN. Although the eigenspace decomposition for 2-D ARMCNN is not derived, this thesis suggests a similar design principle exists for 2-D ARMCNN. 1-D and 2-D examples are given in this thesis to support the ARMCNN design equations.. iv.

(16) * ´&E&Ý¼0>0Ò¥$>0lîtÝÝ ¯ 3&}ÿ ° Íì2&¼0ÜÃ r@_>03ó.§¡î&Ý¼ 0¬¼0&Ñ@Ý@~ÝV rO>0¯&^WÅ

(17) §Ý ¬`2&È¯&.êÕóÅ

(18) §ÝÃÍæ§ zW±>0 ÐrD>0@s&EûßéC0gÛ&éÝ·¶ > 0xX¯>03;Gæ§C ^îÝ>0 WÉ >0|>0 ðð <& r>0?Â×>0±í>0¡ZòÝ h²&ô ø 307 @ÝÒ+/>0Þ¼>0Wó$>0¯&b ×ÍY?Ý

(19) ®( 307 @Ý.!.|C.¯Æ39¿ O &&9ÝQRnï©½Î3é8Ýÿa3@éÝÚ' 3ç¦î ¯Æ&ô Ã§ÝÉÐÅ 3Å¼îÝQR. b¯Æ&Ýßú?Y? h²&©½ &Ý

(20) Debbie |C&Ýß. ¯ÆÝY¹ <¸&bæW9 @~3h=T2'îF=t¡GÞ&Ý¡Z¤ &ÝÝlÒ. |8K »ñø;. ºÓ»×yOÚ` v.

(21) vi.

(22) Contents Z`. i. English Abstract. iii. *. v. List of Tables. xi. List of Figures. xiii. 1. 2. 3. Introduction. 1. 1.1. Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 1. 1.2. Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 3. Standard CNN. 5. 2.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 5. 2.2. Standard CNN State Equations . . . . . . . . . . . . . . . . . . . . . . .. 6. 2.3. CNN Image Processing Examples . . . . . . . . . . . . . . . . . . . . .. 9. 2.3.1. Connected Component Detection (CCD) . . . . . . . . . . . . .. 9. 2.3.2. Global Connectivity Dectection (GCD) . . . . . . . . . . . . . .. 10. ARMCNN. 15. 3.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 15. 3.2. ARMCNN Learning Period . . . . . . . . . . . . . . . . . . . . . . . . .. 16. 3.2.1. 16. Operation with Elapsed Time . . . . . . . . . . . . . . . . . . . vii.

(23) 3.2.2 3.3. 4. 5. 18. ARMCNN Recognition Period . . . . . . . . . . . . . . . . . . . . . . .. 19. 3.3.1. ARCMNN State Equations . . . . . . . . . . . . . . . . . . . . .. 19. 3.3.2. Equilibrium Point . . . . . . . . . . . . . . . . . . . . . . . . . .. 23. 3.3.3. Relationship to Standard CNN . . . . . . . . . . . . . . . . . . .. 25. Lyapunov Stability Analysis. 27. 4.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 27. 4.2. Stability Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 27. 4.3. Lyapunov Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 29. 4.3.1. A Lyapunov Local Asymptotic Theorem . . . . . . . . . . . . .. 29. 4.4. ARMCNN Stability Analysis . . . . . . . . . . . . . . . . . . . . . . . .. 30. 4.5. Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 38. Numerical Stability Analysis. 45. 5.1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 45. 5.2. Diffusion Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 46. 5.2.1. Exact Solution of the Diffusion Model . . . . . . . . . . . . . . .. 49. 5.2.2. Computational Method . . . . . . . . . . . . . . . . . . . . . . .. 51. 5.2.3. Maximum Time Step . . . . . . . . . . . . . . . . . . . . . . . .. 52. 5.2.4. Single Mode Operation . . . . . . . . . . . . . . . . . . . . . . .. 56. 5.2.5. One End Neuron Enters the Saturation Region . . . . . . . . . .. 57. 5.2.6. Both End Neurons Enter the Saturation Regions with the Same Sign 63. 5.3. 6. Operation without Elapsed Time . . . . . . . . . . . . . . . . . .. ARMCNN Convergence Analysis . . . . . . . . . . . . . . . . . . . . .. 66. 5.3.1. 68. Simulation Verification . . . . . . . . . . . . . . . . . . . . . . .. Conclusions and Future Works. 79. 6.1. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 79. 6.2. Recommendations for Future Investigation . . . . . . . . . . . . . . . . .. 80. Appendix A Bitmap File format. 83 viii.

(24) Appendix B Eigenvalues of 1-D ARMCNN. 85. B.1 All Neuros in Linear Regions . . . . . . . . . . . . . . . . . . . . . . . .. 85. B.2 One End Neuron in Saturation Region . . . . . . . . . . . . . . . . . . .. 86. B.3 Both End Neurons Enter the Saturation Region . . . . . . . . . . . . . .. 87. Appendix C Discrete cosine transform. 89. C.1 All Neurons in the Linear Regions . . . . . . . . . . . . . . . . . . . . .. 89. C.2 One End Neuron Enters the Saturation Region . . . . . . . . . . . . . . .. 92. Bibliography. 95. F. 103. ix.

(25) x.

(26) List of Tables 3.1. Equivalent representations between the N-ARMCNN and the S-CNN . .. 26. 6.1. Comparison of various CNN works on associative memory. . . . . . . . .. 80. xi.

(27) xii.

(28) List of Figures 2.1. The PWL output or activation function. . . . . . . . . . . . . . . . . . .. 2.2. A 2-D CNN consisting of a 4 × 4 rectangular array with 4 rows counted. 7. in the i direction, and 4 columns counted in the j direction. The radius of the sphere of influence is Ra = 1. Virtual cells connceted to cell C(4, 1) are shown in the dotted line . . . . . . . . . . . . . . . . . . . . . . . . .. 8. 2.3. Input image applied as vxij (0) before CCD operation. . . . . . . . . . . .. 10. 2.4. Output image, vyij (200τ), after CCD operation. Tsim = 200τ. . . . . . . .. 11. 2.5. The input image vuij for the GCD task. . . . . . . . . . . . . . . . . . . .. 12. 2.6. The state vxij (0) for the GCD task. Compared with Figure 2.5, the state at (3, 3) and (3, 21) are white. The coordinates follow the definition in Figure 2.2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 12. 2.7. The output image, vyij (50τ), after 50τ. . . . . . . . . . . . . . . . . . . .. 13. 2.8. The output image, vyij (150τ), after 150τ. . . . . . . . . . . . . . . . . .. 13. 3.1. (a) Chinese character ONE. (b) Chinese character TWO. (c) Chinese character FOUR. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 3.2. The survived ratio weights in the 9×9 ARMCNN after learning the three Chinese characters ONE, TWO, and FOUR of Figure 3.1. 3.3. 3.4. 19. . . . . . . . .. 20. The neuron model of ARMCNN located at the ith row and jth column. If without self-feedback, wijij = 0. . . . . . . . . . . . . . . . . . . . . . .. 22. The transfer characteristic of the PWL activation function, σp (·). . . . . .. 23. xiii.

(29) 3.5. A 1-D five-neuron (5N) system having two binary equilibrium points (x1 (∞), x2 (∞), x3 (∞), x4 (∞), x5 (∞)) as ±(Isat R, Isat R, Isat R, Isat R, Isat R). 24. 4.1. Ωc = {x ∈ Rn |V (x) = xT P x ≤ c} is bounded and contained in D. . . . .. 30. 4.2. The shifted PWL activation function, σ(zi + 1). . . . . . . . . . . . . . .. 31. 4.3. The shifted PWL activation function, σ(zi − 1). . . . . . . . . . . . . . .. 32. 4.4. A 1-D three-neuron (3N) subsystem with two binary equilibrium points of black-black-white and white-white-black, or (x1 (∞), x2 (∞), x3 (∞)) = ±(Isat R, Isat R, −Isat R). . . . . . . . . . . . . . . . . . . . . . . . . . . .. 4.5. The shifted sinusoidal activation function of σ(z1 + 1). The sinusoidal activation function is defined in (4.19). . . . . . . . . . . . . . . . . . .. 4.6. 39 40. The trajectories in thick lines at three different initial states converge to the (1, 1, −1). The projection onto the plane x3 = −1 is shown in the thin dotted lines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 4.7. The normalized trajectories. kz(t)k kz(0)k. at three different initial points converge. faster than e−0.2t towards the (0, 0, 0). e−0.2t is plotted in the solid line. . . 5.1. 43 44. A circuit implementation for a 1-D 5N ARMCNN. For each synapse connection, two voltage to current converters (V/I) and two current mirrors are required.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 47. 5.2. An equivalent diffusion circuit for (5.1). . . . . . . . . . . . . . . . . . .. 48. 5.3. Decompostion of the initial state vector. Each rk is multiplied by a corresponding coefficient ak . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 5.4. 54. Eigenvectors rk fall on the continuous cosine functions. As k increases, the corresponding eigenvector sequence (j = 0, . . . , P −1) oscillates more rapidly. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 55. 5.5. An equivalent diffusion circuit for (5.30). 58. 5.6. The initial point is (0.05, −0.67, 0.14, 0.31, 0.37). g = 1.4 and The x-axis variable is. 5.7. t τ. . . . . . . . . . . . . . . . . .. The x-axis variable is. = 0.83.. with t = n∆t, n = 0, 1, 2, . . .. . . . . . . . . . . .. The initial point is (0.05, −0.67, 0.14, 0.31, 0.37). g = 1.4 and t τ. ∆t τ. ∆t τ. = 0.02.. with t = n∆t, n = 0, 1, 2, . . .. . . . . . . . . . . . xiv. 70 71.

(30) 5.8. Different sets of initial states are shown. The optimal decision boundary line is shown in the dotted line.. 5.9. . . . . . . . . . . . . . . . . . . . . . .. 72. The numbers of initial state vectors in set A,B,C,D are plotted at various time steps. The neuron gain is 1.4. The simulation time is 200τ. . . . . . ∆t τ. .. 74. time steps. The neuron gain is 2.0. The simulation time is 200τ. . . . . .. 75. 5.10 The number of state vectors in set E increases as. is larger than 0.83.. 73. 5.11 The numbers of initial state vectors in set A,B,C,D are plotted at various 5.12 The number of state vectors in set E increases as. ∆t τ. is larger than 0.66.. .. 76. FOUR by ARMCNNs. . . . . . . . . . . . . . . . . . . . . . . . . . . .. 78. 5.13 The RR of recognizing three 9 × 9 Chinese characters ONE, TWO, and. A.1 Transformation between the read-in byte from the BMP file and the CNN state value. Bt denotes the read-in byte. . . . . . . . . . . . . . . . . . .. xv. 84.

(31) Chapter 1 Introduction 1.1. Motivation. Some basic image tasks such as image segementation, recognition and associatvie memory [1, 2] are difficult to have a satisfying result using digital image processing algorithms, while these basic image tasks are considered easy for huamn beings to perform. For example, if a cat is hidden partially in a picture, and one is asked to push the button to answer whether the cat is in the picture, it takes less than half a second to identify this cat. To quote Jeff Hawkins from [3], “... From the time light enters your eye to the time you press the button, a chain no longer than one hundred neurons could be involved...”. On the other hand, it is difficult for a digital computer to excute one hundred steps to identify which kind of animal is in a picture. Even with many parallel computers, each computer still need billions of steps to accomplish its own task. According to Jeff Hawkins, this is because the brain “retrieves the answers from memory” [3]. Therefore this thesis focuses on biology-inspired circuits such as cellular neural (nonlinear) network (CNN) [4, 5] to perform the task of associative memory. The locally connected structure in CNN, also observed in the human retina, reduces the wire connections in analog VLSI. From the mathematical point of view, CNN are coupled nolinear differential equations. Given initial conditions and a set of input image data, each trajectory must converge and propagate to some equilibrium point based on the coupled nolinear differential equations. To properly perform an image task, such as pattern recognition, CNNs are required to be 1.

(32) 2. CHAPTER 1. INTRODUCTION. completely stable. A circuit is completely stable if every trajectory tends to some equilibrium point [6]. Many equilibrium points in a CNN are possible. As Chua stated [7]: “Patterns are associated with complete stable circuits where each trajectory corresponding to a given initial condition must necessarily converge to some dc equilibrium point, usually among many others, as in cellular neural networks.” Therefore, understanding steady state values and the stability property is essential. However, previous studies in [8, 9, 10, 11, 12] do not guarantee that neuron states will converge to the equilibrium states. In [13, 14], the analysis of equilibrium points is mainly based on the standard activation function [4] and the space-invariant templates whose center element is larger than 1. Hence, this thesis seeks to find a domain of attraction for ARMCNN (Autonomous Ratio-Memory Cellular Nonlinear Network) solving the tasks of associative memory and pattern recognition.. Discrete time CNN is equivalent to Euler-integrated CNNs [15, 16]. A key question to discrete time implementation and computer simulation is numerical stability. Hence, from the digital emulation and the discrete implememtation view points, it is crucial to know the maximum time step in the numerical integration algorithm without causing numerical instability. In literature, locally regular (LR) CNN is well known to have a maximum forward Euler (FE) time step equal to the internal time constant of the CNN cell [17]. In addition, the equilibrim point is not changed after this time step. An interesting question arises as whether other CNNs have this property or not. Hence, this thesis aims to obtain the maximum time step by linking the ARMCNN dynamic equations into a diffusion model and using eigenspace decomposition for the ARMCNN functioned as associative memory. From the eigenspace decomposition, an analytic gain is derived to lower the amount of spurious memory points. Furthermore, the decomposition implied a sufficient condition to guarantee that the forward Euler ARMCNN has the same binary output as the continuous time ARMCNN. Although we do not derive the eigenspace decomposition for 2-D ARMCNN, this thesis suggests that a similar design principle also exists for 2-D ARMCNN. 1-D and 2-D examples are given to support the ARMCNN design concepts..

(33) 1.2. ORGANIZATION. 1.2. 3. Organization. The organization of the thesis is described as follows. Chapter 2 is the overview of the standard CNN. Relevant definitions are discussed. Chapter 3 describes the operation, algorithm and dynamic state space equations of ARMCNNs. Chapter 4 deals with the Lyapunov stability analysis and the conservative domain of attraction. From this analysis, the activation function and the ratio weight can vary without affecting the stability. The amount of the ratio weight variation is also discussed in this chapter. Chapter 5 analyzes the maximum time step in the simulation without causing the numerical instability or oscillation. The analysis is based on the eigenspace decomposition of the initial state vector. From this decomposition, the amount of spurious equilibrium points can be reduced by enhancing the low frequency component of the initial state vector. Furthermore, the decomposition implied a sufficient condition to guarantee that the forward Euler ARMCNN has the same binary output as the continuous time ARMCNN. Finally, conclusions and recommendations for future works are given in Chapter 6..

(34) 4. CHAPTER 1. INTRODUCTION.

(35) Chapter 2 Standard CNN. 2.1. Introduction. Cellular neural/nonlinear networks (CNNs) are nonlinear continuous time operating structures composed of locally connected cells in an array. Locally connected structures are also found in the silicon retina which has been modeled as a resistive grid by Mead [18]. Each CNN cell receives the input image simultaneously and processs the image signal in the continuous time domain. The processed image signal propagates away from the locally connected cell. The connected weight between each neighboring cell is govered by simple templates. Locally connected structures and the continouous time signal processing capability make CNNs suitable for analog VLSI implementation and image processing tasks. [19]. Among various image processing tasks, linear and nonlinear image filtering applications were demonstrated using CNNs as resistive grids in [20]. Many other CNN image processing examples such as connected component detector are available in [21, 22]. This chapter is organized as follows. Section 2.2 provides the mathematical definitions for the standard CNN. Image processing examples are illustrated by simple templates in Section 2.3. 5.

(36) 6. CHAPTER 2. STANDARD CNN. 2.2. Standard CNN State Equations. A standard CNN [4] consists of an M × N rectangular array with the cell or neuron C(i, j) located at the ith row and jth column. The dynamics of the standard CNN [4] is governed by X X dvxij vxij =− + A(i, j; k, l)vykl + B(i, j; k, l)vukl + z dt Rx C(k,l)∈N (i,j) C(k,l)∈N (i,j) Ra Ra X X dvxij = −vxij + Rx A(i, j; k, l)vykl + Rx B(i, j; k, l)vukl + Rx z Rx C dt C(k,l)∈N (i,j) C(k,l)∈N (i,j) C. Ra. Ra. τ = Rx C. (2.1). where vxij , vyij , vukl , z are the state, output, input, and threshold of cell C(i, j), respectively, and τ is the cell time constant. The units for vxij , vyij , vukl , z are Volt, Volt, Volt and Ampere, respectively. Rx A(i, j; k, l), Rx B(i, j; k, l) are unitless, and Rx z has the unit of Volt. The constraint conditions are |vxij (0)| ≤ 1, vuij ≤ 1. (2.2). where “black” is coded as +1, “white” is coded as −1 and the gray level is between +1 and −1. The image can be applied to the CNN in the form of the state vxij (0) or the input vuij of the cell C(i, j), or both. Several definitions regarding (2.1) are listed below: • Output function (also called activation fucntion) The only nonlinear element in Equation 2.1 is the output function, which is also called the activation function, and it has the form of vyij =. 1 (|vxij + 1| − |vxij − 1|). 2. (2.3). Figure 2.1 shows the piecewise-linear (PWL) characteristic in (2.3), and the range for vyij is −1 ≤ vyij ≤ 1. • Sphere of influence of cell C(i, j) The sphere of influence NRa (i, j) of cell C(i, j) is defined as NRa (i, j) = {C(k, l)|1 ≤ k ≤ M, 1 ≤ l ≤ N, max(|k − i|, |l − j|) ≤ Ra }. (2.4).

(37) 2.2. STANDARD CNN STATE EQUATIONS. 7. vyij. 1. 1. −1. vxij. −1. Figure 2.1: The PWL output or activation function. where Ra is a positive integer. For Ra = 1, C(i, j) is connected to its eight nearest neighbor cells C(k, l). • Space invariant A CNN is space invariant if synaptic operators A(i, j; k, l), B(i, j; k, l) and the threshold z do not vary with space. In this case, A(i, j; k, l) and B(i, j; k, l) are written as A(i, j; k, l) = A(k − i, l − j), B(i, j; k, l) = B(k − i, l − j). • Virtual cell Any cell C(k, l), with |k − i| ≤ Ra , |l − j| ≤ Ra , and k, l ∈ / {1, 2, . . . , M} is called a virtual cell, and the associated xkl , ykl are called virtual state and virtual output. Virtual values are assigned to these virtual cells. For example, Figure 2.2 shows a CNN consisting of a two-dimensional (2-D) M × N rectangulary array with M = 4, N = 4 and cells C(i, j), i = 1, 2, 3, 4, j = 1, 2, 3, 4. Total 16 cells are in this CNN array. Ra = 1 means only one layer of neighboring neurons is connected. For simplicity, only the virtual cells connceted to cell C(4, 1) are shown in the dotted line..

(38) 8. CHAPTER 2. STANDARD CNN. j C(1,1) 1. C(1,2) 2. C(1,3) 3. C(1,4) 4. C(2,1) 5. C(2,2) 6. C(2,3) 7. C(2,4) 8. C(3,0). C(3,1) 9. C(3,2) 10. C(3,3) 11. C(3,4) 12. C(4,0). C(4,1) 13. C(4,2) 14. C(4,3) 15. C(4,4) 16. C(5,0). C(5,1). C(5,2). i. Figure 2.2: A 2-D CNN consisting of a 4 × 4 rectangular array with 4 rows counted in the i direction, and 4 columns counted in the j direction. The radius of the sphere of influence is Ra = 1. Virtual cells connceted to cell C(4, 1) are shown in the dotted line.

(39) 2.3. CNN IMAGE PROCESSING EXAMPLES. 9. (2.1) implies that the input and the output of the activation fuction are voltage signals. Hence, A(i, j; k, l) and B(i, j; k, l) have the unit of Ω −1 . A voltage mode operation example uses several op amps to implement (2.1) in [4]. However, the voltage mode operation has been challenged recently in [23]. Therefore this thesis modifies the original CNN equation, and adopts the current mode approach as shown in chapter 3 for ARMCNN. Nevertheless, these two approaches have the same normalization form as shown in chapter 3.. 2.3. CNN Image Processing Examples. This section shows several CNN processing examples. Although CNN is locally connected, it can have several global image processing abilities. In this section, the test images use the bit map files with eight-bit resolution. The transformation between the read-in byte and the CNN state in (2.1) is detailed in Appendix A. These image processing examples include connected component detection (CCD) and global connectivity detection (GCD). These two tasks can be solved by locally regular (LR) CNNs [17], and can be numerically integrated with a step size τ [24, 17]. Many binary image tasks can be processed by LR-CNNs. Connected component detection (CCD) and global connectivity detection (GCD) are simulated in this chapter with a forward Euler step size of τ, where τ = Rx C. More CNN templates for various image tasks are available in [25].. 2.3.1. Connected Component Detection (CCD). CCD counts the number of contiguous blocks in the horizontal direction. Each contiguous block is denoted as a black pixel, and shifted to the right. The A,B and z templates for the CCD task are shown as, .  0 0 0      , BRx = 0 0 0 , zRx = 0, ARx =  1 2 −1     0 0 0 0 0 0 0 0. 0. . . (2.5).

(40) 10. CHAPTER 2. STANDARD CNN. Figure 2.3: Input image applied as vxij (0) before CCD operation. and the boundary condition for each vitual cell is the fixed type with ykl = 0. The test image state vxij (0) for CCD is shown in Figure 2.3. Total simulation time is Tsim = 200τ. Forward Euler integration method is applied with each time step equal to τ. The output image after CCD operation is shown in Figure 2.4. In the second row of Figure 2.3, two blacks are seperated by white pixels. So in the second row of Figure 2.4, two black pixels are found in the right-hand side.. 2.3.2. Global Connectivity Dectection (GCD). GCD finds the contiguous part. Two static input image patterns applied at the input vuij and the state vxij (0) are given. The initial state vxij (0) contains almost the same image as the input vuij except some black pixels in vxij (0) are modified into white ones. These modified pixels can be thought as the igniters, whereas the other unchanged ones are considered as ignitable objects [26]. Therefore, at the steady state, or the end of the fire, only the disconnected part remains. In other words, the global rule is that all connected black pixels of the modified pixles are changed to white or burned. The local rule is ˆ j), and that if an output black pixle C(i, j) have a neighboring output white pixel C(i, ˆ j) has a black input, then the original output black pixle C(i, j) turns into the pixel C(i,.

(41) 2.3. CNN IMAGE PROCESSING EXAMPLES. 11. Figure 2.4: Output image, vyij (200τ), after CCD operation. Tsim = 200τ. a output white pixel. For example, in Figure 2.6, the state at location (i, j) = (4, 3) is black. Consequently, the output pixle C(4, 3) is black. The neighboring cell state at (3, 3) is white. The output is therefore white at (3, 3). From Figure 2.5, the input is black at (3, 3). According to the local rule, the original output black pixle C(4, 3) turns into a output white pixel. It is like the original output black pixle C(i, j) = C(4, 3) gets a fire ˆ j) = C(3, 3). This is an interesting result because locally from the neighboring pixel C(i, connected CNNs can solve a global connectivity problem [27]. The A,B and z templates for the GCD task are shown as,     0 0.5 0 0 −0.5 0      , zRx = −4.5V,  , BRx = −0.5 ARx =  3 −0.5 0.5 3 0.5     0 −0.5 0 0 0.5 0. (2.6). and the boundary condition for each vitual cell is the fixed type, yij = −1, uij = −1. The input image for the GCD example is shown in Figure 2.5. Two black pixels are modified from Figure 2.5 at location (3,3) and (3,21) and stored as white state values shown in Figure 2.6. After 50τ, the output is shown in Figure 2.7. After 150τ, the output is shown in Figure 2.8..

(42) 12. CHAPTER 2. STANDARD CNN. Figure 2.5: The input image vuij for the GCD task.. Figure 2.6: The state vxij (0) for the GCD task. Compared with Figure 2.5, the state at (3, 3) and (3, 21) are white. The coordinates follow the definition in Figure 2.2..

(43) 2.3. CNN IMAGE PROCESSING EXAMPLES. Figure 2.7: The output image, vyij (50τ), after 50τ.. Figure 2.8: The output image, vyij (150τ), after 150τ.. 13.

(44) 14. CHAPTER 2. STANDARD CNN.

(45) Chapter 3 ARMCNN 3.1. Introduction. Information storage is called associative memory if it has the capability to recall a partial information [28]. The associative memory can retrieve a stored pattern from the input noisy messed-up pattern. Some researchers believe that our memories correspond to attractors in the brain’s huge phase space, since the human brain has more than 1012 neurons [29]. From this viewpoint, convergence to an appropriate attractor is called recognition. Hopfield proposed a model of a large fully connected network with symmetrical weights [30] that functions as an associative memory with the capability to recognize patterns. However, the limitation of Hopfield’s approach is that it requires fully connected symmetrical weights, which makes it difficult to connect wires in integrated circuits. Because of its connectivity, researchers consider the Cellular Nonlinear Network (CNN) [4] to be a potential architecture in future nano-electronic systems. To implement the associative memory by CNN, two key aspects of Hebb’s postulate, locality and cooperativity [31], are fully exploited in the Ratio Memory CNN (RMCNN) or AutonomousRMCNN (ARMCNN) [8, 9, 10, 11, 12]. Locality means that the change of synaptic efficacy only depends on local variables, which is the main point of CNN. Cooperativiy [31] means that the presynaptic and postsynaptic neurons must be active simultaneously for a synaptic weight to change, which is the learning rule in RMCNNs or ARMCNNs. This agrees with the correlation property in image data. Pixels located next to each other 15.

(46) 16. CHAPTER 3. ARMCNN. are more correlated than pixels that are in different regions of the image. The RM weight (or ratio weight) can be determined either through the elapsed time, as in [10, 11], or can be determined without the elapsed time, as in [9, 32]. While previous studies have proposed several different CNNs on associative memory [33, 34, 35, 36, 37], few have actually been implemented in analog VLSI [38], mainly due to complex mathematics required to make this possible. In literature, [33, 34] proposed a design method for the realization of associative memories through singular value decomposition technique (SVD). Another study [35, 36] showed the weights through solving linear matrix inequality (LMI) and generalized eigenvalue (GE) problems, whereas the work in [37] computed the CNN parameters by solving a set of linear equations via pseudoinversion techniques. In particular, the CNNs in [35, 36, 37, 33, 34] require mathematical operations such as SVD, LMI, and pseudoinversion. These operations are more complex than the simple methods proposed in this chapter and in [39], and more difficult to implement in analog VLSI [38]. ARMCNN has two periods of operation: the learning period and the recognition period. In the learning period, ratio weights are learned from the learning patterns. In the subsequent recognition period, the output pattern evolves from the noisy input pattern based on the learned ratio weights.. 3.2. ARMCNN Learning Period. This section discusses the ARMCNN consisting of an M × N rectangular array. Cell C(i, j) is located in the ith row and jth column. Two types of learning are distinguished in ARMCNN, learning with an elapsed time and learning without an elapsed time.. 3.2.1. Operation with Elapsed Time. In the learning period, assume that ARMCNN must learn m patterns. The learned weight zijkl from neuron C(k, l) to neuron C(i, j) can be determined by the Hebbian learning rule [31] zijkl =. m X p=1. p. p. uij ukl. (3.1).

(47) 3.2. ARMCNN LEARNING PERIOD. 17. p. where uij is the learning pixel image at the ith row and jth column of the pth pattern out p. of m input patterns, and ukl is the learning pixel image at the kth row and lth column in 0 0 the set of NRa (i, j). NRa (i, j) is the set of a Ra-neighborhood system without the neuron. C(i, j). The Ra-neighborhood system NRa (i, j) of the neuron C(i, j) is defined as the set of all neurons including C(i, j) and its neighboring neurons. NRa (i, j) = {C(k, l)|1 ≤ k ≤ M, 1 ≤ l ≤ N, max(|k − i|, |l − j|) ≤ Ra} where Ra is an integer called the radius of the sphere of influence. Besides, the correlation between two neighboring neurons at C(i, j) and C(k, l) is positively correlative if p. p. uij = ukl , forp = 1, . . . , m. (3.2). p. (3.3). or negatively correlative if p. uij = −ukl , forp = 1, . . . , m.. For example, for a ARMCNN without self-feedback, assume four learning patterns with +1 for a black pixel image and −1 for a white pixel image. All possible learned weights zijkl are {+4, +2, 0, −2, 0, −4}. The +4 occurs when the input image pair at C(k, l) and C(i, j) are positively correlative for all four input patterns. The −4 occurs when the input image pair at C(k, l) and C(i, j) are negatively correlative for all four input patterns. After learning all input patterns, the learned weight zijkl is transformed into the ratio weight wijkl [8, 9, 10, 11, 12] wijkl = P. zijkl 0 C(k,l)∈NRa (i,j). |zijkl |. .. 0 For a ARMCNN with self-feedback (SARMCNN), simply replace NRa (i, j) with NRa (i, j).. If k = i, l = j, then zijij is the self-feedback weight and wijij is the self-feedback ratio weight. The sum of the absolute values of the ratio weights for a SARMCNN is also equal to one. In the elapsed period, the inevitable leakage current [11, 40] associated with the stored weight reduces the absolute value of zijkl . However, this leakage enhances the ratio weights whose absolute values are larger than the average of the absolute values of wijkl connected to C(i, j) [12]. This feature enhancement effect was shown in [12, 41, 42, 43] as a result of storing weights as ratios..

(48) 18. CHAPTER 3. ARMCNN. 3.2.2. Operation without Elapsed Time. The learning algorithm for determining the ratio weights in ARMCNNs (or SARMCNNs), which requires no elapsed time, is summarized as follows: [32, 39] • Obtain the weights zijkl of (3.1) after m patterns are learned. • Set Gmax =. m X. p. p. |uij ||ukl |,. p=1. where. p uij. and. p ukl. are defined in (3.1).. • Compare |zijkl | with Gmax . The weight zijkl with its absolute value equal to Gmax will be kept. On the contrary, the weight with its absolute value less than Gmax will be set to zero. • Transform the connecting weights of neuron C(i, j) as wijkl =. Sgn(zijkl ) if |zijkl | = Gmax 0 (i, j) P NNRa. (3.4). 0 0 (ij) is the number of preserved weights in N where P NNRa Ra (i, j) with respect to. neuron C(i, j). wijkl = 0 if |zijkl | < Gmax . 0 For SARMCNNs, simply replace NRa (i, j) with NRa (i, j) and if k = i, l = j, then. zijij is the self-feedback weight. To simplify wire connections, only those neighboring neurons located on horizontal and vertical positions are connected, that is, up, down, left, right. Neighboring neurons located on diagonal sites are not connected. The algorithm without the elapsed time is used in this thesis. Therefore, all possible ratio weights are ±1/4, ±1/3, ±1/2, ±1 for a ARMCNN without self-feedback, whereas for a SARMCNN, all possible ratio weights are ±1/5, ±1/4, ±1/3, ±1/2, ±1. For example, after learning the three Chinese characters in Figure 3.1, the survived ratio weight connection is shown in Figure 3.2. There are one 2-D 18N subsystem (top two rows), one 2-D 25N subsystem (indicated with an arrow), six 1-D 2N subsystems, one 2-D 8N subsystem and two 1-D 9N systems (bottom two rows). The thick bond in.

(49) 3.3. ARMCNN RECOGNITION PERIOD. (a). 19. (b). (c). Figure 3.1: (a) Chinese character ONE. (b) Chinese character TWO. (c) Chinese character FOUR. Figure 3.2 is due to a negatively correlative input image at the two connected neurons and the thin bond is due to a positively correlative input image at the two connected neurons.. 3.3 3.3.1. ARMCNN Recognition Period ARCMNN State Equations. In the ARMCNN recognition period, the neuron or cell dynamic equation is X dxij (t) −xij (t) = + dt Rij Cij C(k,l)∈N. Ra. wijkl ykl (t) Cij (i,j). (3.5). ykl (t) = σ(xkl (t)) X |wijkl | = 1 C(k,l)∈NR (i,j). as shown in Figure 3.3, where ARMCNN variables, xij , yij , Isat are represented as electrical signals such as voltages and currents [32, 40] for hardware implementation. xij is the neuron state voltage, yij is the neuron output current, Isat is the output saturation current, g is the neuron gain, and Rij , Cij are the resistor and capacitor associated with the neuron C(i, j) respectively. wijkl is the ratio weight from neuron C(k, l) to neuron C(i, j) and is implemented by current mirrors in integrated circuits [32]. The activation function, σ(·) is actually a voltage to current converter (V/I). Activation functions of different types are possible. For example, the piecewise linear (PWL) activation function, having.

(50) 20. CHAPTER 3. ARMCNN. 2–D 25N. Figure 3.2: The survived ratio weights in the 9×9 ARMCNN after learning the three Chinese characters ONE, TWO, and FOUR of Figure 3.1.

(51) 3.3. ARMCNN RECOGNITION PERIOD. a transconductance value Gm of g/Rij , is

(52)

(53)

(54)

(55)

(56) Isat Rij

(57)

(58) Isat Rij

(59) g

(60)

(61)

(62)

(63) xkl + − xkl − σp (xkl ) = 2Rij

(64) g

(65)

(66) g

(67)

(68)

(69)

(70)

(71)

(72)

(73)

(74) I Gm

(75) I sat sat

(76) xkl +

(77) −

(78) xkl −

(79) =

(80)

(81)

(82) 2 Gm Gm

(83)

(84)

(85)

(86)

(87)

(88)

(89) xkl

(90) gIsat

(91) 1 1 x kl

(92)

(93) −

(94)

(95) = + −

(96) Isat Rij g

(97)

(98) Isat Rij g

(99) 2. 21. (3.6). as illustrated in Figure 3.4. The PWL type is the mostly commonly used one. For PWL activation function, a state xij is in the linear region if −Isat Rij Isat Rij gxij ≤ xij ≤ , yij = σp (xij ) = . g g Rij. (3.7). and a state xij is in the saturation region if Isat Rij , yij = Isat g Isat Rij , yij = −Isat . or xij < − g xij >. The input initial states (input image) are defined as −Isat Rij ≤ xij (0) ≤ Isat Rij .. (3.8). To perform the ARMCNN stability analysis in chapter 4, an alternative row-wise ordering of the state variables instead of (3.5) is as follows, x˙ = − Cx + T σ(x) y =σ(x). (3.9). where x ∈ Rn is the state vector, y is the output vector, and T = [Tij ] ∈ Rn×n is the connection matrix. C = diag[(R1 C1 )−1 , ..., (Rn Cn )−1 ], with Ri , Ci > 0 for i = 1, ..., n, and σ(x) = [σ(x1 ), ..., σ(xn )]T represents the activation function. The input image, that is, the initial state, satisfies |xi (0)| ≤ Isat Ri for i = 1, . . . , n. n = M × N for a M × N pattern. For the ratio memory requirement, n X k=1. |Tik | =. 1 , i = 1, ..., n Ci. (3.10).

(100) 22. CHAPTER 3. ARMCNN. ratio weight. wijij. yij. xij Rij. Cij. Activation Function. cell or neuron model wijkl. ratio weight. ratio weight. ratio weight. ykl. Figure 3.3: The neuron model of ARMCNN located at the ith row and jth column. If without self-feedback, wijij = 0..

(101) 3.3. ARMCNN RECOGNITION PERIOD. 23. yij. Isat −Isat Rij g. xij. Isat Rij g. −Isat. Figure 3.4: The transfer characteristic of the PWL activation function, σp (·). where Tik is the ratio weight from neuron k to neuron i multipled by 1/Ci . Many other ordering methods for the neurons are possible such as diagonal ordering and column-wise ordeing [44]. The row-wise odering in this chapter represents the two dimensional indices of Cab and Rab by one dimensional Ci and Ri using the following rule: i = (a − 1) × N + b. For example, row-wise ordering of state variables of a 4 × 4 array is shown in Figure 2.2. As seen, neuron (2, 1) in Figure 2.2 is ordered as 5, and the corresponding T is 16 × 16.. 3.3.2. Equilibrium Point. The equilibrium points of (3.9) can be obtained by solving Cxeq = T σ(xeq ), xeq ∈ Rn .. (3.11). q. In general, we can have q equilibrium points, x1eq , . . . , xeq . The resultant ratio weights generate many small subsystems. Each subsystem has two binary equilibrium points in the form of (±Isat R, . . . , ±Isat R). The choice of the sign is determined as follows. Set the.

(102) 24. CHAPTER 3. ARMCNN. x1. x2. x3. x4. x5. Figure 3.5: A 1-D five-neuron (5N) system having two binary equilibrium points (x1 (∞), x2 (∞), x3 (∞), x4 (∞), x5 (∞)) as ±(Isat R, Isat R, Isat R, Isat R, Isat R). equilibrium state of the first neuron C(i, j) to Isat R (−Isat R). If a negatively correlative input image is found between C(k, l) and C(i, j), then set the equilibrium state of the connected neuron C(k, l) to −Isat R (Isat R). Otherwise, if positively correlative, then set the equilibrium state of the connected neuron C(k, l) to Isat R (−Isat R). This is true for any number of neurons [8]. Further, the ratio memory requirement shown in (3.10) confines the ranges of all possible equilibrium states as: |xmul eq,i | ≤ Isat Ri for i = 1, ..., n,. mul = 1, ..., q. where xmul eq,i is the ith equilibrium neuron state for the multh equilibrium point. This is because, for the ith neuron (i = 1, . . . , n) n n X Isat 1 mul X mul xeq,i = Tik σ(xeq,k ) ≤ Isat |Tik | = R i Ci Ci k=1 k=1. ⇒ |xmul eq,i | ≤Isat Ri. for. mul = 1, . . . , q.. Because q is possible to be greater than 2, some undesired equilibrium points, that is, spurious memory points, may exist besides the two binary equilibrium points. To show spurious memory points, consider a 1-D 5N system shown in Figure 3.5. The neuron resistor (Ri = R, i = 1, 2, . . . , 5), capacitor (Ci = C, i = 1, 2, . . . , 5) and voltage to current converter (V/I) are as defined in Figure 3.3 and Figure 3.4. For the 1-D 5N system in Figure 3.5 under Ri = 1, Ci = 1, i = 1, 2, . . . , 5, we see that x˙ 1 = − x1 + σ(x2 ) x˙ 2 = − x2 + 0.5σ(x1 ) + 0.5σ(x3 ) x˙ 3 = − x3 + 0.5σ(x2 ) + 0.5σ(x4 ) x˙ 4 = − x4 + 0.5σ(x3 ) + 0.5σ(x5 ) x˙ 5 = − x5 + σ(x4 ).. (3.12).

(103) 3.3. ARMCNN RECOGNITION PERIOD. 25. The thin bond is due to a positively correlative input learning image pair at the two connected neurons. For the PWL activation function with g > 1, (1, 1, 1, 1, 1) and (−1, −1, −1, −1, −1) are two equilibrium points as can be verified by substituting these equilibrium points into (3.12). If g = 2, then one possible spurious memory point is at x1eq = [0.877, 0.4389, −0.0611, −0.5611, −1]T . Another possible spurious memory point is at x2eq = [1, 0.5, 0, −0.5, −1]T . In chapter 5, these two spurious memory points are eliminated by reducing the neuron gain g. Notice, in this example, |xmul eq,i | ≤ 1 for mul = 1, 2 and i = 1, 2, . . . , 5.. 3.3.3. Relationship to Standard CNN. (3.5) is suitable for the current mode approach with the wijkl implemented using replicas of current mirrors. In addition, if xij is normalized with respect to Isat Rij and yij is normalized with respect to Isat , then (3.5) is expressed as, X xij d + wijkl σp (xkl ) Cij xij = − dt Rij C(k,l)∈N (i,j) R X xij d gIsat xkl (t) 1 xkl (t) 1 ⇒ Cij xij = − + wijkl (| + |−| − |) dt Rij C(k,l)∈N (i,j) 2 Isat Rij g Isat Rij g R X x (t) x (t) d xkl (t) 1 1 g xkl (t) ij ij ⇒τ ( (| )=− + + |−| − |) wijkl dt Isat Rij Isat Rij C(k,l)∈N (i,j) 2 Isat Rij g Isat Rij g R. ⇒τ. dx˜ ij (t) = −x˜ ij (t) + dt C(k,l)∈N. X. wijkl σ˜ p (x˜ kl ). (3.13). R (i,j). where xij (t) , τ = Rij Cij Isat Rij σp (xkl ) g 1 1 σ˜ p (x˜ kl ) = (|x˜ kl + | − |x˜ kl − |) = 2 g g Isat x˜ ij (t) =. −1 ≤x˜ ij (0) ≤ 1.. (3.14). σp (xkl ) is defined in (3.6). Obviously, x˜ ij and wijkl play the same role as the vxij and Rx A(i, j; k, l) of (2.1) do respectively. The constraint condition on the initial state is also equivalent as seen from (2.2) and (3.14). Hence, it is convinent to use the normalized variables as. xij Isat Rij. and. σp (xij ) Isat. in the simulation. Table 3.1 compares the equivalence between.

(104) 26. CHAPTER 3. ARMCNN. Table 3.1: Equivalent representations between the N-ARMCNN and the S-CNN N-ARMCNN S-CNN xij (t) state vxij (t) Isat Rij output weight. yij (t) Isat. wij. vyij (t) Rx A(i, j; k, l). the normalized ARMCNN (N-ARMCNN) in (3.13) and the standard CNN (S-CNN) in (2.1)..

(105) Chapter 4 Lyapunov Stability Analysis 4.1. Introduction. For pattern recognition, stability analysis is important to understand the behavior near the steady state values. Whether or not initial states can successfully converge to the desired equilibrium point determines recognition performance. However, previous studies in [8, 9, 10, 11, 12] do not guarantee that neuron states will converge to the equilibrium states. Usually, obtaining the range in which the state converges or obtainging the convergence rate is more meaningful than knowing the stability type of the equilibrium point. Hence, this chapter aims to exploit a Lyapunov theory to obtain the domain of attraction (DOA).. 4.2. Stability Definitions. For linear systems, there is no ambiguous definition about stablity. The eigenvalues of the linear system are required to be negative to ensure that the trajectory asymptotically conveges to the equilibrium point. However, when the system is nonlinear, various definitions are available such as stability, unform stability, exponential stability [45]. 27.

(106) 28. CHAPTER 4. LYAPUNOV STABILITY ANALYSIS. We consider a nonlinear autonomous system       f1 (x) x1 x˙ 1       f (x) x  x˙   2   2  2 x˙ = f (x), where f : Rn → Rn , f (x) =  .  , x =  .  , x˙ =  .   ..   ..   ..        fn (x) xn x˙ n. (4.1). where xi is the state variable, and x˙ i denotes the derivative of xi with respect to the time variable t. Because f does not depend explicitly on time, the system defined by (4.1) is called autonomous. A point or state vector xe ∈ Rn is an equilibrium point or an equilibrium state vector of the system if f (xe ) = 0. Several relevant definitions of stability in this thesis are as follows [46]. The equilibrium point xe is • stable (stability in the sense of Lyapunov) if for each ε, there is δ = δ(ε) > 0, such that kx(0) − xe k ≤ δ ⇒ kx(t) − xe k < ε, ∀t ≥ 0 • unstable if it is not stable • locally asymptotically stable (LAS) at xe if it is stable and an δ > 0 exists such that kx(0) − xe k ≤ δ ⇒ x(t) → xe as t → ∞ The above definition of stability implies that the trajectory can be kept arbitrarily close to the equilibrium point by starting sufficiently close to the equilibrium point. This kind of stability is also called stability in the sense of Lyapunov. For locally asymptotically stability, we need two conditions to hold. The equilibrium point is stable and the trajectories approach to the equilibrium point. When the equilibrium poin is locally asymptotically stable, the next interesting question is how far from the equilibrium point the initial state vector can be and the trajectory from that inital state vector still converges to the same equilibrium point. A domain of attraction of the equilibrium point is the set such that if the initial state vector is located within this set, then the trajectory converges to the equilibrium point. The proof process of the stability about the equilibrium point usually implies a conservative DOA as in Section 4.4..

(107) 4.3. LYAPUNOV THEORY. 4.3. 29. Lyapunov Theory. Without actually solving (4.1), Lyapunov theory can be used to decide the stability of the equilibrium point. For (4.1), a typical Lyapunov theorem has the following expression [47], • if there exists a function V : Rn → R that satisfies some condition on V and V˙ , where n. n. X ∂V dV (x) X ∂V V˙ (x) = = x˙ j = fj (x) = ∇V (x)T f (x) dt ∂x ∂x j j j=1 j=1. (4.2). • then conclusions can be made about the trajectories of (4.1). Such a function V is called a Lyapunov function, and V˙ (x) is a rate of decrease or increase of V (x) along the trajectory.. 4.3.1. A Lyapunov Local Asymptotic Theorem. Suppose x = 0 is an equilibrium point for (4.1). If a function V satisfies the following, • V (0) = 0, V (x) > 0 in D − 0 • V˙ (0) = 0, V˙ (x) < 0 in D − 0, and if Ωc = {x ∈ Rn |V (x) ≤ c} is bounded and contained in D, then every point starting in Ωc remains in Ωc and approaches the origin. Hence, Ωc is invariant and is a conservative domain of attraction (DOA) [48]. For example, Figure 4.1 shows an ellipsoid surface to represent V (x) = xT P x = c, where c is a constant. D is assumed to be a sphere as kxk ≤ r. To maximize the DOA, c is chosen as, c = min V (x) = min xT P x. kxk=r. kxk=r.

(108) 30. CHAPTER 4. LYAPUNOV STABILITY ANALYSIS. D xT P x=c r 0. Figure 4.1: Ωc = {x ∈ Rn |V (x) = xT P x ≤ c} is bounded and contained in D.. 4.4. ARMCNN Stability Analysis. Without loss of generality, Ri , Ci and Isat in (3.9) are normalized to unities. This is because these parameters only affect the convergent time which is proportional to 1/(Ri Ci ), and the state values and output currents can be normalized as xi /(Isat Ri ) and yi /Isat . The normalizations of state values and output currents are shown in (3.13). The normalized binary equilibrium point is expressed as: Eqp = eq1 eq2 · · · eqn where eqi is in the form of ±1. Consider the behavior around one normalized binary equilibrium point using new state variable zi . The new state variable zi with respect to the normalized binary equilibrium point is defined as zi = xi − eqi. for i = 1, ..., n.. (4.3). For instance, if the normalized equilibrium state in the ith neuron is ±1, then zi = xi ∓ 1. After this transformation from x to z, the new system has an equilibrium point at the origin and is represented as z˙ = −z + g(z). (4.4).

(109) 4.4. ARMCNN STABILITY ANALYSIS. 31. σ(zi+1) |σ(zi+1)−1| zi=−1. | zi |. 1. zi. | zi | σ(zi+1). Figure 4.2: The shifted PWL activation function, σ(zi + 1). where g(z) = [g1 (z) · · · gn (z)]T " n # X gi (z) = Tik σ(zk + eqk ) − eqi .. (4.5). k=1. Figure 4.2 shows the shifted PWL activation function of σ(zi + 1), and Figure 4.3 shows the shifted PWL activation function of σ(zi − 1). For the PWL activation, (4.6) and (4.7) must be satisfied in the following stability analysis. |σ(zi + 1) − 1| ≤ |zi | for |zi | < 1, σ(1) = 1 and. |σ(zi +1)−1| |zi | zi →0. lim. = 0 for i = 1, . . . , n. |σ(zi − 1) + 1| ≤ |zi | for |zi | < 1, σ(−1) = −1 and. |σ(zi −1)+1| |zi | zi →0. lim. = 0 for i = 1, . . . , n.. (4.6). (4.7). Theorem 1 For the PWL activation function defined in Figure 3.4, (4.6) and (4.7) are satisfied if and only if the slope g is greater than 1. Proof: From Figure 3.4, it is obvious that σ(1/g) = 1 for the normalized ARMCNN. The fact that g must be greater than 1 can be argued from the following analysis..

(110) 32. CHAPTER 4. LYAPUNOV STABILITY ANALYSIS. σ(zi−1) | zi | 1. | zi |. σ(zi−1) zi zi=1 |σ(zi−1)+1|. Figure 4.3: The shifted PWL activation function, σ(zi − 1). 1. If g < 1, then 1/g > 1. The PWL activation function in Figure 3.4 is monotonically increasing from x = 0 to x = 1/g >1. Therefore, σ(0) = 0 < σ(1) < σ(1/g) = 1 which does not meet (4.6), that is, σ(1) = 1. 2. If g = 1, then |σ(zi + 1) − 1| = |zi + 1 − 1| = |zi | for. − 1 ≤ zi ≤ 0. therefore, lim−. zi →0. |σ(zi + 1) − 1| |zi | = lim− = 1, for i = 1, . . . , n, zi →0 |zi | |zi |. which does not meet (4.6), that is, |σ(zi + 1) − 1| = 0, for i = 1, ..., n. zi →0 |zi | lim. 3. If g > 1, then |zi | < 1 is divided into the following two ranges to show the PWL activation function defined in Figure 3.4 satisfies (4.6)..

(111) 4.4. ARMCNN STABILITY ANALYSIS. 33. Range 1: if −1 < zi < −1 + 1/g < 0, then −g < gzi < −g + 1 |σ(zi + 1) − 1| = |g × (zi + 1) − 1| = 1 − gzi − g > 0 |zi | = −zi |zi | − |σ(zi + 1) − 1| = −zi − 1 + gzi + g = (g − 1) + zi (g − 1) = (g − 1)(zi + 1) > 0 ⇒ |σ(zi + 1) − 1| < |zi | for −1 < zi < −1 + 1/g < 0. Range 2: if −1 + 1/g ≤ zi < 1, then ⇒ 1/g ≤ zi + 1 < 2 ⇒ σ(zi + 1) = 1 ⇒ |σ(zi + 1) − 1| = 0 ≤ |zi | 0 |σ(zi + 1) − 1| = lim =0 zi →0 |zi | zi →0 |zi |. ⇒ lim. In a similar way, |zi | < 1 is divided into the following two ranges to show that the PWL activation function defined in Figure 3.4 satisfies (4.7). Range 3: if 0 < 1 − 1/g < zi < 1, then g − 1 < gzi < g |σ(zi − 1) + 1| = |g × (zi − 1) + 1| = gzi − g + 1 > 0 |zi | = zi |zi | − |σ(zi − 1) + 1| = zi − 1 − gzi + g = zi (1 − g) + (g − 1) = (g − 1)(1 − zi ) > 0 ⇒ |σ(zi − 1) + 1| < |zi | for 0 < 1 − 1/g < zi < 1..

(112) 34. CHAPTER 4. LYAPUNOV STABILITY ANALYSIS. Range 4: if −1 < zi ≤ 1 − 1/g, then ⇒ −2 < zi − 1 ≤ −1/g ⇒ σ(zi − 1) = −1 ⇒ |σ(zi − 1) + 1| = 0 ≤ |zi | 0 |σ(zi − 1) + 1| = lim =0 zi →0 |zi | zi →0 |zi |. ⇒ lim. For the PWL activation function, the above case (3) shows that if g > 1 then (4.6) and (4.7) are satisfied. And the above case (1) and case (2) show that for the PWL activation function, if (4.6) and (4.7) are satisfied, then g < 1 and g = 1 are not possible. Therefore, for the PWL activation function defined in Figure 3.4, (4.6) and (4.7) are satisfied if and only if the slope g is greater than 1. The ARMCNN defined in (3.9) is generated after performing the algorithm in Section 3.2 without elapsed time. The ratio weights of this ARMCNN satisfy (3.10), and the activation functions of each neuron are required to satisfy (4.6) and (4.7). Theorem 2 shows that the resultant ARMCNN converges to one of the binary equilibrium points.. Theorem 2 ARMCNN stability analysis with a conservative Domain of Attraction (DOA). For the normalized ARMCNN (Ri = R = 1, Ci = C = 1, Isat = 1) defined in (4.4) with the activation function satisfying (4.6) and (4.7), there exists a conservative domain of attraction (DOA), i.e., kzk < r, so that the ARMCNN will converge to one of the normalized binary equilibrium points for kzk < r.. Proof: We need to first prove the asymptotic stability of (4.4), that is, z˙ = −z + g(z). Define a positive quadratic Lyapunov function V (z) = zT P z..

(113) 4.4. ARMCNN STABILITY ANALYSIS. 35. Choosing P = 0.51n , where 1n is the n × n identity matrix, we have from (4.2), V˙ (z) = ∇V (z)T (−z + g(z)) = 2zT P (−z + g(z)) = −2zT P z + 2zT P g(z) = −zT z + zT g(z) = −kzk2 + zT g(z) ≤ −kzk2 + kzkkg(z)k. It is obvious that kg(z)k determines the sign of the derivative of V (z). To find kg(z)k, the range of gi (z) is analyzed first from (4.5) as " n # X gi (z) = Tik σ(zk + eqk ) − eqi k=1. =. n X. Tik [σ(zk + eqk ) − eqk ]. k=1. =. n X. (4.8). Tik Ik. k=1. where Ik ≡ σ(zk + eqk ) − eqk . The reason for the validity of (4.8) is as follows. Denoting these two opposite equilibrium states as S 1 and S −1 : the index set of all neurons with the normalized equilibrium states equal to 1 is S 1 and the index set of all neurons with the normalized equilibrium states equal to −1 is S −1 , then n X. Tik [σ(zk + eqk ) − eqk ]. k=1. =. X. Tik [σ(zk + 1) − 1] +. k∈S 1. =. X. Tik [σ(zk − 1) + 1]. k∈S −1. X. Tik σ(zk + 1) −. k∈S 1. +. X. X k∈S −1. Tik. k∈S 1. Tik σ(zk − 1) +. X. Tik .. k∈S −1. If the equilibrium state in the ith neuron is 1, then Tik > 0, for k ∈ S 1 (positivelycorrelative) Tik < 0, for k ∈ S −1 (negativelycorrelative). (4.9).

(114) 36. CHAPTER 4. LYAPUNOV STABILITY ANALYSIS. and from (3.10), we have X. X. −Tik +. k∈S 1. Tik = −1,. k∈S −1. so (4.9) is expressed as n X. Tik [σ(zk + eqk ) − eqk ]. k=1. X. =. X. Tik σ(zk + 1) +. k∈S 1. Tik σ(zk − 1) − 1. k∈S −1. which is (4.5). If the equilibrium state in the ith neuron is −1, then Tik < 0, for k ∈ S 1 (positively correlative) Tik > 0, for k ∈ S −1 (negatively correlative) and from (3.10), we have X k∈S 1. X. −Tik +. Tik = 1,. k∈S −1. so (4.9) is expressed as n X. Tik [σ(zk + eqk ) − eqk ]. k=1. =. X k∈S 1. X. Tik σ(zk + 1) +. Tik σ(zk − 1) + 1. k∈S −1. which is (4.5). Therefore, (4.8) is proved. From (4.8) and equations of (4.6) and (4.7), we have gi (z) =. n X k=1. |Ik | = 0 for k = 1, . . . , n. zk →0 |zk |. Tik Ik , with lim. (4.10). This implies that for any ε (0 < ε < 1), there exists an rk , such that |Ik | < ε|zk | for |zk | < rk ,. k = 1, ..., n.. In fact rk can be found from Figure 4.2 and Figure 4.3 given the ε value. Let r = min (r1 , r2 , ..., rn ). (4.11).

(115) 4.4. ARMCNN STABILITY ANALYSIS. 37. then if |zk | < r. |Ik | < ε|zk | for k = 1, ..., n.. ⇒. Or, if kzk < r ⇒ |zk | < r ⇒ |Ik | < ε|zk | for k = 1, ..., n. From (3.10), we can see that n X. |Tik zk | ≤. q |z1 |2 + |z2 |2 + . . . + |zn |2 = kzk.. (4.12). k=1. From (4.10), (4.11) and (4.12) and let Ii∗. ≡. n X. |Tik Ik |,. k=1. the upper bound of the norm of g(z) can be found as p kg(z)k = g1 (z)2 + · · · + gn (z)2 q ≤ (I1∗ )2 + · · · + (In∗ )2 v u n n X uX = t( |T1k Ik |)2 + · · · + ( |Tnk Ik |)2 k=1. k=1. v u n n X uX t 2 < ( |T1k εzk |) + · · · + ( |Tnk εzk |)2 k=1. k=1. v u n n X u X t 2 2 2 = ε ( |T1k zk |) + · · · + ε ( |Tnk zk |)2 k=1. ≤. p. k=1. √ ε2 kzk2 n = ε nkzk = γkzk. for kzk < r. (4.13). √ where γ ≡ ε n. Using (4.13), we have V˙ (z) ≤ −kzk2 + kzkkg(z)k √ ≤ (ε n − 1)kzk2 = (γ − 1)kzk2 for kzk < r. If γ < 1, then V˙ (z) < 0, which yields an asymptotic stable ARMCNN defined in (4.4). Since √ γ ≡ ε n,.

(116) 38. CHAPTER 4. LYAPUNOV STABILITY ANALYSIS. we have √ 1 0<γ=ε n<1⇒ε< √ . n. (4.14). Therefore, by choosing a proper ε from the above (4.14), we can find r from (4.11) such that kzk < r ⇒ V˙ (z) ≤ (γ − 1)kzk2 kzk < r, kzk = 6 0 ⇒ V˙ (z) < 0 kzk = 0 ⇒ V˙ (z) = 0 Finally, apply the Lyapunov local asymptotic theorem in Section 4.3.1, we have V (z) = zT P z = 0.5kzk2 < 0.5r2 ⇒ kzk < r ⇒ V˙ (z) < 0, V˙ (0) = 0. (4.15). that is, kzk < r is the conservative DOA. The above derivation is applicable to any activation function satisfying (4.6) and (4.7) that includes the PWL function of gain greater than unity, as Theorem 1 shows. Furthermore, only the unity sum of the absolute values of the ratio weights is required. The ratio weights can vary a lot as long as the signs of the corresponding ratio weights in the above proof are kept. Therefore, the ARMCNN can converge to the correct equilibrium states even when there are deviations in the ratio weights due to different VLSI processes.. 4.5. Examples. Example 1. Ratio weight variations do not change the equilibrium points. For the 1-D 3N system in (4.16) of Figure 4.4 under Ri = 1, Ci = 1, Isat = 1, i = 1, 2, 3, we have x˙ 1 = − x1 + σ(x2 ) x˙ 2 = − x2 + 0.5σ(x1 ) − 0.5σ(x3 ) x˙ 3 = − x3 − σ(x2 ). (4.16). where the activaion function σ(·) satisfies (4.6) and (4.7). Hence, σ(1) = 1, σ(−1) = −1. The thick bond in Figure 4.4 is due to a negatively correlative input image at the two.

(117) 4.5. EXAMPLES. 39. x1. x2. x3. Figure 4.4: A 1-D three-neuron (3N) subsystem with two binary equilibrium points of black-black-white and white-white-black, or (x1 (∞), x2 (∞), x3 (∞)) = ±(Isat R, Isat R, −Isat R). connected neurons and the thin bond is due to a positively correlative input image at the two connected neurons. (1, 1, −1) is an equilibrium point as can be verified by substituting (1, 1, −1) into (4.16). If the process variation perturbs the system into x˙ 1 = −x1 + σ(x2 ) x˙ 2 = −x2 + 0.7σ(x1 ) − 0.3σ(x3 ) x˙ 3 = −x3 − σ(x2 ),. (4.17). then after substitution of the original equilibrium point of (1, 1, −1) into the right hand side of (4.17), we have −1 + 1 = 0 −1 + 0.7 − 0.3(−1) = 0 −(−1) − 1 = 0. Therefore (1, 1, −1) is still an equilibrium point. Example 2. Finding a DOA for a three-neuron system in Figure 4.4. The new state variable zi with respect to the normalized equilibrium point (1, 1, −1) is defined as z1 =x1 − 1 z2 =x2 − 1 z3 =x3 + 1. After this transformation from x to z, the new system exhibits equilibrium at the origin and is described as z˙ = −z + g(z). (4.18).

(118) 40. CHAPTER 4. LYAPUNOV STABILITY ANALYSIS. σ(z1 +1) 0.42. 1 0.79. 1. z1. 0.42 1. Figure 4.5: The shifted sinusoidal activation function of σ(z1 + 1). The sinusoidal activation function is defined in (4.19).. where. g(z) = [g1 (z). g2 (z). g3 (z)]T. g1 (z) = σ(z2 + 1) − 1 = I2 g2 (z) = 0.5σ(z1 + 1) − 0.5σ(z3 − 1) − 1 = 0.5 (σ(z1 + 1) − 1) − 0.5(σ(z3 − 1) + 1) = 0.5I1 − 0.5I3 g3 (z) = −(σ(z2 + 1) − 1) = −I2 .. Assume that the activation function is the sinusoidal type defined as π σ(zi ) = sin( zi ) if |zi | ≤ 1 2 σ(zi ) = 1 if zi ≥ 1 σ(zi ) = −1 if zi ≤ −1.. (4.19). Also assume that each neuron has the same activation function of (4.19). Figure 4.5 shows the shifted sinusoidal activation function of σ(z1 + 1). (4.19) satisfies the activation function requirement in (4.6) and (4.7). To show this, consider first 0 < z1 < 1. It follows.

(119) 4.5. EXAMPLES. 41. that, ⇒z1 + 1 > 1 ⇒σ(z1 + 1) = 1 ⇒|σ(z1 + 1) − 1| = |I1 | = 0 ≤ |z1 | ⇒ lim+ z1 →0. 0 |σ(z1 + 1) − 1| = lim+ = 0. z1 →0 |z1 | |z1 |. (4.20) (4.21). Next consider −1 < z1 ≤ 0. It follows that, z1 + 1 1 π ≤ ⇒σ(z1 ) = sin( z1 )and0 < 2 2 2 z1 + 1 ⇒0 < σ(z1 + 1) = sin(π ) ≤ 1. 2 The sinc function of sinc(z) ≡. sin πz πz. is monotonically decreasing from z = 0 to z = 1. [49]. So we have the following, sin(π z12+1 ) 1 z1 + 1 ⇒sinc( ) ≤ = sinc( ) < sinc(0) z +1 1 2 2 π 2 z +1 2 sin(π 12 ) ≤ <1 π π z12+1 z1 + 1 ) = σ(z1 + 1) ⇒z1 + 1 ≤ sin(π 2 ⇒z1 ≤ σ(z1 + 1) − 1 = I1. ⇒. ⇒|z1 | = −z1 ≥ 1 − σ(z1 + 1) = |σ(z1 + 1) − 1| ≥ 0. (4.22). Hence the limit as z1 approaches from 0− is |σ(z1 + 1) − 1| 1 − σ(z1 + 1) = lim− z →0 |z1 | −z1 1 z1 +1 1 − sin(π 2 ) −(π/2) cos(π z12+1 ) = lim− = |z1 =0 = 0. z1 →0 −z1 −1 lim. z1 →0−. (4.23). (4.23) is from L’Hospital’s rule. From (4.20), (4.21), (4.22) and (4.23), we have shown that (4.19) satisfies the activation function requirement in (4.6). In a similar way, (4.19) can be shown to satisfy (4.7). From Theorem 2, the finding of the conservative DOA is to find the r such that if |zk | < r ⇒ |Ik | < ε|zk | for k = 1, 2, 3. (4.24).

(120) 42. CHAPTER 4. LYAPUNOV STABILITY ANALYSIS. where p ε < 1/ 3 = 0.577 for a three-neuron system. Because of the odd symmetry property of (4.19), it suffices to use z1 and I1 to find the r. Figure 4.5 illustrates this process in finding the r such that if |z1 | < r, then |I1 | ≤ 0.5|z1 | < 0.577|z1 |, which can be solved iteratively in the following: 1. Try r = 0.5, we have π sin( (−0.5 + 1)) = 0.707, z1 = −0.5 2 |I1 /z1 | = (1 − 0.707)/(0.5) = 0.586 > 0.5. 2. Then select r = 0.45, we have π sin( (−0.45 + 1)) = 0.76, z1 = −0.45 2 |I1 /z1 | = (1 − 0.76)/(0.45) = 0.53 > 0.5. 3. Finally, the following converged solution is π sin( (−0.42 + 1)) = 0.79, z1 = −0.42 2 |I1 /z1 | = (1 − 0.79)/(0.42) = 0.5. (4.19) has the property that

(121)

(122)

(123)

(124)

(125) I1

(126)

(127) I1

(128)

(129)

(130)

(131)

(132) < , for − 1 < z1b < z1a < 0,

(133) z1

(134)

(135) z1

(136) z1 =z1a z1 =z1b or

(137)

(138)

(139) d

(140)

(141) I1

(142) = d

(143) dz1 z1

(144) dz1. sin(π z12+1 ) − 1 z1. ! < 0, for − 1 < z1 < 0.. So it follows that kzk < r = 0.42 is a conservative DOA for this three-neuron system. The trajectories at three different initial points are shown in Figure 4.6. These three initial points x(0) are (0, 1, 0), (0.5, 0, −1), (1.2, 0.9, −1.5). The projection onto the plane x3 = −1 is also shown. Clearly, these three points converge to the equilibrium point of (1, 1, −1), even these three initial points have the initial norm kz(0)k > 0.42. This implies that kzk < r = 0.42 is a conservative DOA for this three-neuron system..

(145) 4.5. EXAMPLES. 43. (0,1,0). 0.5 0. -0.5. -0.5 x3. 0. -1-0.5 -1.5. (0.5,0,-1) 0. 0.5 x2. 0.5 1. 1.5 (1.2,0.9,-1.5). 1. x1. 2 1.5. Figure 4.6: The trajectories in thick lines at three different initial states converge to the (1, 1, −1). The projection onto the plane x3 = −1 is shown in the thin dotted lines. Figure 4.7 shows that the simulated norm of the trajectory kz(t)k satisfies kz(t)k ≤ e−0.2t kz(0)k, where q p kz(t)k = z21 + z22 + z23 = (x1 − 1)2 + (x1 − 1)2 + (x1 + 1)2 ..