細胞神經網絡的空間複雜度

全文

(1)國立交通大學應用數學系博士論文細胞神經網絡的空間複雜度 Spatial Complexity in Some Class of Cellular Neural Networks. 研究生: 張志鴻指導教授: 林松山教授. 中華民國九十七年六月.

(2) 細胞神經網絡的空間複雜度. Spatial Complexity in Some Class of Cellular Neural Networks 研究生: 張志鴻. Student:Chih-Hung Chang. 指導教授: 林松山. Advisor:Song-Sun Lin. 國立交通大學應用數學系博士論文. A Thesis Submitted to Department of Applied Mathematics College of Science National Chiao Tung University in partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Applied Mathematics June 2008 Hsinchu, Taiwan, Republic of China. 中華民國九十七年六月.

(3) Dedicated to the Memory of Grandpa, 1923 - 2008.

(4) 細胞神經網絡的空間複雜度研究生: 張志鴻. 指導教授: 林松山教授. 國立交通大學應用數學系摘要本文旨在研究一維的多層細胞神經網絡所產生的花樣其空間複雜度以及二維非均勻空間細胞神經網路其拓樸熵的稠密性。在多層細胞神經網絡部分, 如果我們在輸出空間給予適當的符號, 輸出空間會等價於一個 sofic 移動空間。我們給出兩個物理上的不變量, 拓樸熵和動態 ζ-函數的公式。這同時給了抽象的 sofic 移動空間一個實際上可以應用的例子。除此之外, 我們發現了當考慮多層細胞神經網絡時, 拓樸熵的對稱性會被破壞掉。這也再一次證明多層細胞神經網絡和單層細胞神經網絡是兩個性質極端不同的系統。更進一步地, 當我們考慮非均勻空間的細胞神經網絡系統, 其拓樸熵會稠密的分佈在 [0, log 2] 這個封閉區間之中。換句話說, 非均勻空間的細胞神經網絡系統有著極為豐富的物理現象蘊含在裡面。. i.

(5) Spatial Complexity in Some Class of Cellular Neural Networks Student:Chih-Hung Chang. Advisor:Song-Sun Lin. Department of Applied Mathematics National Chiao Tung University Abstract This dissertation consists two parts. The first part investigates the complexity of the global set of output patterns for one-dimensional multi-layer cellular neural networks with input; the second part focus on the dense entropy of two-dimensional inhomogeneous cellular neural networks with/without input. For the first part, applying labeling to the output space produces a sofic shift space. Two invariants, namely spatial entropy and dynamical zeta function, can be exactly computed by studying the induced sofic shift space. This study gives sofic shift a realization through a realistic model. Furthermore, a new phenomenon, the broken of symmetry of entropy, is discovered in multi-layer cellular neural networks with input. The second part is strongly related to the learning problem (or inverse problem); the necessary and sufficient conditions for the admissibility of local patterns must be characterized. The entropy function is dense in [0, log 2] with respect to the parameter space and the radius of the interacting cells, indicating that, in some sense, such system exhibits a wide range of phenomena.. ii.

(6) 誌. 謝. 僅以此論文, 獻給未能來得及分享我的喜悅的祖父大人−張公水發先生−一位對晚輩疼愛有加的慈祥長者; 以及我那即將臨世的孩兒。. 求學生涯中 50% 都在新竹的我, 經常被笑說是 “披著台中皮的新竹人”。因緣際會下, 我來到了新竹展開我的大學生活, 沒想到就此和新竹結下了不解之緣。在這十多年的求學過程中, 我經歷了清、交兩校截然不同的校風−清大的理論與交大的應用, 體會到數學這門浩瀚無邊的學問兩種不同的面貌之美; 讓我更加清楚明白的知道自己未來的方向與興趣所在。在順利獲得博士學位的同時, 我要向這一路上協助我的人致上最深的謝意。首先我要感謝我的指導教授−林松山老師, 在我初來乍到交大時, 不但親切的帶領我認識學校的環境, 還引領我瞭解有關動態系統以及微分方程各個分支的學問。當我遇到困惑難解的瓶頸時, 老師總會適時的伸出援手, 指引我方針; 並且鼓勵我四處參加研討會, 增廣自己的見聞。更進一步的開啟了我跨領域研究的視野, 讓我清楚的看到: 原來數學不是只有理論的推衍, 它的觸角還可以延伸到許多與我們息息相關的切身問題。除了帶領我看到數學豐富的層次之外, 無論我選擇哪一條路, 老師總是給予我無限的支持與祝福, 讓我可以放心大膽地去嘗試不同的事物。嚴謹且一絲不苟的學術態度, 深入淺出的教學方法, 更是我努力學習的圭臬。另外, 感謝靜宜大學的羅主斌老師。如果當初不是老師慷慨的與我分享下榻的旅館, 我不可能參加歐洲數學協會所舉辦關於心臟生理的 summer school, 在繁忙的會議期間, 還不厭其煩地教導我許多知識。回到台灣之後, 更不在意我那不紮實的 background, 帶領我一步步的進入生物數學的殿堂。舉凡最基礎的生理介紹、數學模型、當今生物數學家們所關心的問題, 一直到從事跨領域研究所需具備的知識與觀念, 老師都不厭其煩的講述, 更犧牲他寶貴的時間與我討論。而老師對學術研究那無止盡的熱情, 不但激勵了我的鬥志, 更在我疲憊想偷懶時, 為我注入一劑強心針。無時無刻不在提醒著我要更加努力, 才不枉費老師的一番苦心。如果說我 iii.

(7) 今天得以略窺生物數學的門徑, 無非都是老師的功勞。家人在背後給我的支持與鼓勵, 更是讓我在這條漫長的求學生涯中, 能夠一路走來、始終如一的重要支柱。不管家裡的經濟狀況有多艱辛, 爸爸和媽媽總不要我代為分憂解勞, 他們只要我專心於學術研究。三不五時和弟弟的鬥嘴, 讓我忘記了生活中偶發的不愉快與研究工作上所遇到的瓶頸。我至愛的另一半−玫珍, 謝謝你在我背後無怨無悔的付出。因為你的支持與鼓勵, 我才能毫無後顧之憂的在學業上打拼, 並且對自己的未來充滿了信心; 也謝謝我的岳父、岳母願意將他們的寶貝女兒託付與我。沒有你們大家, 就不可能成就今天的我。感謝這幾年教導我的老師們: 石至文老師為我開啟了微分方程的一扇窗; 李明佳老師總能用簡單的一兩句話闡述一個抽象的觀念, 在微分方程專題討論的課堂上, 一步一腳印的帶我們探討各種層面的問題與現象。謝謝靜宜大學的田慧君老師在數值技巧與程式方面的指導, 以及生活上關心的問候。此外, 我還要謝謝伴我度過這幾年博士生涯的好朋友。班榮超學長, 除了是從大學至今的學長, 也是我研究工作上的合作伙伴。你對自我要求的高標準以及學識的淵博, 是我致力追求的目標; 在尋找問題、挖掘問題、解決問題, 乃至於論文的撰寫, 你都提供我許多寶貴的意見, 無私的交流讓我獲益良多。林吟衡學姊, 我們的師門之花; 紮實的基本功與細心的研究態度, 不啻為我這粗心的男生樹立了一個研究人員的典範, 更親切的與我分享應對進退中該注意的小細節。謝謝同師門的學弟妹−文貴和倖綺−在研究的過程中不厭其煩的和我討論枝微末節; 同研究室的耀漢與明杰, 與你們的討論讓我收穫良多。盈智, 我從大三開始同居的室友, 也是研究路上的好伙伴, 雖然博士班後專攻的領域不同, 在心得交換中發現的共通之處讓我們更確切的知道, 數學果然是一個大家庭。怡菁、小巴、恭儉、康伶、玉雯、佳玲 · · ·, 謝謝你們的陪伴, 豐富了我的博士班生活。. 最後, 我想引用證嚴上人的一句話−也是我的座右銘−作為結尾:. iv.

(8) 做就對了。張志鴻. v. 2008/6/16. 於交大.

(9) Contents 1 Introduction. 1. 2 One-layer Cellular Neural Networks with Input. 9. 2.1. Ordering Matrix and Transition Matrix . . . . . . . . . . . .. 9. 2.1.1. Partition of Parameter Space . . . . . . . . . . . . . .. 9. 2.1.2. Ordering Matrix . . . . . . . . . . . . . . . . . . . . .. 11. 2.1.3. Transition Matrix. . . . . . . . . . . . . . . . . . . . .. 14. 2.1.4. Patterns Generation . . . . . . . . . . . . . . . . . . .. 16. 2.2. Definition and Background of Sofic Shifts . . . . . . . . . . .. 18. 2.3. Entropy and Zeta Function . . . . . . . . . . . . . . . . . . .. 20. 2.3.1. Sofic Shift . . . . . . . . . . . . . . . . . . . . . . . . .. 20. 2.3.2. Entropy . . . . . . . . . . . . . . . . . . . . . . . . . .. 23. 2.3.3. Zeta Function . . . . . . . . . . . . . . . . . . . . . . .. 26. 3 Multi-layer Cellular Neural Networks. 30. 3.1. Partition of Parameter Space . . . . . . . . . . . . . . . . . .. 30. 3.2. Ordering Matrix . . . . . . . . . . . . . . . . . . . . . . . . .. 32. 3.3. Transition Matrix . . . . . . . . . . . . . . . . . . . . . . . . .. 33. 3.4. Entropy and Zeta Function . . . . . . . . . . . . . . . . . . .. 35. i.

(10) 3.5. The Broken of Symmetry . . . . . . . . . . . . . . . . . . . .. 39. 4 Study of an Example. 41. 5 Inhomogeneous Cellular Neural Networks. 46. 5.1. Separation property . . . . . . . . . . . . . . . . . . . . . . .. 6 Inhomogeneous Cellular Neural Networks without Input. 47. 53. 6.1. Two-dimensional subshift of finite type . . . . . . . . . . . . .. 53. 6.2. Two-dimensional inhomogeneous cellular neural networks . .. 58. 7 Inhomogeneous Cellular Neural Networks with Input. 63. Reference. 65. ii.

(11) 1. Introduction. This dissertation includes two investigations. First we study the spatial complexity in multi-layer cellular neural networks, and what comes next is the elucidation of the dense property of topological entropy in inhomogeneous cellular neural networks. The cellular neural network (CNN) proposed by Chua and Yang is a large aggregate of analogue circuits [12; 13]. The system presents itself as an array of identical cells which are all locally coupled. Many such systems have been studied as models for spatial pattern formation in biology [16; 17; 18; 26; 27], chemistry [19], physics [10], image processing and pattern recognition [11]. The complexity of the set of global patterns for one- or two-dimensional cellular neural networks has been widely discussed [3; 5; 6; 7; 8; 22; 24; 38]. However, this study is the first to explore the complexity for one-dimensional multi-layer CNN. The two-dimensional sofic and two-dimensional multi-layer CNN are discussed in other papers. A one-dimensional multi-layer CNN system with input is realized as the following form, (n). dxi dt. (n). = −xi. +. X. (n) (n). ak yi+k +. |k|≤d. X. (n) (n). bk ui+k + z (n) ,. (1). |k|≤d. for some d ∈ N, 1 ≤ n ≤ N ∈ N, i ∈ Z, where (n). ui. (n−1). = yi. for 2 ≤ n ≤ N,. (1). ui. = ui ,. xi (0) = x0i ,. (2). and y = f (x) =. 1 (|x + 1| − |x − 1|) 2. (3) (n). (n). is the output function. For 1 ≤ n ≤ N , parameter A(n) = (a−d , · · · , ad ) is (n). (n). called the feedback template; B (n) = (b−d , · · · , bd ) is called the controlling 1.

(12) (n). template, and z (n) is the threshold. The quantity xi. denotes the state (n). of a cell Ci in the n-th layer. The stationary solutions x ¯ = (¯ xi ) of (1) (n). are essential for understanding the system, and their outputs y¯i (n). (n). = f (¯ xi ). (n). are called patterns. A mosaic solution (¯ xi ) satisfies |¯ xi | > 1 for all i, n. Hence the investigation of stationary solution of N -layer CNN is to study a N -coupled map lattice.  X (1) (1) X (1) (1) (1)   ak yi+k + bk ui+k + z (1) , xi =     |k|≤d |k|≤d    X X (2) (1)   (2) (2) (2)  ak yi+k + bk yi+k + z (2) ,   xi =           (N )   xi =  . |k|≤d. .. . X. |k|≤d. (N ) (N ). ak yi+k +. |k|≤d. X. (N ) (N −1). bk yi+k. (4). + z (N ) .. |k|≤d. One-layer CNN with input is first considered. Let P n+2 = {(A, B, z) : A, B ∈ M1×(2d+1) (R), z ∈ R},. (5). where n = 4d + 1. The parameter space P n+2 can be partitioned into finite sub-regions, such that each region has the same mosaic patterns. Once the region of the parameters space is chosen, the basic set of admissible local patterns B ⊆ {+, −}Z3×2 is then determined. The ordering matrix of all local patterns in {+, −}Z3×2 is defined. For a given basic set B, the transition matrix T(B) is then obtained, and a shift space is induced. For simplicity, considering the case d = 1, i.e., each cell can only interact with their nearest neighbors. In one-dimensional one-layer CNN without input, every partition is associated with a unique set of admissible patterns B = B3×1 and the transition matrix T = T(B3×1 ) [24]. Let Y = {(yi )i∈Z | yi−1 yi yi+1 ∈ B for all i ∈ Z},. (6). then Y is a shift of finite type (SOFT). The number of global admissible patterns with length n and the number of periodic patterns with period m 2.

(13) can then be formulated from the transition matrix T. However, this can not be done when the basic set of admissible local patterns B = B3×2 is derived from the one-layer CNN with input. More precisely, each pattern that is produced from the system is a coupled pattern y1 y2 y3 , where y1 y2 y3 u1 u2 u3 denotes the output pattern, and u1 u2 u3 denotes the input pattern. For simplicity, rewriting the coupled pattern as y1 y2 y3 ⋄ u1 u2 u3 . The output space is defined as    Z Z  (· · · y−1 y0 y1 · · · ) ∈ {+, −} : there exists (· · · u−1 u0 u1 · · · ) ∈ {+, −}   YU = ,     such that (· · · y−1 y0 y1 · · · ⋄ · · · u−1 u0 u1 · · · ) ∈ Σ(B) (7). where Σ(B) ⊆ {+, −}Z∞×2 is a subshift space generated by B ⊆ {+, −}Z3×2 . Analytical results indicate that YU is not a SOFT, but a sofic shift (Theorem 2.13). Under this situation, the formula of spatial entropy (entropy) h(B) (Theorem 2.17) and dynamical zeta function (zeta function) ζσ (t) (Theorem 2.24) can be computed. Therefore, the dynamics of the mosaic solutions of multi-layer CNN are understood. Conversely, the sofic shift is realized through a realistic model. The analysis gets more complicated in N -layer CNN, N ≥ 2. However, once recognizing the elaborate content of one-layer CNN with input, all results for one-layer CNN with input can be extended to general case with analogous method. We like to emphasize that each layer induces a sofic shift and the N -layer coupled system induces the convolution of N -many independent sofic shifts. Hence, Section 2 studies one-layer CNN with and without input and emphases the difference. Without input, the dynamical system is subshift of finite type and then sofic when input appears. Section 3 consists those general results introduced in Section 2. The dynamics of multi-layer CNN with input produce a phenomenon that is never seen in one-layer CNN without input. The entropy of the 3.

(14) one-layer CNN without input has a symmetry about the parameters. More precisely, consider the one-dimensional CNN, dxi = −xi + al yi−1 + ayi + ar yi+1 + z, dt. (8). and select one of the partitions of parameter space {(al , ar ) : al , ar ∈ R} = R2 . The parameters a and z thus have 25 subregions, each with the same entropy. Furthermore, h(B([m, n])) = h(B([n, m])), for 0 ≤ m, n ≤ 4.. (9). The details as in [24]. However, when considering multi-layer CNN with input, not only the entropy and zeta function are varied, but the symmetry of the entropy is broken even for the simplest case one-layer CNN with input. Hence, input adding for a CNN system is the main mechanism that breaks the symmetry of entropy. Since the spatial entropy and dynamical zeta function can be formulated in a theoretical procedure, it is comprehensible to ask how complicated such system can be. In other words, how many phenomena can be observed in the system? From the viewpoint of application aspect, most media in natural systems, including physical, biological and electronic systems, are spatially inhomogeneous [21; 33; 37; 25; 20; 14]. This motivates the study of inhomogeneous cellular networks (ICNN). A two-dimensional ICNN system is of the form,  P  −x + z + ak,l f (xi+k,j+l )  i,j   |k|,|l|≤d   dxi,j P = (10) + bk,l ui+k,j+l , i, j ≡ 0 mod m;  dt  |k|,|l|≤d     −xi,j + z ′ + a0,0 f (xi,j ), otherwise.. for some m ∈ N, i, j ∈ Z. Restated, the difference between CNN and ICNN is that the templates and threshold at each cell Ci,j are spatially invariant for CNN but variant for ICNN. 4.

(15) It is well-known that an important class of applications is steady-state solutions, including mosaic solutions and defect solutions [11; 22; 24]. In recent years, the complexity of steady-state solutions has been extensively studied, and much attention has been paid to the complexity of the set of global patterns, with particular reference to entropy [1; 4; 2; 5; 6; 9; 10; 22; 23; 24; 28; 29; 30; 32]. To study how rich phenomena a the ICNN can achieve, it is equivalent to ask the following question. Question 1.1. For CNN with/without input, if the radius of the interacting cells d is treated as a parameter, is {h(A, B, z, d)}/{h(A, z, d)} dense in [0, log 2]? Multifractal analysis is introduced to a specified dynamical system when one of its invariant is essentially the same as an interval (See [34] for more detail), this motivates us to consider such question. However, since the well-known fact that the entropy of subshift of finite type take a family of specific values, called Perron number [32], the dense assumption cannot be removed. The main difficulty in solving the question is related to the fact that the admissible local patterns that are produced by CNN are very limited [22; 24]. Restated, there exists U ⊆ {1, −1}Zn×n such that U 6= B(A, z, d)/B(A, B, z, d) for all chosen values of the parameters A, B, z, d, where n = 2d + 1. For example, consider the one-dimensional CNN without input, and the length of interaction d = 1. Figure 1 is the bifurcation diagram that relates admissible local patterns to the parameters A = (al , a, ar ) and z; readers may reference [22; 24] for more details. First, choosing (al , ar ), yields a total of eight partitions, as shown in Fig. 1. Second, the (a − 1, z) plane has 25 regions such that the admissible local patterns will be uniquely determined once the region is chosen. For instance, if the parameters A, z are in region. 5.

(16) ar al = ar. al = −ar. + + + + + + + +. + + + +. -. -. + + + +. -. + + + +. III II I IV VIII V VI VII. -. + + + +. -. + + + +. -. al + + + +. -. -. Figure 1: The bifurcation diagram of 1-D CNN [3, 4] of partition IV, the admissible local patterns are B = {− ⊕ +, − ⊕ −, + ⊕ +, + ⊖ −, + ⊖ +, − ⊖ −, − ⊖ +}. That is, “3” indicates that the three patterns with “+” in the center should be chosen from the bottom, and “4” indicates that all four patterns with “−” in the center can be chosen in IV. Thus, Figs. 1 and 2 show all admissible local patterns of 1-D CNN with d = 1. However, let U ⊆ {1, −1}Z3×1 be the set of patterns which are listed as follows. U = {− ⊕ −, − ⊕ +, + ⊕ −, − ⊖ −, − ⊖ +, + ⊖ −}. Notably, U consists of patterns that are selected from different partitions for al and ar . More precisely, the patterns with “+” in the center are located in partition V such that the parameters al and ar must satisfy the conditions al < 0 and ar > 0. Moreover, the patterns with “−” in the center are selected 6.

(17) a−1 [4,4 ] [3,4 ] [2,4 ] [ 1,4 ] [0,4 ]. [4,3 ] [3,3 ]. [2,3 ] [ 1,3 ]. [0,3 ]. [4,2 ] [3,2 ]. [2,2 ] [ 1,2 ]. [0,2 ]. [4, 1] [3, 1]. [2, 1] [ 1, 1]. [0, 1]. [4, 0]. z. [3, 0] [2, 0]. [ 1, 0] [0, 0]. Figure 2: The partition of a − z plane of 1-D CNN from partition I, in which the associated parameters al , ar must then satisfy al , ar > 0. Accordingly, there does not exist A, z such that B(A, z) = U . Thus, some values of entropy cannot be attained for all choices of 3 × 1 basic sets for d = 1. In a work on dense entropy, Quas and Trow [36] showed that every subshift of finite type (SFT) X with positive entropy has proper SFT X′ which is a subsystem of X whose entropy is strictly less than the entropy of X, but whose entropy is arbitrarily close to that of X. However, they cannot be guaranteed to be mixing [35]. Recently, Desai [15] proved that for any Zd -SFT R of positive entropy, the SFT subsystems achieve dense entropy in [0, h(R)]. Thus, if R is treated as a full shift, then the SFT is dense in [0, log |A|], where A denotes the symbols of R, and this result can be generalized to sofic systems. Restated, given a Zd sofic shift T, the sofic shift. 7.

(18) subsystems achieve dense entropy in [0, h(T)]. However, a difficulty similar to that associated with CNN arises in solving the problem of ICNN. The difficulty is to guarantee that the patterns that would achieve the desired entropy can be produced by an ICNN system with/without input. This investigation proposes a necessary and sufficient condition for the admissibility of local patterns of ICNN, and demonstrates that suitable local patterns can be found that achieve the given t ∈ [0, log 2] (according to Theorem 6.11 for ICNN without input and Theorem 7.3 for the case with input). Finding these patterns solves the dense entropy problem for ICNN. About the same question to classical CNN, we have the following conjecture.. Conjecture. For any ǫ > 0 and λ ∈ [0, log 2], there exists template A and threshold z such that |h(B(A, z)) − λ| < ǫ.. This dissertation is organized as follows. Section 2 describes the complexity of the global set of output patterns for one-layer CNN with input. The entropy and zeta function can be exactly computed through the induced sofic shift space. Section 3 extends all results in Section 2 to N -layer CNN, where N ≥ 2. Section 4 lists the detail of the partition in Example 2.5. Section 5 introduces the two-dimensional ICNN model and preliminaries that constitute the background for two-dimensional CNN and extends to two-dimensional ICNN. Section 6 then presents a general theory that yields details about how ICNN relates to a shift of finite type. The solution to the dense entropy problem is also addressed. Section 7 extends the results in Section 6 to ICNN with input.. 8.

(19) 2. One-layer Cellular Neural Networks with Input. The complexity of the global set of output patterns for one-layer CNN with input is investigated in this section.. 2.1. Ordering Matrix and Transition Matrix. In this section, the parameter space P n+2 as in (5) will be partitioned into finite sub-regions, such that each region has the same mosaic patterns. Once the region of the parameters space is chosen, the basic set of admissible local patterns B ⊆ {+, −}Z3×2 is then determined. Then, the ordering matrix of all local patterns in {+, −}Z3×2 will be defined. For a given basic set B, the transition matrix T(B) will be obtained.. 2.1.1. Partition of Parameter Space. This subsection explores the relationship between the parameters of templates and the admissible local output patterns. The differential equation of CNN with input is of the form X X dxi = −xi + ak yi+k + bℓ ui+ℓ + z, dt |k|≤d. (1). |ℓ|≤d. where A = [−ad , · · · , a, · · · , ad ], B = [−bd , · · · , b, · · · , bd ] are the feedback 1 and controlling templates, respectively, y = f (x) = (|x + 1| − |x − 1|) is 2 the output function, z is the threshold, and a ≡ a0 , b ≡ b0 . The quantity xi represents the state of the cell at i. The stationary solution x ¯ = (¯ xi ) of (1) satisfies x ¯i =. X. ak y¯i+k +. |k|≤d. X. |ℓ|≤d. 9. bℓ ui+ℓ + z..

(20) The output y¯ = (¯ yi ) is called output pattern. A mosaic solution x ¯ satisfies |¯ xi | > 1 and its corresponding pattern y¯ is called a mosaic output pattern. Consider the mosaic solution x ¯, the necessary and sufficient conditions for state “+” at cell Ci , i.e., x ¯i > 1, is a − 1 + z > −(. X. ak y¯i+k +. 0<|k|≤d. X. bℓ ui+ℓ ).. (2). |ℓ|≤d. Similarly, the necessary and sufficient conditions for state “−” at cell Ci , i.e., x ¯i < −1, is a−1−z >. X. ak y¯i+k +. 0<|k|≤d. X. bℓ ui+ℓ .. (3). |ℓ|≤d. For simplicity, denoting y¯i by yi and rewriting the output patterns y−d · · · y · · · yd coupled with input u−d · · · u · · · ud as y−d · · · y · · · yd u−d · · · u · · · ud. = Y ⋄ U,. (4). where Y = y−d · · · y · · · yd , U = u−d · · · u · · · ud . Let V n = {v ∈ Rn : v = (v1 , v2 , · · · , vn ), and |vi | = 1, 1 ≤ i ≤ n}, where n = 4d + 1, (2) and (3) can be rewritten in a compact form by introducing the following notation. Denote α = (a−d , · · · , a−1 , a1 , · · · , ad ), β = (b−d , · · · , b, · · · , bd ). Then, α can be used to represent A′ , the surrounding template of A without center, and β can be used to represent the template B. The basic set of admissible local patterns with “+” state in the center is defined as B(+, A, B, z) = {v ⋄ w ∈ V n : a − 1 + z > −(α · v + β · w)},. (5). where · is the inner product in Euclidean space. Similarly, the basic set of admissible local patterns with “−” state in the center is defined as B(−, A, B, z) = {v ′ ⋄ w′ ∈ V n : a − 1 − z > α · v + β · w}. 10. (6).

(21) Furthermore, the admissible local patterns induced by (A, B, z) can be denoted by B(A, B, z) = (B(+, A, B, z), B(−, A, B, z)).. (7). P n+2 = {(A, B, z)| A, B ∈ M1×(2d+1) (R), z ∈ R},. (8). Let. where Mr×s (R) means a r × s real matrix. P n+2 can be partitioned so that each subregion generates the same mosaic patterns, when the controlling template B ≡ 0 is proved in [22]. The general results for B 6= 0 can be obtained similarly, so the detailed proof is omitted for simplicity. Theorem 2.1. There exists positive integer K(n) and an unique collection n+2 satisfying of open subsets {Pk }K k=1 of P. K. (i) P n+2 = ∪ P k . k=1. (ii) Pk ∩ Pℓ = ∅ for all k 6= ℓ. e B, e ze) if and only if (A, B, z), (A, e B, e ze) ∈ Pk for some (iii) B(A, B, z) = B(A, k.. Here P is the closure of P in P n+2 .. 2.1.2. Ordering Matrix. This subsection defines the ordering matrix X = X3×2 of all possible local patterns in {+, −}Z3×2 . First, the notation of the pattern with size 3 × 1 is considered. Let a00 = −−, a01 = −+, a10 = +−, a11 = ++, 11. (9).

(22) −− −− −− −− −− −+ −− +− −− ++ −+ −− −+ −+ −+ +− −+ ++ +− −− +− −+ +− +− +− ++ ++ −− ++ −+ ++ +− ++ ++. −− −− −+ +−. −− ++. −+ −−. −+ −+ −+ +−. −−− −−− −−− −−+. ∅ ∅. −−+ −−+ −−− −−+. ∅ ∅. ∅ ∅. −−− −−− −+− −++. ∅ ∅. −−+ −−+ −+− −++. −−− −−− +−− +−+. ∅ ∅. −−+ −−+ +−− +−+. ∅ ∅. ∅ ∅. −−− −−− ++− +++. ∅ ∅. −−+ −−+ ++− +++. ∅. +− −−. −+ ++. +− ++. ∅. ∅. +−− +−− −−− −−+. ∅ ∅. +−+ +−+ −−− −−+. ∅ ∅. ∅ ∅. +−− +−− −+− −++. ∅ ∅. +−+ +−+ −+− −++. +−− +−− +−− +−+. ∅ ∅. +−+ +−+ +−− +−+. ∅ ∅. ∅ ∅. +−− +−− ++− +++. ∅ ∅. +−+ +−+ ++− +++. ∅. +− +− −+ +−. ∅. ++ −−. ++ ++ −+ +−. ∅. −+− −+− −−− −−+. ∅ ∅. −++ −++ −−− −−+. ∅ ∅. ∅ ∅. −+− −+− −+− −++. ∅ ∅. −++ −++ −+− −++. −+− −+− +−− +−+. ∅ ∅. −++ −++ +−− +−+. ∅ ∅. ∅ ∅. −+− −+− ++− +++. ∅ ∅. −++ −++ ++− +++. ∅. ∅. ++− ++− −−− −−+. ∅ ∅. +++ +++ −−− −−+. ∅ ∅. ∅ ∅. ++− ++− −+− −++. ∅ ∅. +++ +++ −+− −++. ++− ++− +−− +−+. ∅ ∅. +++ +++ +−− +−+. ∅ ∅. ∅ ∅. ++− ++− ++− +++. ∅ ∅. +++ +++ ++− +++. Figure 3: The ordering matrix of all local patterns in Z3×2. 12. ++ ++.

(23) defining ai1 i2 ai′2 i3 = ∅ ⇔ i2 6= i′2 .. (10). If ai1 i2 ai′2 i3 6= ∅, then denoting it by ai1 ai2 ai3 and it is a pattern with size 3 × 1. Define  X  11  X21 X=  X31  X41. X12 X13 X14. . . xij;11 xij;12 xij;13 xij;14. .       xij;21 xij;22 xij;23 xij;24  X22 X23 X24   , Xij =   (11)    xij;31 xij;32 xij;33 xij;34  X32 X33 X34     X42 X43 X44 xij;41 xij;42 xij;43 xij;44. for 1 ≤ i, j ≤ 4 as Figure 3. xij;kl means the pattern. ar1 r2 ar2′ r3 , where as1 s2 as′2 s3. . i−1 j−1 ′ , r2 = i − 1 − 2r1 , r2 = , r3 = j − 1 − 2r2′ , r1 = 2 2 k−1 l−1 ′ s1 = , s2 = k − 1 − 2s1 , s2 = , s3 = l − 1 − 2s′2 , 2 2 (12) and [·] is the Gauss function. If ar1 r2 ar2′ r3 = ∅ or as1 s2 as′2 s3 = ∅, then xij;kl = ∅. Furthermore, if xij;kl 6= ∅, then it is denoted by the pattern ar1 ar2 ar3 in {+, −}Z3×2 . Hence, as1 as2 as3 the self-similar property appear in X as in Figure 3, i.e., the upper pattern of Xij is the same as the lower pattern of xkl;ij , for 1 ≤ i, j, k, l ≤ 4. Once the basic set of admissible local patterns B ⊆ {+, −}Z3×2 is given, defining Σn×2 (B) the collection of all patterns with size n × 2 generated by B as Zn×2 yi−1 yi yi+1 y y · · · y 1 2 n Σn×2 (B) = ∈ {+, −} : ∈ B for all 2 ≤ i ≤ n − 1 . ui−1 ui ui+1 u1 u2 · · · un (13) For simplicity, rewriting y1 y2 · · · yn as y1 y2 · · · yn ⋄ u1 u2 · · · un , where yi , ui ∈ u1 u2 · · · un {+, −}, 1 ≤ i ≤ n. To measure the complexity of the global set of output patterns, the following subshift space in {+, −}Z is considered. Defining the output space 13.

(24) YU ≡ YU (B) by      (· · · y−1 y0 y1 · · · ) ∈ {+, −}Z : there exists (· · · u−1 u0 u1 · · · ) ∈ {+, −}Z  , YU =     such that (· · · y−1 y0 y1 · · · ⋄ · · · u−1 u0 u1 · · · ) ∈ Σ(B) (14). where Σ(B) ⊆ {+, −}Z∞×2 is a subshift space generated by B ⊆ {+, −}Z3×2 .. 2.1.3. Transition Matrix. This subsection derives the transition matrix for a given basic set B. The transition matrix T is defined as T(B) = (Tij ), 1 ≤ i, j ≤ 4, where Tij = (tij;kl ) ∈ M4×4 (R) and   1, if x ij;kl ∈ B; tij;kl =  0, if x Z3×2 \ B or x ij;kl ∈ {+, −} ij;kl = ∅,. (15). (16). ar1 r2 ar2′ r3 satisfies (12). Once T(B) is constructed, it is then as1 s2 as′2 s3 rewritten as T(B) = (tpq ) ∈ M16×16 (R), where. where xij;kl =. tpq = tij;kl , for p = 4(i − 1) + k, q = 4(j − 1) + l.. (17). If templates A, B and threshold z are given, then the basic set B(A, B, z) is obtained from Theorem 2.1. Moreover, the transition matrix T is immediately derived from (16) and (17). If a set of input patterns U ≡ {u1 u2 u3 } ⊆ {+, −}Z3×1 is assigned, then the basic set of admissible local patterns is denoted by y y y 1 2 3 B((A, B, z); U ) = ∈ B(A, B, z) : u1 u2 u3 ∈ U . u1 u2 u3. 14. (18).

(25) The transition matrix for U is defined by U = (uij ) ∈ M4×4 (R), where   1, if u1 u2 u3 ∈ U uij = (19)  0, otherwise.. b denotes the transition matrix of B((A, B, z); U ), then the following If T. theorem is obtained. Before the theorem is stated, two products of matrices are defined as follows. Definition 2.2. For any two matrices M = (mij ) ∈ Mk×k (R), N = (ni′ j ′ ) ∈ Mℓ×ℓ (R), the Kronecker product (tensor product) M ⊗ N of M and N is defined by M ⊗ N = (mij N) ∈ Mkℓ×kℓ (R).. (20). Next, for any P = (pij ), Q = (qij ) ∈ Mr×r (R), the Hadamard product P ◦ Q of P and Q is defined by P ◦ Q = (pij qij ) ∈ Mr×r (R).. (21). Theorem 2.3. If (A, B, z) and U are given, then b T(B((A, B, z); U )) = T(B(A, B, z)) ◦ (E4 ⊗ U),. (22). where ◦ and ⊗ is the Hadamard product and Kronecker product in Definition 2.2, respectively, E4 = (eij ) ∈ M4×4 (R) with eij = 1 for all i, j, is the full matrix.. b = (tˆpq ) ∈ M16×16 (R), rewriting tˆpq = Proof. Let the transition matrix T. tˆij;kl , where. . p−1 i= + 1, 4. j = p − 4(i − 1),. . q−1 k= + 1, 4. and l = q − 4(j − 1). By (16) and (18), tˆpq = tˆij;kl = tij;kl · ukl is obtained. This completes the proof. 15.

(26) 2.1.4. Patterns Generation. This subsection introduces the patterns generation problem induced by (A, B, z). Some results of patterns generation problem induced by (A, z) must be recalled before stating the main theory [32]. The basic set of admissible local patterns B(A, z) ≡ B is then determined once (A, z) is given. The shift space generated by B, denoted by Σ(B), is given by Σ(B) = {y = (yi )i∈Z ∈ {+, −}Z : yi−1 yi yi+1 ∈ B for all i ∈ Z}.. (23). The shift space Σ(B) is thus a subshift of finite type for all B = B(A, z). Let Σn (B) denote the set of n-blocks (i.e., the pattern with size n × 1) in Σ(B), and Γn (B) denote the number of the set of n-blocks. The theorem follows below [32]. Theorem 2.4. If B = B(A, z) is given, T is the transition matrix induced P by B, then Γn (B) = |Tn−2 | for all n ∈ N, n ≥ 3, where |T| ≡ |tij | for 1≤i,j≤k. all T = (tij ) ∈ Mk×k (R).. Considering (A, B, z) with B 6= 0, let B(A, B, z) ≡ B denote the basic set of admissible local patterns, Σn (YU ) denote the set of n-blocks in YU , i.e., Σn (YU ) =.     y = (yi )ni=1 ∈ {+, −}Zn×1 : ∃ u = (ui )ni=1 ∈ {+, −}Zn×1    . such that y ⋄ u ∈ Σn (B).  . ,. (24). and Γn (YU ) denote the number of n-blocks of output patterns generated by B. Theorem 2.4 is invalid in general for deriving the precise value of Γn (YU ) for n ∈ N. An example is given below. The Appendix explains the theorem in detail. Example 2.5. Let A = [al , a, ar ], B = [bl , b, br ] satisfy the following conditions. 16.

(27) (i) al > bl > ar > b > br > 0. (ii) al + b > ar + bl + br , al + br > bl + b. (iii) bl + b > al > ar + b + br . (iv) ar + b > bl + br , bl > ar + br , ar > b + br .. − The positions of ℓ+ i and ℓj on the (a − 1, z) plane are determined exactly as + − in the Appendix. Given region R = [23, 18], i.e., R is bounded by ℓ+ 23 , ℓ24 , ℓ18. and ℓ− 19 , and the set of input patterns is given by U = {−+−, −++, +−+}. Thus, Figure 4 illustrates the basic set of admissible local patterns B = B((A, B, z); U ).. −⊖− −⊞−. −⊖− −⊞+. −⊖− +⊟+. −⊖+ −⊞−. −⊖+ −⊞+. −⊖+ +⊟+. +⊖− −⊞−. +⊖− −⊞+. +⊕+ +⊟+. +⊕+ −⊞+. +⊕+ −⊞−. +⊕− +⊟+. +⊕− −⊞+. +⊕− −⊞−. −⊕+ +⊟+. −⊕+ −⊞+. Figure 4: The basic set of patterns for some templates A, B, z and input U According to Theorem 2.3, the transition matrix of B((A, B, z); U ) is . Tb11 Tb12.    0 b b T = T(B((A, B, z); U )) =  b T31  0. 17. 0. 0. 0. 0. 0. 0. Tb43. 0. .   Tb24  ,  0   Tb44.

(28) where. .  0       0 0 1 1 0  , Tb31 =  Tb11 = Tb12 = Tb43 = Tb44 =     0 1 0 0 0    0 0 0 0 0 0 0 0 0. .   0 0 0 0 0     0 0 0 1 1  , Tb24 =    0 1 0 0 0   0 0 0 0 0. From the transition matrix, the output patterns {−++, ++−, +−−} exist. By the concept of subshift of finite type, the output pattern − + + − − is admissible. However, there exists no u1 u2 u3 u4 u5 ∈ Σ5 (U ) such that − + + − − ⋄ u1 u2 u3 u4 u5 ∈ Σ5 (B). This finding shows that the inner structure needs to be considered. More precisely, since Tb24 Tb43 Tb31 = 0, no input could. possibly produce the output pattern − + + − −.. b n−2 | in general, since So far, this work has shown that Γn (YU ) 6= |T. different input patterns might have the same output pattern. To overcome. this difficulty, the next subsection introduces the concept of sofic shift in the symbolic dynamical system.. 2.2. Definition and Background of Sofic Shifts. This subsection recalls some definitions and main results of sofic shifts. Lind and Marcus has described sofic shifts in detail [32]. Definition 2.6. A labeled graph G = (G, L) consists of an underlying graph G with edge set E, and the labeling L : E → A assigns to each edge a label from the finite alphabet A. A sofic shift is defined by X = XG for some labeled graph G. Definition 2.7. A labeled graph G = (G, L) is right-resolving if, for each vertex of I of G, the edges starting from I carry different labels. In other words, G is right-resolving if, for each I, the restriction of L to EI is oneto-one, where EI consists of those edges starting from I. 18. 0 0. .   0 1 .  0 0  0 0.

(29) The following theorem shows that every sofic shift has a right-resolving presentation. The method for finding an explicit right-resolving presentation is called the subset construction method.. Subset Construction Method. Let X be a sofic shift over the alphabet A having a presentation G = (G, L) so that X = XG . If G is not right-resolving, then a new labeled graph H = (H, L′ ) is constructed as follows. The vertices I of H are the nonempty subsets of the vertex set V(G) of G. If I ∈ V(H) and a ∈ A, let J denote the set of terminal vertices of edges in G starting at some vertices in I and labeled a, i.e., J is the set of vertices reachable from I using the edges labeled a. There are two cases.. 1. If J = ∅, do nothing. 2. If J 6= ∅, J ∈ V(H) and draw an edge in H from I to J labeled a.. Carrying this out for each I ∈ V(H) and each a ∈ A produces the labeled graph H. Then, each vertex I in H has at most one edge with a given label starting at I. This implies that H is right-resolving. Theorem 2.8. Let G = (G, L) be a labeled graph which is not right-resolving, H = (H, L′ ) be a right-resolving labeled graph constructed under the subset construction method. Then XG = XH , i.e., G and H presents the same shift space.. 19.

(30) 2.3. Entropy and Zeta Function. This subsection investigates the entropy and zeta function for the global set of output patterns using the concepts of sofic shifts.. 2.3.1. Sofic Shift. This subsection shows that the output space of one-layer CNN with input is a sofic shift. For a given basic set B, the transition matrix T is defined as (15), (16) and (17). Let the alphabet S = {sij }1≤i,j≤4 , where sij = ar1 r2 ar2′ r3 , ark rk+1 is defined in (9), and i−1 r1 = , r2 = i − 1 − 2r1 , 2. r2′. . j−1 = , 2. (25). r3 = j − 1 − 2r2′ . (26). By (10), sij = ∅ if r2 6= r2′ . The symbolic transition matrix is defined as S = (sij Tij )1≤i,j≤4 = (sij tij;kl ) ∈ M16×16 (R),. (27). spq ), where where sij tij;kl = ∅ if sij = ∅ or tij;kl = 0. Rewriting S = (˜ s˜pq = sij tij;kl for p = 4(i − 1) + k,. q = 4(j − 1) + l.. Let GT be the underlying graph induced by T with edge set E = {epq : tpq = 1, 1 ≤ p, q ≤ 16}, and the labeling L : E → S defined by L(epq ) = sij . GS = (GT , L) is thus a labeled graph as in Figure 5. By (25), a word si1 i2 si2 i3 in S Z can be defined by si1 i2 si2 i3 = ar1 r2 ar2 r3 ar3 r4 . The edge shift with alphabet S is defined by    Z  (· · · si−1 i0 si0 i1 si1 i2 · · · ) ∈ S : there exists (· · · k−1 k0 k1 · · · )  XG S = .     such that tij ij+1 ;kj kj+1 6= 0 for all j ∈ Z (28) 20.

(31) 00 00. 000. 00 11. 00 01. 10 00. 1 00. 10 11. 10 01 10 10. 00 10 1 01 0 01 01 00 01 11. 01 01. 11 0. 01 0. 0 11. 01 10. 11 00 11 01. 11 11. 111. 11 10. Figure 5: The labeled graph of CNN with input The following theorem is thus obtained. Theorem 2.9. X = XGS is a sofic shift.. The relationship between output space YU and induced sofic shift XGS is then investigated. Definition 2.10. Let A, U be finite alphabets, X be a shift space over A, Bk (X) denote the set of k-blocks that occur in points in X, Φ : Bm+n+1 (X) → U be a block map. Then the map φ : X → U Z defined by y = φ(x) with yi = Φ(xi−m · · · xi−1 xi xi+1 · · · xi+n ) = Φ(x[i−m,i+n] ) is called the sliding block code with memory m and anticipate n induced by Φ.. Let Σ3 (YU ) be the set of 3-blocks in YU as in (24). A block map Φ : Σ3 (YU ) → S is thus defined by Φ(yi yi+1 yi+2 ) = sΨ(yi yi+1 )Ψ(yi+1 yi+2 ) ,. 21. (29).

(32) where Ψ(yi yi+1 ) = 1 + ϕ(yi+1 ) + 2ϕ(yi ) and ϕ is defined by ϕ(+) = 1 and ϕ(−) = 0. Then the map φ : YU → S Z defined by φ(· · · y−1 y0 y1 · · · ) = (· · · si−1 i0 si0 i1 si1 i2 · · · ). (30). with sik ik+1 = Φ(yik yik+1 yik+2 ) is a sliding block code from YU to XGS . Definition 2.11. Let φ : X → Y be a sliding block code, then φ is a conjugacy if φ is invertible. It is also called X is conjugate to Y. Theorem 2.12 ([32]). If a sliding code is one-to-one and onto, then it is a conjugacy.. The conjugacy between YU and XGS can be proved now. Theorem 2.13. Given a basic set B, the transition matrix T and the output space YU are then obtained. Let GS = (GT , L) be the labeled graph induced by B, then YU is conjugate to XGS under the sliding block code φ defined in (30).. Proof. It suffices to prove that φ is one-to-one and onto. If there exist x 6= y ∈ YU such that φ(x) = φ(y), without loss of generality, assuming that there is a number n such that xn 6= yn and xi = yi for all i < n. This means that at state xn−1 = yn−1 and xn−2 = yn−2 , Ψ(xn−1 xn ) = 1 + ϕ(xn−1 ) + 2ϕ(xn ) 6= 1 + ϕ(yn−1 ) + 2ϕ(yn ) = Ψ(yn−1 yn ). Therefore, Φ(xn−2 xn−1 xn ) = sΨ(xn−2 xn−1 )Ψ(xn−1 xn ) 6= sΨ(yn−2 yn−1 )Ψ(yn−1 yn ) = Φ(yn−2 yn−1 yn ). This contradicts to φ(x) = φ(y). Thus, φ is one-to-one. For every s = (· · · si−1 i0 si0 i1 si1 i2 · · · ) ∈ XGS , by (28) there exists a sequence (· · · k−1 k0 k1 · · · ) such that tij ij+1 ;kj kj+1 6= 0 for all j ∈ Z. By 22.

(33) (16), the related pattern · · · xi−1 i0 ;k−1 k0 xi0 i1 ;k0k1 xi1 i2 ;k1 k2 · · · is admissible, i.e., · · · ar−1 ar0 ar1 · · · is admissible. By the definition of output space (14), · · · as−1 as0 as1 · · · the pattern · · · ar−1 ar0 ar1 · · · ∈ YU . Moreover, by the relation in (12),   1, if ar = +; 0 ik = 1 + rk+1 + 2rk , and rk =  0, if a = −. r0. Hence, there exists · · · ar−1 ar0 ar1 · · · ∈ YU such that φ(· · · ar−1 ar0 ar1 · · · ) = s. This shows that φ is onto, and the proof is completed.. 2.3.2. Entropy. Let Σn (YU ) ⊆ {+, −}Zn×1 denote the set of n-blocks in YU as in (24). The spatial entropy of YU is defined by h(YU ) = lim. n→∞. log Γn (YU ) , n. (31). where Γn (YU ) is the cardinal number of Σn (YU ). Example 2.5 demonstrates that Theorem 2.4 is invalid in computing the spatial entropy of YU . However, Theorem 2.13 shows that the output space YU is conjugate to the sofic shift XGS . The entropy of YU can be computed by the following theorems show in [32]. Theorem 2.14 ([32]). If two shift spaces X and Y are conjugate, then h(X) = h(Y). Theorem 2.15 ([32]). Let G = (G, L) be a labeled graph. If G is rightresolving, then h(XG ) = h(XG ). Theorem 2.16. For YU is given, GS is the sofic shift induced by YU . If GS is right-resolving, then h(YU ) = log ρ(T), where ρ(T) denotes the maximal eigenvalue of T.. 23.

(34) Proof. Since YU is conjugate to XGS , and GS is right-resolving, by Theorem 2.14, Theorem 2.15, and Perron-Frobenius theorem, h(YU ) = h(XGS ) = h(XT ) = log ρ(T).. (32). This completes the proof.. In general, GS induced by YU (B) might not be right-resolving. However, a sofic shift H = (H, L′ ) which is right-resolving can be constructed and still conjugate to YU (B) via the subset construction method stated in Subsection 2.2. Thus Theorem 2.16 can be extended to the general case. Theorem 2.17. For a given B ⊆ {−, +}Z3×2 , let YU ≡ YU (B) be the shift space induced by B. Then there exists a labeled graph representation H = (H, L′ ) such that h(YU ) = h(XH ) = log ρ(H).. (33). Proof. For the admissible local patterns B given, let T and the labeled graph representation GS defined as above. If GS is already right-resolving, then it is done. If not, using subset construction method, there is a labeled graph representation H such that XGS = XH and is right-resolving. Note that the underlying graph and transition matrix of H, H and H, are also derived. Moreover, by Theorem 2.8 and Theorem 2.13, YU is conjugate to XH . Thus, by Theorem 2.16, h(YU ) = h(XH ) = log ρ(H). Example 2.18 (Continued). Let (A, B, z) be the same as in Example 2.5, S = {s11 , s12 , s24 , s31 , s43 , s44 }, by (27), the symbolic transition matrix is . s11 T11 s12 T12.    0 S=  s31 T31  0. 0. 0. 0. 0. 0. 0. s43 T43 24. 0. .   s24 T24  .  0   s44 T44. (34).

(35) Then, the labeled graph representation of B, GS = (GT , L), is not rightresolving. For simplicity, denoting the vertex set of GT by V = {1, 2, · · · , 16}. Using the subset construction method to construct another labeled graph H = (H, L′ ) as Figure 6. Theorem 2.8 shows that XGS = XH . {3}. a. b. a {2}. a. b {11, 12}. c. b. {3, 4}. {6}. c. {16}. d {7, 8}. e. f. e. {15}. {14}. f. c. f. {10} e {15, 16}. {7}. Figure 6: A right-resolving labeled graph via subset construction. The transition matrix of XH  0 0 1 0    1 0 0 1    1 0 0 1    0 0 0 0    0 0 0 0    0 0 0 0  H=  0 0 1 0    0 0 0 0     0 0 0 0    0 0 0 0    0 0 0 0  0 0 0 0. is obtained as below. 0 1 0 0 0 0 0 0. .   0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0    0 0 0 0 0 0 1 0    0 0 0 0 1 0 0 0    0 0 0 0 1 0 0 0   . 0 0 0 0 0 0 0 0    0 0 0 0 0 0 0 0     0 0 0 1 0 0 0 1    0 0 1 0 1 0 0 0    0 0 0 0 0 0 0 0   0 0 1 0 1 0 0 0 25. (35).

(36) . Thus, h((A, B, z); U ) = log λ > 0, where λ = 1.324718 is the root of f (t) = t6 − 2t4 + t2 − 1. In the next subsection, the zeta function of YU will be discussed.. 2.3.3. Zeta Function. Given a sofic shift YU with shift map σ, invariant values and invariant functions of the shift space (YU , σ) are interested. In the last subsection, the entropy h(B) is studied. This subsection examines the zeta function ζσ (t) with respect to the shift map σ. Let pn (σ) = {y = (yi )i∈Z ∈ YU |σ n (y) = y} be the collection of all periodic patterns of period n. The zeta function of σ is defined as ζσ (t) = exp(. ∞ X pn (σ). n=1. where exp(x) =. ∞ X xn n=0. n!. n. tn ),. (36). is the classical exponential function.. If a shift space X is a shift of finite type, then there is an edge shift XA conjugate to X. The following theorem computes the zeta function of any shift of finite type. Theorem 2.19 ([32]). Let A be a k × k nonnegative integer matrix, σA the associated shift map. Then ζσA (t) =. 1 , det(Ik − tA). (37). where Ik is the k × k identity matrix. Thus the zeta function of a shift of finite type is the reciprocal of a polynomial.. The following notations are needed to investigate the zeta function of 26.

(37) sofic shift. Let F = {f1 , f2 , · · · , fm } be a finite set. A permutation π of F is given below as an impression (fi1 , · · · , fim ), i.e., π(fℓ ) = fiℓ for 1 ≤ ℓ ≤ m. Definition 2.20. A permutation π is said to be even (odd, resp.) if the number of interchanges (or transpositions) needed to generate the permutation is even (odd, resp.) Moreover, the sign of π is defined as   1, π is even; sgn(π) =  −1, π is odd.. (38). Let H = (H, L′ ) be a labeled graph which is right-resolving with r many vertices, assuming that V = {1, 2, · · · , r}. Let HS denote the symbolic transition matrix of H and H the transition matrix of the underlying graph H. For 1 ≤ k ≤ r, constructing a labeled graph Hk with alphabet {±sij : sij ∈ S} as follows.. 1. The vertex set of Hk is the set Vk of all subsets of V having k elements, i.e., |Vk | = kr . Moreover, the ordering on the states in each element of Vk is fixed.. 2. For each sij ∈ S, we denote sij (I) the terminal state of sij starts at I. For I = {I1 , · · · , Ik }, J = {J1 , · · · , Jk } ∈ Vk , there is an edge from I to J provided there exists sij ∈ S such that sij (I1 ), · · · , sij (Ik ) are well-defined and (sij (I1 ), · · · , sij (Ik )) is a permutation of J . More than this, the edge is labeled as sij (−sij , reps.) if the permutation is even (odd, resp.) Otherwise, there is no edge with label ±sij from I to J . Definition 2.21. Let HSk denote the symbolic transition matrix of Hk , Hk be obtained from HSk by setting all the symbols in S equal to 1. We call Hk the k-th signed subset matrix of H. 27.

(38) Theorem 2.22 ([32]). Let H = (H, L′ ) be a right-resolving labeled graph with r many vertices, and Hk be its k-th signed subset matrix. Then ζσH (t) =. r Y. k. det(I − tHk )(−1) ,. (39). k=1. where I is the identity matrix. Theorem 2.23 ([32]). If two shift spaces X and Y are conjugate, then ζσX (t) = ζσY (t).. Therefore, the zeta function of the sofic shift XGS can be derived using the following theorem. Theorem 2.24. For a given B ⊆ {−, +}Z3×2 , let YU ≡ YU (B) be the shift space induced by B with shift map σ. Then there exists a labeled graph representation H such that ζσ (t) =. n Y. k. det(I − tHk )(−1) ,. (40). k=1. where Hk is the k-th signed subset matrix of H, and n is the cardinal number of the underlying graph H.. Proof. Let GS be the labeled graph representation of YU , and XGS be the sofic shift induced by GS with shift map σGS . By Theorem 2.13, XGS is conjugate to YU . Thus, we have ζσ (t) = ζσGS (t).. (41). If GS is right-resolving, then it is done by Theorem 2.22. Otherwise, constructing a labeled graph H which is right-resolving and represents the same shift space as GS via subset construction method. This completes the proof.. 28.

(39) Example 2.25 (Continued). Continuing with Example 2.5 and Example 2.18, for the investigation of zeta function, all the k-th signed subset matrix Hk of H are needed to be  0  0   0   0  H2 =  0   0    0  0. constructed. Therefore, H1 = H as in (35),  −1 −1 0 0 0 0 0  −1 −1 0 0 0 0 0   0 0 0 −1 0 0 0   0 0 0 −1 0 0 0  , 0 0 0 0 0 0 0   0 0 0 0 0 −1 −1    0 0 0 0 0 −1 −1  0 0 0 0 0 0 0. and H3 = H4 = · · · = H12 = 0, the zero matrix. Hence, by Theorem 2.22, (1 + t)2 ζσ (t) = . 1 − 2t2 + t4 − t6. 29.

(40) 3. Multi-layer Cellular Neural Networks. In this section, all results in one-layer CNN with input will be extended to multi-layer CNN.. 3.1. Partition of Parameter Space. As in (1), an N -layer CNN system with input is of the form, (n). dxi dt. (n). = −xi. +. X. (n) (n). ak yi+k +. |k|≤d. X. (n) (n). bk ui+k + z (n) ,. (1). |k|≤d. for some d ∈ N, 1 ≤ n ≤ N ∈ N, i ∈ Z, where (n). ui. (n−1). = yi. for 2 ≤ n ≤ N,. (1). ui. = ui ,. xi (0) = x0i .. (2). The feedback and controlling templates of each layer are (n). (n). (n). (n). (n). (n). A(n) = (a−d , a−d+1 , · · · , ad ) and B (n) = (b−d , b−d+1 , · · · , bd ), where 1 ≤ n ≤ N . The parameter space and the admissible local patterns of each layer can be represented by P (n) = {(A(n) , B (n) , z (n) )} and B (n) (A(n) , B (n) , z (n) ), where 1 ≤ n ≤ N . Let A = (A(1) , A(2) , · · · , A(N ) ), B = (B (1) , B (2) , · · · , B (N ) ), z = (z (1) , z (2) , · · · , z (N ) ), P m = (P (1) , P (2) , · · · , P (N ) ), (n) (n). (n). where m = N (2d + 1) − 1, Y (n) = y−d y−d+1 · · · yd , where 1 ≤ n ≤ N , and U = u−d u−d+1 · · · ud , then     (N ) (N −1) (1) Y  ⋄Y ⋄ ··· ⋄ Y ⋄ U : B(A, B, z) = ,   Y (n) ⋄ Y (n−1) ∈ B (n) for 2 ≤ n ≤ N, and Y (1) ⋄ U ∈ B (1)  . (3). where ⋄ is defined in (4). The generalized partition theorem of N -layer CNN then follows. Theorem 3.1. There exists K(m) ∈ N and unique collection of open subsets K(m). {Pk }k=1 of P m such that 30.

(41) K(m). [. (i) P m =. P¯k .. k=1. (ii) Pk. T. Pj = ∅ for k 6= j.. ˜ B, ˜ z˜) ⇔ (A, B, z), (A, ˜ B, ˜ z˜) ∈ Pk for some k. (iii) B(A, B, z) = B(A,. Proof. For simplicity, the case N = 2 is proved. The general case can be done analogously, the details are omitted here. By Theorem 2.1, there exist Ki ∈ N and an unique collection of open (i). (i) such that i subsets {Pk }K k=1 of P. (1). 4d+3 P(i). =. Ki [. (i). Pk .. k=1 (i) T. (2) Pk. (i). Pℓ. = ∅ for k 6= ℓ.. ˜ (i) , z˜(i) ) if and only if (3) B (i) (A(i) , B (i) , z (i) ) = B (i) (A˜(i) , B ˜ (i) , z˜(i) ) ∈ P (i) for some k, (A(i) , B (i) , z (i) ), (A˜(i) , B k (1). (2). where i = 1, 2. Let K ′ = K1 · K2 , define Pk′ = (Pk1 , Pk2 ), where k = (k1 − 1)K2 + k2 , 1 ≤ k1 ≤ K1 , 1 ≤ k2 ≤ K2 . Let P1 = Pi′ , where i = min{k| there exists Y (2) ⋄Y (1) ∈ B (2) and Y (1) ⋄U ∈ B (1) }, (4) and Pℓ = Pi′ , where i = min {k| there exists Y (2) ⋄Y (1) ∈ B (2) and Y (1) ⋄U ∈ B (1) }, k>kℓ−1. (5) for ℓ ≥ 2 and kj ∈ N such that Pk′ j = Pj . Then there exists an positive integer K ≤ K ′ such that {Pk }K k=1 satisfies (i), (ii), (iii), then the proof is completed. 31.

(42) 3.2. Ordering Matrix. The ordering matrix X3×N of all possible local patterns in {+, −}Z3×N is defined recursively as. X3×N. where. . Xi1 j1.   ∅ X11 X12 ∅      ∅ ∅ X23 X24  , =   X31 X32 ∅ ∅    ∅ ∅ X43 X44. Xi1 j1 ;11 Xi1 j1 ;12. ∅. ∅. (6). .      ∅ ∅ Xi1 j1 ;23 Xi1 j1 ;24   , =  Xi1 j1 ;31 Xi1 j1 ;32 ∅ ∅    ∅ ∅ Xi1 j1 ;43 Xi1 j1 ;44. (7). Xi1 j1 ;i2 j2 ;··· ;ik jk =   ∅ ∅ Xi1 j1 ;i2 j2 ;··· ;ik jk ;11 Xi1 j1 ;i2 j2 ;··· ;ik jk ;12      ∅ ∅ Xi1 j1 ;i2 j2 ;··· ;ik jk ;23 Xi1 j1 ;i2 j2 ;··· ;ik jk ;24 ,     Xi1 j1 ;i2 j2 ;··· ;ik jk ;31 Xi1 j1 ;i2 j2 ;··· ;ik jk ;32  ∅ ∅   ∅ ∅ Xi1 j1 ;i2 j2 ;··· ;ik jk ;43 Xi1 j1 ;i2 j2 ;··· ;ik jk ;44 (8) for 1 ≤ k ≤ N − 2, and Xi1 j1 ;i2 j2 ;··· ;iN−1 jN−1 =   ∅ ∅ xi1 j1 ;··· ;iN−1 jN−1 ;11 xi1 j1 ;··· ;iN−1 jN−1 ;12      ∅ ∅ xi1 j1 ;··· ;iN−1 jN−1 ;23 xi1 j1 ;··· ;iN−1 jN−1 ;24 ,     xi1 j1 ;··· ;iN−1 jN−1 ;31 xi1 j1 ;··· ;iN−1 jN−1 ;32  ∅ ∅   ∅ ∅ xi1 j1 ;··· ;iN−1 jN−1 ;43 xi1 j1 ;··· ;iN−1 jN−1 ;44 (9) where 1 ≤ ik , jk ≤ 4, and 1 ≤ k ≤ N . The construction contains a selfsimilarity property in X3×N . As in Subsection 2.1.2, xi1 j1 ;i2 j2 ;··· ;iN−1 jN−1 ;iN jN 32.

(43) means the pattern ′ r ) ⋄ (ar r ar ′ r ) ⋄ · · · ⋄ (ar a′ ) (ar11 r12 ar12 21 22 N1 rN2 rN2 rN3 13 22 23 ′ r in {+, −}Z3×N , where ark1 rk2 ark2 is defined in (10), and k3 ik − 1 jk − 1 ′ ′ , rk2 = ik −1−2rk1 , rk2 = , rk3 = jk −1−2rk2 . rk1 = 2 2 ′ r The pattern is ∅ if ark1 rk2 ark2 = ∅ for some 1 ≤ k ≤ N . Otherwise, it is k3. denoted by the pattern (ar11 ar12 ar13 ) ⋄ (ar21 ar22 ar23 ) ⋄ · · · ⋄ (arN1 arN2 arN3 ) in {+, −}Z3×∞ . As long as the basic set of the admissible local patterns B ⊆ {+, −}Z3×(N+1) is given, Σm (B) denotes the collection of all m-blocks generated by B. The subshift space of {+, −}Z is then defined by     Y (N ) = (yi(N ) )i∈Z : there exist U, Y (1) , Y (2) , · · · , Y (N −1)   YU = , (10)     such that Y (N ) ⋄ Y (N −1) ⋄ · · · ⋄ Y (1) ⋄ U ∈ Σ(B) where Σ(B) ⊆ {+, −}Z∞×(N+1) is generated by B ⊆ {+, −}Z3×(N+1) .. 3.3. Transition Matrix. The basic set of admissible local patterns B = B(A, B, z) can be determined from the N -layer CNN parameters (A, B, z). Denote by Tn the transition matrix induced by B (n) ⊆ {+, −}Z3×2 , where B (n) is the basic set of admisb N = T(B; U ) be sible local patterns in the n-th layer, and 1 ≤ n ≤ N . Let T. the transition matrix induced by B with the set of input patterns U . The following theorem is then obtained. Theorem 3.2. b N = (TN ⊗ E4N−1 ) ◦ (E4 ⊗ TN −1 ) ∈ M4n+1 ×4n+1 (R), T 33. (11).

(44) where Tn = (Tn ⊗ E4n−1 ) ◦ (E4 ⊗ Tn−1 ) ∈ M4n+1 ×4n+1 (R),. for 2 ≤ n ≤ N − 1, (12). and T1 = T1 ◦ (E4 ⊗ U) ∈ M16×16 (R),. (13). U is the transition matrix of U . Hence T1 is the transition matrix given in Theorem 2.3. In particular, if N = 2, b 2 = (T2 ⊗ E4 ) ◦ (E4 ⊗ (T1 ◦ (E4 ⊗ U))). T. (14). Proof. For simplicity, the case N = 2 is proved. For N ≥ 2, it can be done by mathematical induction, thus is omitted. b 2 = (Tbi j )1≤i ,j ≤4 and T2 = (Ti j )1≤i ,j ≤4 , where Tbi j ∈ Denoting T 1 1 1 1 1 1 1 1 1 1. M16×16 (R) and Ti1 j1 ∈ M4×4 (R) for 1 ≤ i1 , j1 ≤ 4. The case i1 = j1 = 1 is proved, the others can be treated analogously. t11;i2 j2 ;i3 j3 )1≤i3 ,j3 ≤4 ∈ Denoting Tb11 = (Tb11;i2 j2 )1≤i2 ,j2 ≤4 , where Tb11;i2 j2 = (b. M4×4 (R), for fixed 1 ≤ i2 , j2 ≤ 4, and T11 = (t11;i2 j2 )1≤i2 ,j2 ≤4 ∈ M4×4 (R). Since the output patterns of the first layer will be treated as the input patterns of the second layer, let U2 be the output patterns of the first layer coupled with input U . By Theorem 2.3, the transition matrix of U2 is T1 = T1 ◦ (E4 ⊗ U) ∈ M16×16 (R).. (15). Denoting T1 = (T i2 j2 )1≤i2 ,j2 ≤4 ,. T i2 j2 = (ti2 j2 ;i3 j3 )1≤i3 ,j3 ≤4 ∈ M4×4 (R).. (16). Then tˆ11;i2 j2 ;i3 j3 = 1 ⇔ t11;i2 j2 = 1 and t¯i2 j2 ;i3 j3 = 1, 34. (17).

(45) for 1 ≤ i2 , j2 , i3 , j3 ≤ 4. That is, Tb11 = (T11 ⊗ E4 ) ◦ (T1 ◦ (E4 ⊗ U)).. (18). The proof is completed.. 3.4. Entropy and Zeta Function. This subsection introduces the formula for calculating entropy and zeta func(n). tion of N -layer CNN. Let S (n) = {sij }1≤i,j≤4 be the alphabets, and let b N for Sn and S be the symbolic transition matrices of Tn over S (n) and T. 1 ≤ n ≤ N . By Theorem 2.9, XGSn is a sofic shift induced by B (n) , where GSn is the labeled graph representation of the n-th layer. Furthermore, YU is the output space induced by the N -layer CNN as defined in (10). The following theorem can be obtained by the same method in Theorem 2.13, so the details are omitted. Theorem 3.3. YU is conjugate to XGS . The definition of convolution is given below. Definition 3.4. Let X, Y be two shift spaces with graph representation GX = (VX , EX ), GY = (VY , EY ), resp., then the convolution of X, Y, denoted by X∗Y, is the shift space with underlying graph GX∗Y = (VX∗Y , EX∗Y ), where VX∗Y = {f (x) ∈ EY | x ∈ VX }. (19). for some f : VX → EY .. The convolution theorem for an N -many sofic shift is then obtained. Theorem 3.5. Let XGS be the sofic shift induced by B, then XGS = XGSN ∗ · · · ∗ XGS2 ∗ XGS1 35. (20).

(46) is the convolution of XGS1 , · · · , XGSN ,. where. b N = (SN ⊗ E4N−1 ) ◦ (E4 ⊗ SN −1 ), S. Sn = (Sn ⊗ E4n−1 ) ◦ (E4 ⊗ Sn−1 ) ∈ M4n+1 ×4n+1 (R),. (21). for 2 ≤ n ≤ N − 1, (22). and S1 = S1 ◦ (E4 ⊗ U) ∈ M16 (R).. (23). Proof. This can be done using the same method used in the proof of Theorem 3.2, the details are omitted.. Thus, the theorems for entropy and zeta function can be found via the same methods as described in the last section. Theorem 3.6. For a given B ⊆ {+, −}Z3×(N+1) , let YU ≡ YU (B) be the shift space induced by B. Then there exists a labeled graph representation H = (H, L′ ) such that h(YU ) = h(XH ) = log ρ(H), and ζσ (t) =. r Y. k. det(I − tHk )(−1) ,. (24). (25). k=1. where Hk is the k-th signed subset matrix of H, and r is the cardinal number of the underlying graph H.. An example for 2-layer CNN is illustrated here. ¯ B (1) = B (2) ≡ B, ¯ Example 3.7. Consider (A, B, z) with A(1) = A(2) ≡ A, ¯ B ¯ and z¯ satisfy the same condition described in z (1) = z (2) ≡ z¯, and A, Example 2.5. Moreover, the set of input patterns is given by U = {− + 36.

(47) −, − + +, + − +}. Then B (1) = B(A(1) , B (1) , z (1) ; U ) is consisting of the following patterns. −⊖− −⊞−. −⊖− −⊞+. −⊖− +⊟+. −⊖+ −⊞−. −⊖+ −⊞+. −⊖+ +⊟+. +⊖− −⊞−. +⊖− −⊞+. +⊕+ +⊟+. +⊕+ −⊞+. +⊕+ −⊞−. +⊕− +⊟+. +⊕− −⊞+. +⊕− −⊞−. −⊕+ +⊟+. −⊕+ −⊞+. Denote U2 the output patterns of B (1) , i.e., U2 = {− − −, − − +, + − −, + + +, + + −, − + +}. Then B (2) = B(A(2) , B (2) , z (2) ; U2 ) is consisting of the following patterns.. +⊕+ +⊞+. +⊕+ +⊞−. +⊕+ +⊟−. +⊕− +⊞+. +⊕− +⊞−. +⊕+ −⊞+. +⊕− +⊟−. −⊕+ +⊞+. +⊕+ −⊟+. −⊕+ +⊞−. +⊕+ −⊟−. +⊕− −⊞+. −⊕+ +⊟−. −⊕− +⊞+. +⊕− −⊟+. −⊕− +⊞−. +⊕− −⊟−. −⊕+ −⊞+. −⊖− −⊟−. −⊖− −⊟+. −⊖− −⊞+. −⊖+ −⊟−. −⊖+ −⊟+. −⊖− +⊟−. −⊖+ −⊞+. +⊖− −⊟−. −⊖− +⊞−. +⊖− −⊟+. −⊖− +⊞+. −⊖+ +⊟−. + ⊖− − ⊞+. 37.

(48) b = T((A, B, z); U ) is The transition matrix T  b b 12 T T 0  11  b 23  0 0 T b = T b T31 0 0  b 43 0 0 T. where. b 43 b 11 = T T. b 23 T and.  0   0 =  0  0. 0 0 0 0.  T T1 0  1  0 0 0 b 44 =  =T  T2 0 0  0 0 T1   0 0 0     0 0 0 , T b 24 =    T2 0 0   0 T1 T1.  0   0 T1 =   0  0. 0. .   0 0 0 0 0     0 0 0 1 1  , T2 =    0 0 1 0 0   0 0 0 0 0. 0 0 0. 0. .   b T24  ,  0   b T44 . (26). T1 T1 0.   0 0 0 b 12 =  T  T2 0 0  0 0 0   0 0 T   1   0 0 T3  , T b 31 =    0 0 0   T1 T1 0.   T3  ,  0  T1 0. then.   0 0 0 0     0 0 1 1  , T3 =    0 1 0 0   0 0 0 0. 0. .   T3  ,  0  0 T1 0. 0 0. 0 0 0. 0.   0 T3  ,  0 0  0 0. .   0 1 .  0 0  0 0. Let S = {s11 , s12 , s23 , s24 , s31 , s43 , s44 }, the symbolic transition matrix is   b 11 s12 T b 12 s11 T 0 0    b 23 s24 T b 24   0  0 s23 T , S= (27)  b  s31 T31 0 0 0    b 43 s44 T b 44 0 0 s43 T. which is not right-resolving. Using subset construction method, the spatial . entropy then can be found, h((A, B, z); U ) = log λ, where λ = 1.49676 is a root of f (t) = t8 − 2t6 + t4 − 3t2 − 1. Moreover, the zeta function is (1 + t + t3 )(1 + t − t3 ) . ζσ (t) = 1 − 2t2 + t4 − 3t6 − t8 38. .

(49) 3.5. The Broken of Symmetry. The basic set of admissible local patterns B can be determined from (A, B, z). The entropy of each partition is symmetrical in one-dimensional CNN without input, i.e., where B ≡ 0 [24]. For example, if (A, z) is picked such that al > ar > 0, then parameters a and z have 25 regions. Clearly, h(B([m, n])) = h(B([n, m])), for 1 ≤ m, n ≤ 4.. (28). The symmetry is broken for the one-layer CNN with input, as shown below with an example. Consider dxi = −xi + al yi−1 + ayi + ar yi+1 + bl ui−1 + bui + br ui+1 + z, dt. (29). where bl = 0, then the symmetry of entropy is broken, as revealed in Figure 7. Table 1: Some maximal eigenvalues produced in one-layer CNN with input. maximal eigenvalue. characteristic polynomial. λ1 = 2 . λ2 = 1.9479 . λ3 = 1.8832 . λ4 = 1.8393 . λ5 = 1.7549 . λ6 = 1.7417 . λ7 = 1.6992 . λ8 = g = 1.618 . λ9 = 1.5618 . λ10 = 1.5289. t−2 t5 − 2t4 + t3 − 2t2 + t − 1 t4 − 2t3 + t2 − 2t + 1 t3 − t2 − t + 1 t3 − 2t2 + t − 1 t8 − 2t7 + t6 − t5 + t4 − 2t3 + t2 − 1 t5 − 2t4 + t3 − 2t + 1 t2 − t − 1 t6 − 2t5 + t4 − t2 + t − 1 t5 − 2t4 + t3 − 1. 39.

(50) [16,16]. k1. [15,16]. k1. [14,16]. k1 k1. k1. k1. [13,15]. k5. k5. [6,16]. [6,15]. k5. [6,14]. [5,15]. k8. k8. k4. [9,12]. [9,11]. k8. k8. k8. k8. k8. [7,10]. k8. k8. k8. [9,8]. k8. k8. k8. k8. [5,10]. [7,8]. k8. k8 k8. [13,6]. [14,5]. [13,5]. k5. k8. k8. k8 [12,5]. k8. [11,5]. [10,5]. k8. k8. k8. [7,7]. [16,5]. [15,5]. k8. k8. [11,6]. [8,7]. k8. [14,6]. k8. [10,6]. k5. k8. [12,6]. k8. k8. k8. [16,6]. [15,6]. k5. k8. [10,7]. [9,7]. [8,8]. [14,7]. [12,7]. [11,7]. k5. k5. [13,7]. k8. [16,7]. [15,7]. k5. [12,8]. k8 [10,8] k8. k8. [7,9]. [14,8]. k8. [9,9]. [8,9]. k8. [6,10]. k5. k5. [13,8]. [11,8]. [10,9]. [16,8]. [15,8]. k5. k8 [11,9]. k8. [14,9]. k8. [12,9]. k5. k5. [13,9]. k5. [10,10]. k8. k8. [12,10]. [16,9]. [15,9]. k5. k5. k5. k5. k5. [13,10]. [11,10]. [9,10]. [8,10]. k8. [5,11]. k4. k5. [8,11]. [6,11]. k8. k4. [10,11]. k8. [7,11]. k8. k4. [16,10]. [15,10]. [14,10]. k4. [12,11]. k5. k1. k2. [13,11]. [11,11]. [16,11]. [15,11]. k3. [11,12]. [10,12]. k1. [14,11]. k4. k4. k5. k8. [13,12]. [12,12]. k4. k8. [6,12]. [5,12]. [11,13]. [10,13]. [7,12]. k8. k8. k3. [8,12]. k8. [5,13]. [12,13]. k8. [6,13]. [5,14]. [11,14]. [9,13]. [7,13]. k8. k2 k3. [8,13]. k5. k8. k3 k4. k8. [7,14]. k3 k3. k5. k5. k5. [15,12]. k2 k5. [8,14]. [14,13]. [14,12]. [9,14]. k5. [7,15]. k5. [5,16]. k5. [13,14]. [13,13]. [10,14]. [16,12]. k2. [12,14]. k5. [8,15]. [7,16]. k2. [10,15]. [9,15]. [8,16]. [12,15]. [15,13]. k1. k2. [16,13]. k1. [14,14]. [11,15]. k5 k5. k1. [15,14]. [11,16]. [9,16]. [16,14]. [14,15]. k1 [10,16]. k1. k1. [13,16] [12,16]. k1. [16,15]. [15,15]. k8. (a) [16,16]. k1. [15,16]. k1. [14,16]. k1 k5. k1. k1. [13,15]. [8,16]. k5. k8. k3. k2 k3. k1. [16,11]. k1. [13,12]. [14,11]. k5. k5. k5. k3. k6. [10,13]. k6. [9,13]. [8,13]. k10. [12,11]. k6. k5. [11,11]. [10,11]. k8. [8,11]. k10. k8. [10,10]. [11,9]. k8. k8. [9,10]. [8,10]. k8. k10. [10,9]. k10. [12,7]. [11,8]. k8 [10,8] k8. [11,7]. k10. k8. k8. [9,8] [8,8]. k10. k10. k10. k8. [8,7]. k10. [13,6]. [13,5]. k8. k8. [11,5]. [10,5]. k8. [12,5]. k9. k9. [14,5]. k8. [11,6]. k5. [15,5]. k8. k8. [16,5]. k8 [14,6]. [12,6]. [10,6]. k5. [15,6]. k5. k8. [10,7]. [9,7]. [14,7]. k8. [16,6]. k5. [13,7]. k8. k5. [15,7]. k5. k8. [9,9]. [8,9]. [14,8]. [12,8]. [16,7]. k5. [13,8]. k8. k5. [15,8]. k5. k8. [16,8]. k5. [13,9]. [12,9]. k5. [15,9]. [14,9]. k8. [11,10]. k6. [9,11]. [13,10]. k8. k6. k5. k5. [12,10]. [16,9]. [15,10]. [14,10]. k5. [11,12]. k6. k8. [13,11]. k5. [10,12]. [9,12]. [8,12]. [12,12]. k5. k2. [12,13]. [11,13]. [16,10]. [15,11]. [11,14]. k8 k10. k5 k5. k7. k8. [15,12]. k5. k5. [8,14]. [14,13]. k5. [9,14]. [8,15]. [13,14]. [14,12]. [10,14]. k7. k2. [13,13]. k5. [16,12]. [15,13]. k1. [12,14]. [10,15]. [9,15]. k8. [12,15]. k1. [14,14]. k5. [16,13]. [11,15]. k5 k7. k1. [15,14]. [11,16]. [9,16]. [16,14]. [14,15]. k5 [10,16]. k1. k1. [13,16] [12,16]. k5. [16,15]. [15,15]. k8. k9. k9. (b). Figure 7: The effect of input patterns. The parameters al , ar , b, br are considered as follows. (i) al > ar > b > br > 0, (ii) al < b + br , (iii) al + br < ar + b. Subfigure (a) lists regions that produce positive entropy. Those regions with positive entropy are symmetric, i.e., h([m, n]) = h([n, m]). However, such property would be destroyed when input patterns are given. Subfigure (b) lists the same regions as in (a) but the 40 input patterns U = {−−, −+, +−} are considered. It is seen that the symmetry is no longer hold. Herein, ki = log λi for 1 ≤ i ≤ 10 are listed in Table 1..