TABLE I
WEIGHTSVALUES INCONSEQUENCE
time period for one walking step reduces to 0.52 s. The final values of
waiare listed in Table I.
VI. CONCLUSION
Fuzzy neural network approaches for robotic gait synthesis are pre-sented in this paper. The suggested scheme uses a fuzzy modeling neural network controller with the BTT algorithm in the gait synthesis of a walking robot. The uncertainty of the network size in the con-ventional neural network learning scheme has been overcome by the use of fuzzy modeling network. The fuzzy controller can generate con-trol sequences and drive the biped along a desired pattern of a walking gait. The desired pattern is used only as a reference trajectory. The pro-posed learning scheme trains the controller to follow this given pat-tern as closely as possible. Different pruning algorithms, membership functions, and network structures are investigated. Simulation results demonstrate that the desired goals—crossing over a specific clearance, having a desired step length, and walking at a certain speed—were achieved.
REFERENCES
[1] S. Murakami, E. Yamamoto, and K. Fujimoto, “Fuzzy control of dy-namic biped walking robot,” Proc. IEEE Int. Conf. Fuzzy Systems, vol. 1, no. 4, pp. 77–82, 1995.
[2] C. L. Shih, W. A. Gruver, and Y. Zhu, “Fuzzy logic force control for a biped robot,” Proc. IEEE Int. Symp. Intelligent Control, pp. 269–274, 1991.
[3] L. Magdalena and F. Monasterio, “Fuzzy controlled gait synthesis of a biped walking machine,” Proc. IEEE Int. Conf. Fuzzy Systems, vol. 2, pp. 1334–1339, 1993.
[4] , “Learning gait patterns for the fuzzy synthesis of biped walking,” in Proc. Int. Joint Conf. NAFIPS, IFIS and NASA, 1994, pp. 248–250. [5] , “Evolutionary-based learning applied to fuzzy controllers,” Proc.
IEEE Int. Conf. Fuzzy Systems, vol. 3, no. 4, pp. 1111–1118, 1995. [6] C. Zhou and K. Jagannathan, “Adaptive network based fuzzy control of a
dynamic biped walking robot,” Proc. IEEE Int. Joint Symp. Intelligence and Systems, pp. 109–116, 1996.
[7] J.-G. Juang and C.-S. Lin, “Gait synthesis of a biped robot using back-propagation through time algorithm,” Proc. IEEE Int. Conf. Neural Net-works, vol. 3, pp. 1710–1715, 1996.
[8] J.-G. Juang, “Intelligent path training of a five-link walking robot,” Proc. IEEE Int. Symp. Intelligent Control, pp. 1–6, 1996.
[9] P. Werbos, “Backpropagation through time: What it does and how to do it,” Proc. IEEE, vol. 78, pp. 1550–1560, Oct. 1990.
[10] J. Furusho and M. Masubuchi, “Control of a dynamical biped locomo-tion system for steady walking,” J. Dyn. Syst., Meas., Contr., vol. 108, pp. 111–118, June 1986.
[11] Y. F. Zheng and H. Hemami, “Mathematical modeling of a robot colli-sion with its environment,” J. Robot. Syst., vol. 2, no. 3, pp. 289–307, 1985.
[12] M. Raibert, S. Tzafestas, and C. Tzafestas, “Comparative simulation study of three control techniques applied to a biped robot,” Proc. IEEE Int. Conf. Systems, Man and Cybernetics, vol. 1, pp. 494–502, 1993.
[13] L. A. Pars, A Treatise on Analytical Dynamics. London, U.K.: Heine-mann, 1965, ch. 14.
[14] T. Takagi and M. Sugeno, “Fuzzy identification of systems and its ap-plications to modeling and control,” IEEE Trans. Syst., Man, Cybern., vol. SMC-15, no. 1, pp. 116–132, 1985.
[15] J. R. Jang, “Self-learning fuzzy controllers based on temporal back prop-agation,” IEEE Trans. Neural Networks, vol. 3, pp. 714–723, Sept. 1992. [16] S. Horikawa, T. Furuhashi, and Y. Uchikawa, “On fuzzy modeling using fuzzy neural networks with the back-propagation algorithm,” IEEE Trans. Neural Networks, vol. 3, pp. 801–806, Sept. 1992. [17] S. Horikawa, T. Furuhashi, S. Okuma, and Y. Uchikawa, “Composition
methods of fuzzy neural networks,” Proc. IEEE Int. Conf. Industrial Electronics, Control and Instrumentation, pp. 1253–1258, 1990. [18] S. Horikawa, T. Furuhashi, S. Okuma, and Y. Uchikawa, “A fuzzy
con-troller using a neural network and its capability to learn experts control rules,” in Proc. Int. Conf. Fuzzy Logic and Neural Networks, 1990, pp. 103–106.
[19] C. T. Chen, Linear System Theory and Design. New York: Rinehart, 1990.
[20] J. E. Slotine and W. Li, Applied Nonlinear Control. Englewood Cliffs, NJ: Prentice-Hall, 1991.
[21] K. Tanaka and M. Sugeno, “Stability analysis and design of fuzzy control systems,” Fuzzy Sets Syst., vol. 45, no. 2, pp. 135–156, 1992.
Constructing Hysteretic Memory in Neural Networks Jyh-Da Wei and Chuen-Tsai Sun
Abstract—Hysteresis is a unique type of dynamic, which contains an
im-portant property, rate-independent memory. In addition to other memory-related studies such as time delay neural networks, recurrent networks, and reinforcement learning, rate-independent memory deserves further atten-tion owing to its potential applicaatten-tions. In this work, we attempt to define hysteretic memory (rate-independent memory) and examine whether or not it could be modeled in neural networks. Our analysis results demon-strate that other memory-related mechanisms are not hysteresis systems. A novel neural cell, referred to herein as the propulsive neural unit, is then proposed. The proposed cell is based on a notion related the submemory pool, which accumulates the stimulus and ultimately assists neural net-works to achieve model hysteresis. In addition to training by backpropaga-tion, a combination of such cells can simulate given hysteresis trajectories.
Index Terms—Hysteresis, hysteretic memory, rate independence,
recur-rent network, reinforcement learning, time delay neural network.
I. INTRODUCTION
In addition to its simple and practical nature, network computation is adaptive. Network approximation is thus used in many control sys-tems [31], [52]. Neural networks with a short-term memory, occasion-ally referred to as belonging to dynamic systems, are also highly at-tractive for processing time varying signals [32], [56]. In this work, we study another type of dynamic, i.e., hysteresis, which is a unique case of memory. Whether or not neural networks can model hysteresis behaviors is discussed. Network computation can be more fully imple-mented to achieve a better performance from those systems that are
Manuscript received June 25, 1998; revised June 20, 1999 and March 22, 2000. This work was supported in part by the National Science Council, R.O.C., under Contract NSC87-2213-E009-004. This paper was recommended by As-sociate Editor W. Pedrycz.
The authors are with the Institute of Computer and Information Science, National Chiao Tung University, Hsin-Chu, Taiwan 300, R.O.C. (e-mail: [email protected]).
Publisher Item Identifier S 1083-4419(00)06722-4. 1083–4419/00$10.00 © 2000 IEEE
Fig. 1. Typical branches of hysteresis. The outcome of hysteresis is not only based on the current input value, but is also related to the previous history.
Fig. 2. Hysteresis as a rate-independent memory effect (RIME), where (a) and (b) plot the two different inputs, which have the same successive extreme inputs. These input plots result in the samef -u effect diagram of hysteresis.
hysteresis embedded if neural networks could be constructed as hys-teresis simulators.
Hysteresis is a memory effect, with its literal meaning implying to “lag behind.” Fig. 1 depicts the typical shape of a hysteresis diagram. This figure indicates the characteristic behavior of a hysteresis system, a lag in evocation, and perseverance in recovery. When the input value alternates between increasing and decreasing, the response curve does not continue to follow the original path; instead, it draws a new effect delayed curve. In other words, the outcome of a hysteretic mechanism is not only based on the current input value, but is also related to the previous history. Therefore, hysteresis behavior can be easily distin-guished from conventional static mathematical functions.
Hysteresis also heavily concentrates on an important property, rate
independence. Generally speaking, in the discussion of hysteresis
non-linearity, only the previous extreme input values determine the hys-teresis branches. The speed of input variations is not an influential factor. Fig. 2 illustrates this property. Fig. 2(a) and (b) plot two dif-ferent inputs,u1(t) and u2(t), whose successive extreme values are the same. These inputs initially rise top, fall to q, and are then followed by values such asr; s; t. Let the output of the hysteresis system be f(u). Fig. 2(c) summarizes the results in the samef-u diagram. This prop-erty is known as rate independence. Restated, hysteresis is considered as the rate-independent memory effect (RIME) [47].
As an extreme example of rate independence, we track the output in response to input sequencef1; 2; 5; 8; 8; 8; 8; 8; 9; 12; . . .g. Imagine that the input sequence is adopted in a hysteresis system. The response is varied with 1, 2, 5, 8 sequentially input, and remains on the same measure during the run of unchanged 8’s. To emphasize the difference from other memory-related schemes, we first consider the time delay
neural network (TDNN) [25], [51]. A TDNN uses delay kernels with weight factors to ensure short-term memory. Therefore, the above in-puts cause a TDNN to alter its output even with the same input values, i.e., unless the memory depth is full. According to an earlier investiga-tion on recurrent networks [14], [30], [53], [54], the response cannot remain on the same value either, because the feedback requires some iteration to converge (if at all possible). From another aspect, delayed rewards in reinforcement learning (RL) may be considered as a case of memory. Herein, we focus primarily on delayed rewards in RL by employing methods such as temporal difference learning [TD()] [39] andQ-learning [49]. The notion applied in TD() is credit assignment toward temporal patterns, whileQ-learning is based on Markov chains. These two concepts are both related to the time factor and, thus, differ from hysteresis.
Consider another example, one involving the concept of rate independence mentioned earlier. Assume not only that the sequence
f1; 2; 5; 30; 8; 9; 12; . . .g is the input, but also that the extreme value
30 is not attributed to noise. According to our results, the value 30 impacts the hysteresis system for an extended period. Notably, presenting the value 30 changes the path in effect diagram [f-u diagram in Fig. 2(c)]. Such an influence can be eliminated only after the coming input is less than a certain scale, e.g., 0 or030 in some cases. Our results further indicate that for the same input sent into some memory-related schemes (other than the hysteresis system): a) time plays an important role and b) a certain long time or a sufficiently long sequence of common inputs can neglect a circumstance in which the value 30 occurred. In summing up these two examples, hysteresis is characterized by rate-independent memory (also referred to as “hysteretic memory”). Regardless of how slow or fast the input values appear, only the previous extreme values determine the response, even if the inputs remain for a long time or merely flash at once. This prop-erty makes hysteresis markedly different from other memory-related studies, thereby meriting its thorough investigation.
The rest of this paper is organized as follows. Section II reviews hysteresis related literature, indicating its prominence in diverse fields. This section also briefly introduces the Preisach model, which is the most instructive and important model in study of hysteresis. Section III defines the rate-independent memory by formulation. Section IV de-scribes the structure of our approach to modeling hysteresis and derives a backpropagation procedure to train this model. Concluding remarks and areas for future research are finally made in Section V.
II. HYSTERESISRELATEDLITERATURE
A. Review of Related Studies
The term hysteresis originates from ancient Greek and is first used while describing ferromagnetism [see Visintin ([47], p. 9)]. In fact, many useful models have emerged from this domain [17]–[19]. Another cradle for early hysteresis models to develop is plasticity. After Tresca introduced the maximum shear stress yield criterion in 1864, successive investigations increasingly emphasized this criterion [23], [47]. In addition to these two areas, many studies subsequently followed. According to these studies, hysteresis can also be found in various fields, including microelectronics (ferroelectricity [15], [27]), thermodynamics (thermostat [6], [8], thermal relaxation [9]), and in some recently developed materials (the shape memory effects [1], [2], [28]) and mechanics [5], [44].
Besides physical engineering, this phenomenon also occurs in cognitive engineering. Hysteresis can be observed in spatiotemporal pattern recognition (nonstationary noise clearing [21], [42]), time varying signal processing (phase transitions [4]), and cybernetics (control of plants [29], [41]). Moreover, it is becoming increasingly important in the fields of psychology (long-term memory and painful
Fig. 3. Preliminary hysteresis operators in the Preisach model. The Preisach model of hysteresis is based on the combination of^ operators.
experience [38]) and economics [10], [11], [45], [48]. There are still some unresolved questions in all of the above fields, and rate-inde-pendent memory can be a hint. Why is the wage rigidity [7], [22] considered in the Keynesian model? Does the stock market rise and become different after the indices have risen higher than some level [16]? These are related, to certain extents, to rate-independent memory. As generally known, reversing a situation (or even forgetting it) after it occurs is extremely difficult. Such a situation can persist for a long time, ultimately reducing the influence, not because time has passed but because another new scene (extreme value) occurs.
B. Mathematical Models
As mentioned above, the study of hysteresis begins with ferromag-netism. Pertinent literature regarding ferromagnetic hysteresis, which can be found in [35], indicates that Lord Rayleigh [34] proposed the first model in 1887, and the most important model, the Preisach model, was proposed in 1935 [24], [33].
The Preisach model has received extensive attention [37], [43], [46] since it was published in 1935. This model contains the notion that a complicated system can be constructed as a superposition of sim-plest operators (Fig. 3). The operator^ is the unit of the Preisach model. Footnotes and denote the operating limitation of each op-erator. Each operator works with the current input value,u, and results in bi-valued hysteresis (returns01 or 1) owing to that the ascending paths and the descending paths are switched. While combining opera-tors, such as^ ; ^ , etc., an entire system can be expressed by the following equation:
f(t) =
( ; )[^ u(t)] d d
(1)
where( ; ) is the weight to each 0 operator ^ .
The Preisach model is known as a kind of “nonlocal memory hys-teresis model” [26] since it responds to the outputs while thoroughly considering the previous inputs. Another kind of hysteresis model is called “local memory hysteresis model.” These models generate their next output by consulting the local memory they have produced in the current state. In this case, at most two curves pass through each point in the input-output (I/O) matching diagram, say, thef-u diagram. For
an increasing inputu(t), the rising curve is followed, causing the re-sponsef(u) to rise with u. If the input is decreased, then the falling curve is traced.
The Hystery model [42] is an instance of local memory models. Both the upper and lower branches of a hysteresis loop in this model are described by hyperbolic tangent functions.
III. DEFINITION OFHYSTERESISSYSTEM
In the previous section, we introduced rate-independent memory and briefly mentioned that hysteresis is frequently observed in many fields. However, other memory related mechanisms are not rate-independent and, thus, may not function properly in a hysteresis system. In this sec-tion, we repeat these instances by a formulation. An attempt is initially made to properly define a rate-independent system in terms of digital data processing. In doing so, all of the schemes, such as time delay neural networks, recurrent networks, and some reinforcement learning methods, are distinguished from hysteresis. In the next section, we pro-pose the propulsive model to model the hysteresis behavior in neural networks.
A. Rate-Independent Systems
Initially, we denote a systemT that transduces an input sequence with valuesx[n] into an output sequence with values y[n] as the fol-lowing form:
y[n] = T fx[n]g: (2)
Then, a rate-independent system is defined as a system having the fol-lowing two properties.
Property 1—Ineffective Insertion of Mean Values (IIMV): Consider
an input sequencex = f. . . ; x[k]; x[k + 1]; . . .g and two contiguous input values in it, sayx[k] and x[k + 1]. We insert a mean value xq = p x[k] + (1 0 p)x[k + 1] with 0 p 1 between x[k] and x[k + 1].
In doing so, a new sequencew is obtained as w[i] = x[i] for i
k; w[k + 1] = xq, andw[j] = x[j 0 1] for j > k + 1. Thereafter, x andw are sent to two systems, which are both duplicated from system
T . By assuming that the corresponding outputs are sequences y and z,
we state that the systemT has ineffective insertion of mean values if
z[j] = y[j 0 1] with j > k + 1 for any input sequence x and any
interpolating indexk. (In addition, it is always true that z[i] = y[i] for
i k in any system. The output in response to xq; z[k + 1], is not important here.)
Property 2—Ineffective Removal of Mean Values (IRMV): For an
input sequencex, we select any three contiguous values x[k 0 1]; x[k] andx[k + 1] under the condition x[k] = px[k 0 1] + (1 0 p)x[k + 1] wherep satisfies the condition that 0 p 1. A new sequence w is obtained after we omitx[k] such that w[i] = x[i] for i < k and w[j] =
x[j + 1] for j k. By sending x and w to two duplicate systems both
referring to systemT , let the corresponding output sequences be y and
z. We state that the system T has ineffective removal of mean values if z[j] = y[j + 1] with j k for any input sequence x and any omitting
of indexk. (Also, it is obvious that z[i] = y[i] for i < k in any system.)
Definition 3—Rate-Independent System: A rate-independent system contains both of the above IIMV and IRMV properties.
The inserted mean values could turn to be a subsequence by iter-atively applying the IIMV property on the same interpolating index. Similarly, continuous removed mean values can form a subsequence as the IRMV property is applied repeatedly. That is, insertion (or re-moval) of an input subsequence is also ineffective for a rate-indepen-dent system as long as this subsequence has no extreme value according to the newly generated input sequence.
B. Hysteresis Systems
Consider that a system is memoryless iff the outputy[n] at every value ofn depends only on the input x[n] on the same value of n. That is,y[n] = f(x[n]) for each value of n, where f( ) is one of any func-tions. A memoryless system is surely a rate-independent system. More specifically, a hysteresis system is defined herein as a rate-independent system with a causal memory.
Definition 4—Hysteresis System: A hysteresis system is rate
inde-pendent, causal, and not memoryless.
Restated, a hysteresis system has rate-independent causal memory. It is also referred to as “hysteretic memory.”
A necessary condition for a hysteresis system can be derived from its causality and rate independence.
Lemma 5—Stayed Input Stayed Output (SINSOUT): For a
rate-in-dependent system, an unvaried input subsequence causes an output sub-sequence that also remains unchanged.
Proof: By applying the IIMV property on interpolating indexk
withp = 0, the inserted mean value is xq = x[k + 1], which is also viewed as the unvaried input value. Let us focus on the key I/O subse-quences. The original I/O pair isfx[k]; x[k +1]g and fy[k]; y[k +1]g. By rate independence, the inserted I/O pair isfw[k] = x[k]; w[k+1] =
xq = x[k + 1]; w[k + 2] = x[k + 1]g and fz[k] = y[k]; z[k + 1] = T fxqg; z[k + 2] = y[k + 1]g. Because of causality, T fxqg with xq= x[k + 1] is only based on all the input values x[i] for i k + 1. Therefore,T fxqg = T fx[k + 1]g = y[k + 1]. The unchanged output valuez[k + 2] = y[k + 1] = z[k + 1] is shown here. The proof is completed while we repeatedly apply the IIMV property as above.
C. Multilayer Feedforward Neural Networks versus Hysteresis
The preliminary neural network model is the multilayer feedforward proceptron [52]. The activation of each node,xi, in this preliminary model can be represented as
xi= (neti) + Ii (3)
where
neti= j<i
wijxj (4)
wij denotes the weight factor connecting nodej to node i; ( ) is a squashing function, andIirepresents the bias of nodei. To show the activation of this system in the form of a sequence, (3) and (4) can be rewritten as xi[n] = (neti[n]) + Ii (5) and neti[n] = j<i wijxj[n]: (6)
According to the above equations, this is a memoryless system and obviously not a hysteresis system by definition.
D. Convolution Neural Models versus Hysteresis
A time delay neural network (TDNN) is one of the so-called “convo-lution models” [13]. The activation of the nodes in convo“convo-lution models is improved from (6). Herein, we let
neti[n] = j<i n k=0 wij[n 0 k]xj[k] = j<i n k=0 wij[k]xj[n 0 k]: (7)
For a TDNN, the weight factorswij[k] are only valid in a bounded range, referred to as “memory depth.” Ifk exceeds the memory depth, the weight factor equals zero. For some advanced versions of convo-lution models, e.g., the concentration-in-time net (CITN) [40] and the gamma neural model [13], there is no limitation for memory depth. In the following, we demonstrate that convolution models cannot be hys-teresis systems.
The following discussion focuses on the first node, which operates with memory mechanism as above. This event implies that other nodes earlier than this node are all memoryless. Assume that the node is node
i, and its response on some input x[nt] is concerned with neti[nt] =
j<i n k=0
wij[nt0 k]xj[k]: (8) When the next inputx[nt+ 1] is accepted
neti[nt+ 1] = j<i n +1 k=0 wij[nt+ 1 0 k]xj[k] = j<i n k=0 wij[nt0 k + 1]xj[k] + j<i wij[0]xj[nt+ 1]: (9) The above equations reveal thatneti[nt+1] does not equal neti[nt], al-thoughxj[nt+ 1] = xj[nt] for all j < i. As the signal feeds forward, the output of this network also performs the inequality. This perfor-mance conflicts with the SINSOUT lemma and, therefore, convolution models cannot establish hysteresis systems.
E. Recurrent Networks versus Hysteresis
In contrast to feedforward networks, a latter node in a recurrent net-work may have backlinks connected to earlier ones [31]. Therefore, the signal could feed back to join the computation in the next step, thereby constructing the memory effect. The equation can be expressed as
neti[n] = j<i
wijxj[n] + ji
wijxj[n 0 1]: (10) Among the unresolved issues surrounding, recurrent networks include such questions as “Is it stable, and inner stable?” “Is it convergent?” and “How fast does it converge?”
Next, whether or not it is rate independent is explored.
Consider the node that is initially linked by some latter nodes. As-sume that it is nodei, and one of the latter nodes is node j. Notably, the network structure linked before nodei is feedforward and, thus, memoryless. Assume that the previous input isxc, the current input is
xa; without loss of generality, we denote that the activation of nodej onxc, andxais not equal. Ifxais input again, the feedforward part sends the same signal to nodei, while node j feedback to node i differs from that activated onxcinputs. The inequality occurs and the entire network structure has the starting point to generate a different output.
Again, this performance conflicts with the SINSOUT lemma. We can conclude that recurrent networks cannot construct hysteresis systems.
F. Reinforcement Learning versus Hysteresis
Reinforcement learning is applied to discover how to yield the highest reward, and is characteristic of a trial-and-error and delayed reward. A well-trained mechanism is not discussed herein; in fact, it is usually a memoryless controller. However, our primary concern
is the learning algorithm itself, thus raising the following question: Does it offer rate-independent information during learning? Learning algorithms also impact the system response during the training stage.
Sutton proposed temporal difference learning,TD(), in 1988 [39]. This incremental real-time algorithm adjusts the weights in a connec-tionist network. The learning process follows the form
1wt= (Pt+10 Pt) t k=1
t0kr
wPk (11)
wherePtdenotes the network’s output upon the input patternxtat time
t; w represents the vector of weights that parametrizes the network, and rwPkis the gradient of network output with respect to weights. This method provides a feasible means for supervised learning procedures to solve temporal credit assignment problems. However, it is not rate independent because it may not correspond to the principle of mean value ineffectiveness.
Other reinforcement learning algorithms [50] are not rate indepen-dent either. In fact, some of these algorithms, such asQ-learning [49], function in a discrete finite world, not a numerical world in which hys-teresis has been discussed herein so far. Frequency is what the learning algorithms function to predict, thereby making time a major factor. The learning process operates at every time step by the Markov process. Hysteretic memory is unnecessary for these applications.
Nevertheless, we believe that learning depends, to varying extents, on memory. Exactly what role hysteretic memory may operate in rein-forcement learning is explored later on.
IV. HYSTERETICMEMORY VIA THEPROPULSIVENEURALUNIT The previous section has defined hysteresis system. According to this definition, a system is observably not hysteresis if it is sensitive about input speed. Conventional network strategies, either in refer-ence to unselected and limited historic inputs or using fixed recurrent links, are closely associated with data input rate. In sum, networks based on computational nodes and links cannot purely function as a hysteresis simulator. In the following, we present a propulsive neural cell to assist hysteresis simulation. This neural cell is also trained by backpropagation.
A. Using the Delta as the Input
As far as the SINSOUT lemma is concerned, we speculate that the difference between two contiguous inputs may be the essence of the hysteresis system. This difference is referred to herein as the “delta.” A circumstance in which the network accepts the delta instead of the real input and only action at nonzero delta input could avert the opposite situation to the SINSOUT lemma.
Although consulting with the delta is preferred, directly using the delta for computation is not an appropriate design. Obviously, a
delta-in-delta-out network, which computes the movement of output
instead of the real response value, does not fully consist of the IIMV property. Alternatively, some mechanisms may need to be designed in a neural unit and the delta adopted as well to modify its state. Then, allow the response to refer to the state of this mechanism. Restated, hysteretic memory thus resides in the inner state of a neuron.
B. Propulsive Neural Unit
This study constructs hysteretic memory in a neural unit based on a hypothesis involving the accumulation of the stimulus. Assume herein that a neuron accepts delta signals and pushes them into a submemory
pool. The process of push is named “propulsion” or “propulsive process.” By propulsion, if all the deltas are positive, combining all
of the deltas becomes a longer and deeper potency carrier. Otherwise,
Fig. 4. Propulsive neural models accessing delta inputs. (a) Current state of the neural input, wherek is a variable indicating the number of record couples. (b) Propulsion process skips a carried interval and does not stop until the increased delta is exactly pushed. (c) Decreased delta performs the same algorithm while empty intervals are skipped now. Variablek decreases in (b) and (c). such a combination would be with some empty areas, and indicates a weaker carrier.
For simplicity, this study implements the submemory as an axis,X, which is occupied by carried intervals and empty intervals. Allow the total length of carried intervals to equal the current input. The first car-ried interval thus occurs for the first signal inputs (assuming it is pos-itive). Initially, it starts from zero and ends on the value of the first input because the first delta equals the first input. This carried interval sequentially grows during the run of positive delta inputs. Before it stops, it actually occupies a length of size equal to the first extreme input value. Then, a negative delta inputs and uses its absolute value to generate an empty interval also from zero. After the duration of neg-ative delta inputs, the first carried interval turns from the right end of the first empty interval. Thereafter, the second carried interval grows, and so on. In doing so, any mean value inputs do not alter the state of carried intervals and, thus, could not impact the responding outputs. Regarding the neuron outputs, the response to the current input,u(t), can be expressed by integrating a “reaction function,”D(x), on the union of all carried intervals,At. It is positive and written as
f(t) =
A D(x) dx jA = fcarried intervalsgon u(t): (12) Fig. 4 illustrates how this model works. Two sequences,M’s and
m’s, are made to record the distribution of intervals. The extreme
rising signals are finally recorded by the M’s, while the extreme falling signals are recorded by them’s. Both M and m sequences are in a monotonically descending order, appearing in couples
(Mi; mi)’s with Mi > mi except in the initial condition when
M1 = m1 = 0. Restated, these two sequences concern themselves withM1> m1> M2> m2> M3> . . .. Actually, the jth carried interval is recorded by [mj; Mj], while the jth empty interval is recorded by[Mj+1; mj] or [0; mj]. Therefore, by allowing k to be the total number of record couples, the following equation is valid:
u(t) = k j=1
(Mj0 mj): (13)
When an increasing signal arrives, the present recordMkis gener-ally shifted from a lower position to a higher one. If the shift meets a carried interval[mj; Mj]; Mjreplaces the current record. Restated, this stage must be skipped, and proceed to its right end to continue the propulsive process. The model continues to perform this process until the increased delta is exactly pushed into the carried interval. While a decreasing signal comes, we shiftmkin the same manner except that the skipped stages are now[Mj+1; mj]’s.
Fig. 5. System established on a basic PNU organization. (a) We treat each PNU as the basic unit in the structure. (b) Picture zooms in on thepth PNU, which has a sensible range[L ; H ]. (c) Each PNU functions similarly to a one-layered perceptron by selecting a step function as the reaction function.
Consider the following example of tracing a seriesf25; 15; 20; 18g. The delta sequence is thus f25; 010; +5; 02g, and the M and
m’s arise as (M1; m1) = (25; 0) when the value 25 inputs, (M1; m1) = (25; 10) when the value 15 inputs. When the value 20 in-puts, it turns out to be(M1; m1; M2; m2) = (25; 10; 5; 0). Finally, it is(M1; m1; M2; m2) = (25; 10; 5; 2) when the value 18 inputs. Now, allowing a new signal to arrive as 24, a newM2must occur by propul-sion and the records list becomes(M1; m1; M2; m2) = (25; 10; 9; 0). To summarize, this model allows us to give the response formula again: Input: u(t) = k j=1 (Mj0 mj) Output: f(t) = A D(x) dx; where At= k j=1 [mj; Mj]: (14) C. System Structure
A propulsive neural unit (PNU) is a neuron that accepts the delta and operate propulsive process as designed above. In this section, PNUs are used to construct a simple structure. Before doing so, two questions must be addressed: “How can the PNUs be organized?” and “What is chosen to be the reaction function,D(x)?”
The sensible range for a PNU is initially provided. According to this term, a neuron has its range to accept a signal. Too large or too small a signal must be filtered into a value between a higher bound and a lower one. A PNU operates like the plot in Fig. 2(c) when it accepts a signal with value in the sensible range. An overflow signal makes the submemory pool full, while a deficient signal makes it empty. These two cases cause the PNU to respond to constant values. Cumulatively, a PNU takes the filtered value, counts the delta and integrates the re-sponse after propulsion. Now, (13) and (14) must be adjusted with an offset of the lower bound.
As mentioned earlier, each PNU has its own sensible (working) range and its own effective trajectory (of the shape like Fig. 2(c) shown). Collecting some PNUs is an effective means of simulating various types of hysteresis. Thus, as expected, a network can model hysteresis behaviors if it contains some nodes built as PNUs with distinct sensible ranges. For simplicity, the proposed system is basic and only organized into several parallel PNUs. Fig. 5(a) reveals that
the input signal is simply sent to allL PNUs and the response to the summation of their output is obtained as well. Fig. 5(b) displays the
pth PNU, whose sensible range is [Lp; Hp].
Regarding the decision of the reaction function, we select step func-tions for convenience, although their parameters are numerous. A step functionDp(x) is taken onto the pth PNU. The value of the ith scale, Dp;i, is an adaptable parameter. The collaborative inputrtp;iis calcu-lated as the size of a conjunction of theith stage and the current carried intervals, where the indext is responding to the input utat timet; p denotes that it belongs to thepth PNU and i represents that it should be integrated in theith stage of the step function Dp(x). That is, the effect corresponding to theith stage is Dp;i1rp;it . Therefore, each PNU functions similarly to a one-layered perceptron. If there are a total of
n stages in each [Lp; Hp], the response value of input ut can be ex-pressed as f(ut) = L p=1 n i=1 Dp;i1 rtp;i: (15) The series inpututmust be diffused to each PNU, filtered into the sensible range, changed into the delta value, through the propulsive process and, finally, divided into each respective inputrp;it . The total operation design herein to construct hysteretic memory is termed the
propulsive model. D. Learning Method
The operation phase and the learning phase are separately discussed in the propulsive model. The operative phase of this model has been presented above. Herein, an adaptive method is proposed to determine the system parameters.
The learning phase of the PNU organization adopts the Widrow-Hoff back propagation rule. This rule is based on an iterative gradient de-scent algorithm designed to minimize the mse between the desired target values and the actual response values.
Although (15) has demonstrated how to calculate the response value with respect to the local inputut, a slight transformation must occur on reaction function,Dp(x), to denote each step and complete the adap-tation. Let us take
f(ut) = L p=1 n i=1 Dt p;i1 rtp;i (16)
Fig. 6. Training the PNU organization to follow the loops given by (20) with( ; ) = (4:5=25 ) 0 (3=25 ) 0 (1=25 ) . (a) Target diagram, with 111 points on it. The continuous four figures use four PNUs. (b) Initial trajectory versus target diagram. (c) After one epoch, the mse has the value
3:59 2 10 . (d) After two epochs, MSE = 2:18 2 10 . (e) After 1528
epochs,MSE = 2:82 2 10 . The remaining figure uses six PNUs. (f) After 814 epochs,MSE = 1 2 10 .
and represent the target value asG(ut). The error is expressed as the following equation:
E = 12 N t=1
(G(ut) 0 f(ut))2 (17) whereN denotes that there are total N patterns.
The adaptation rule is formulated as
Dt+1
p;i = Dtp;i+ 1Dtp;i: (18) Thus, it is derived that
1Dt
p;i= 0 @E@Dt p;i
= (G(ut) 0 f(ut)) 1 f(u@Dtt) p;i = (G(ut) 0 f(ut)) 1 L q=1 nj=1rtq;j1 Dq;jt @Dt p;i
= (G(ut) 0 f(ut)) 1 rp;it (19) where the positive value is the learning rate.
Fig. 7. Training the PNU organization to follow the loops given by (20) in a nonlinear form( ; ) = (2=25 ) + (6=25 ) . (a) Target diagram, with 111 points on it. The following figures are trained after 1000 epochs. According to our results, the more PNUs used implies a better performance that it converges. (b) Four PNUs used,MSE = 8:73 2 10 . (c) Six PNUs used,MSE = 2:33 2 10 . (d) Eight PNUs used, MSE = 1:93 2 10 .
Equation (19) can be implemented as an iterative procedure to adapt the system parameters.
E. Adaptation to Given Behavior
As generally known, trajectory traversal is the foundation of system modeling. In this section, the PNU organizational ability is demonstrated to learn some loops. Desired tracks are generated by the Preisach model (see Section II). Initially, a system with four PNUs is trained to follow these trajectory samples. These four PNUs are bounded with distinct sensible ranges as[0; 25]; [5; 20]; [3; 13], and
[12; 22]. In addition, all stage lengths of their reaction functions,
which are step functions, are set to be one equally. To normalize the response value in[0; 1], we merely assign average weight parameters
(1=41(2500)); (1=41(2005)); (1=41(1303)), and (1=41(22012))
to each stage of these four PNUs as the initial parameters. In doing so, the adaptation can be clearly observed.
Equation (1) displays how the Preisach model functions. However, an adjustment must be made such that whole positive trajectories can be generated for the PNU organization to simulate. Regarding (1)
f(t) =
( ; )[^ u(t)] d d
we redefine the target diagram by having the bi-valued operator^ return 0 instead of01 while the descending paths are followed. Assume that this redefined operator is^ . The target diagram now follows the equation
f(t) =
( ; )[^ u(t)] d d :
(20) Next, consider the alternating seriesf0; 25; 3; 21; 7; 18; 9; 16; 12g to perform the target trajectories. All distances are set to “1” sequentially between these extreme values. Therefore, there are cumulatively 111 points on the target sequences. In addition, allow the learning rate to be 0.015. The learning rate is set quite small with respect to the good
Fig. 8. Hysteresis embedded into a fuzzy membership function. We can take a hysteresis-embedded input instead for alternatively considering several membership functions during a memory-involved inference.
deal of set as weights in such a single layer. Later on, more experiments are performed as well in an attempt to accelerate the learning process. Under the page constraint, only two experiments performed herein demonstrate the PNU organizational ability of adaptation. Fig. 6 demonstrates the learning capability of the trajectory with
( ; ) = (4:5=252) 0 (3=253) 0 (1=253) in (20). The adaptive performance appears acceptable. This same figure indicates that using more PNUs carried facilitates a better performance. While two more PNUs are increased with sensible ranges of[17; 23]; [19; 22], it converges faster and better, as indicated by Fig. 6(f). Fig. 7 reveals a similar observation when we trace nonlinear parameters,
( ; ) = (2=254) 2+ (6=254) 2. Fig. 7(b) is traced by the original four PNUs, system in Fig. 7(c) is joined with additional two units with sensible ranges of [17; 23]; [19; 22]. In addition, the system in Fig. 7(d) adopts two more units bounded in[7; 15]; [11; 16].
V. CONCLUSIONS
Four memory-related topics are frequently studied in engineering: memory kernels in convolution neural models [13], recursive efforts of recurrent networks [31], [52], delayed rewards in reinforcement learning [50], and rate-independent memory effect such as hysteresis. In this study, we closely examine the final one.
While focusing mainly on defining hysteretic memory, this study discusses whether or not network computation can be applied to hys-teresis modeling. Analysis results indicate that networks with purely computational nodes and links cannot function as hysteresis simula-tors. A propulsive neural model is also proposed to construct hysteretic memory. The proposed model is based on a neural unit, the neuron. In addition to propulsive operation, a single neuron functions similarly to a one-layered proceptron. In addition, several propulsive neural units with distinct sensible ranges are used to organize a simple system. Fi-nally, a learning method based on backpropagation is designed so that the system can automatically adapt its parameters.
Based on the results presented herein, the areas for future research are made as follows.
A. Hysteretic Memory-Related Applications
Hysteresis, although it is a memory effect whose output must con-sult with historic inputs, responds to the tendence in real time. A (local) highest value cannot be input and expected to receive the co-relative output after some time when the input value is decreasing. Restated, hysteretic memory would not incur aftereffects and delayed reactions.
On the other hand, a mechanism that is used to model delayed reactions may not adequately process hysteretic memory. In fact, the varied input speed is actually a challenge for all of those mechanisms, while it does not influence a hysteresis system owing to rate independence. There-fore, developing a feasible means of combining hysteretic memory with conventional time varying signal processors is of worthwhile interest.
Embedding hysteretic memory into a fuzzy membership descrip-tion is another applicadescrip-tion. The fuzzy inference system must represent the membership of each condition before a decision can be reached. However, a static membership function may not function properly for a memory-involved inference. Alternatively considering several mem-bership functions is an effective means of resolving this problem. A hysteresis embedded input can be developed instead, after we have de-veloped a hysteresis model, as indicated by Fig. 8.
B. Improvement of System Performance
This study has presented a supervised learning method to adjust the system parameters. A future work should focus on how to self-orga-nize the system structure so that the system identification task is com-pleted. The genetic algorithm (GA) is a viable means of generating the structure.
Another point worth mentioning is that hysteresis does not only occur in one feature in a one-dimensional model. Hysteretic memory may appear in several coupled features with correlation. An interesting topic would be how we can more fully elucidate propulsion across these dimensions.
Particular attention should also be paid to the reaction function. This function is closely related to how the neuron operates and is adapted. We believe that the system performance could be markedly improved if a new reaction could be designed to function well. By doing so, con-struction of hysteretic memory would not be so complicated.
Notably, network approximation in other studies [3], [36] has func-tioned as a cooperator with particular hysteresis models. They also per-form well in the aspect of engineering. In contrast, this work deals with hysteresis behavior in a unique manner. Results presented herein allow us to construct hysteretic memory in neural networks. This is also a novel means of imaging how a system generates hysteresis behaviors since the system is always a black box itself.
REFERENCES
[1] K. Bhattacharya, R. D. James, and P. J. Swart, “Relaxation in shape-memory alloys. i. Mechanical model,” Acta. Mater., vol. 45, pp. 4547–4560, Nov. 1997.
[2] , “Relaxation in shape-memory alloys. ii. Thermo-mechanical model and proposed experiments,” Acta. Mater., vol. 45, pp. 4561–4568, Nov. 1997.
[3] S. Bobbio, G. Miano, C. Serpico, and C. Visone, “Models of magnetic hysteresis based on play and stop hysterons,” IEEE Trans. Magn., vol. 33, pp. 4417–4426, Nov. 1997.
[4] M. Brokate, N. Kenmochi, I. Müller, J. F. Rodriguez, and C. Verdi, Phase Transitions and Hysteresis. New York: Springer-Verlag, 1993. [5] J. Cacko, “Simulation experiments in investigation of fatigue
degrada-tion effects of dynamic processes,” in Eur. Simuladegrada-tion Multiconf. 1995, 1995, pp. 92–96.
[6] H. B. Callen, Thermodynamics and an Introduction to Thermo-statics. New York: Wiley, 1985.
[7] C. M. I. Campbell and K. S. Kamlani, “The reasons for wage rigidity: Evidence from a survey of firms,” Q. J. Econ., pp. 759–789, Aug. 1997. [8] S. Can, M. S. Richards, and R. A. Pease, “A 3 v thermostat circuit,” in Proc. 1996 BIPOLAR/BiCMOS Circuits Technol. Meeting, 1996, pp. 105–108.
[9] C. R. Chang and I. Klik, “Thermal relaxation in magnetic materials,” Proc. Natl. Sci. Counc. Rep. China A, Phys. Sci. Eng., vol. 20, pp. 249–264, May 1996.
[10] B. J. Chapman, “Long-term unemployment: The dimensions of the problem,” Aust. Econ. Review (1993 Second Quarter), pp. 22–25, 1993.
[11] S. E. Christophe, “Hysteresis and the value of the U.S. multinational corporation,” J. Bus., pp. 435–462, July 1997.
[12] J. M. Cruz-Hernandez and V. Hayward, “On the linear compensation of hysteresis,” in Proc. 36th IEEE Conf. Decision Contr., vol. 2, 1997, pp. 1956–1957.
[13] B. de Vries and J. C. Principe, “The Gamma model—A new neural model for temporal processing,” Neural Networks, vol. 5, pp. 565–576, 1992.
[14] F. fur Informatik, “Long short-term memory,” Neural Comput., vol. 9, pp. 1735–1780, Nov. 1997.
[15] F. P. Gnadinger, “Ferroelectrics for nonvolatile RAMs,” IEEE Spectrum Mag., pp. 30–33, July 1989.
[16] C. W. J. Granger and Z. Ding, “Varieties of long memory models,” J. Econometrics, pp. 61–77, July 1996.
[17] D. C. Jiles, “Frequency dependence of hysteresis curves in conducting magnetic materials,” J. Appl. Phys., vol. 76, pp. 5849–5855, Nov. 1994. [18] , “Modeling the effects of eddy current losses on frequency depen-dent hysteresis in electrically conducting media,” IEEE Trans. Magn., vol. 30, pp. 4326–4328, Nov. 1994.
[19] D. C. Jiles and D. L. Atherton, “Theory of ferromagnetic hysteresis (in-vited),” J. Appl. Phys., vol. 55, pp. 2115–2120, Mar. 1984.
[20] K. Jin’no and T. Saito, “Analysis and synthesis of continuous-time hys-teretic neural networks,” Electron. Commun. Jpn., pt. 3, vol. 76, no. 4, 1993.
[21] J. F. Jøgensen, K. Carneiro, and L. L. Madsen, “Hysteresis correction of scanning tunneling microscope images,” J. Vac. Sci. Technol. B., vol. 12, pp. 1702–1704, May/June 1994.
[22] M. Kandil, “Sticky wage or sticky price? Analysis of the cyclical be-havior of the real wage,” Southern Econ. J., pp. 440–459, Oct. 1996. [23] C. Koiter, “General theorems for elastic-plastic solids,” in Progress
in Solid Mechanics, I. N. Sneddon and R. Hill, Eds, Amsterdam, The Netherlands: North-Holland, 1960, vol. I, pp. 165–221.
[24] M. Krasnoselskii and A. Pokrovskii, Systems with Hysteresis, Moscow, Russia: Nauka, 1983.
[25] D. T. Lin, J. E. Dayhoff, and P. A. Ligomenides, “Trajectory production with the adaptive time-delay neural network,” Neural Networks, vol. 8, no. 3, pp. 447–461, 1995.
[26] I. D. Mayergoyz, Mathematical Models of Hysteresis. New York: Springer-Verlag, 1991.
[27] J. Millman and A. Grabel, Microelectronics, 2nd ed. New York: Mc-Graw-Hill, 1987.
[28] Y. Mualem, Theory of Universal Hysteretical Properties of Unsaturated Porous Media, Water Resour. Res. Public., 1979.
[29] I. Nagy and Z. Suto, “Repetitive and chaotic processes in current con-trolled induction motor,” in ISIE’96. Proc. IEEE Int. Symp. Ind. Elec-tron., vol. 2, 1996, pp. 946–951.
[30] D. Obradovic, “On-line training of recurrent neural networks with con-tinuous topology adaptation,” IEEE Trans. Neural Networks, vol. 7, pp. 222–228, Jan. 1996.
[31] O. Omidvar and D. L. Elliott, Neural Systems for Control. New York: Academic, 1997.
[32] V. Petridis and A. Kehagias, “A recurrent network implementation of time series classification,” Neural Comput., pp. 357–372, Feb. 1996. [33] F. Preisach, “Über die magnetische nachwirkung,” Z. Physik, vol. 94,
pp. 277–302, 1935.
[34] L. Rayleigh, “On the behavior of iron and steel under the operation of feeble magnetic forces,” Philos. Mag., vol. 1887, pp. 225–248, 1887. [35] G. Rieder, “Elastic obstacles for bloch walls,” in Models of Hysteresis,
A. Visintin, Ed. Harlow, London, U.K.: Longman, 1993, pp. 143–157. [36] C. Serpico and C. Visone, “Magnetic hysteresis modeling via feed-for-ward neural networks,” IEEE Trans. Magn., vol. 34, no. 3, pp. 623–628, May 1998.
[37] R. C. Smith, “Hysteresis modeling in magnetostrictive materials via Preisach operators,” Inst. Comput. Applicat. Sci. Eng., Tech. Rep. TR-97-23, May 1997.
[38] N. L. Stein, P. A. Ornstein, B. Tversky, and C. Brainerd, Memory for Everyday and Emotional Events: Lawrence Erlbaum Associates, 1997. [39] R. S. Sutton, “Learning to predict by the methods of temporal
differ-ences,” Mach. Learn., pp. 9–44, 1988.
[40] D. W. Tank and J. J. Hopfield, “Concentrating information in time: Analog neural networks with applications to speech recognition problems,” IEEE Trans. Acoust., Speech, Signal Processing, vol. 37, pp. 328–339, Mar. 1989.
[41] G. Tao and P. V. Kokotovic, “Adaptive control of plants with unknown hysteresis,” IEEE Trans. Automat. Contr., vol. 40, pp. 200–212, Feb. 1995.
[42] M. D. Tom and M. F. Tenorio, “A neural computation model with short-term memory,” IEEE Trans. Neural Networks, vol. 6, pp. 387–397, Mar. 1995.
[43] E. D. Toore and F. Vajda, “Parameter identification of the complete-moving hysteresis model using major loop data,” IEEE Trans. Magn., vol. 30, pp. 4987–5000, Nov. 1994.
[44] C. Truesdell and W. Noll, The Nonlinear Field Theories of Mechanics. In Encyclopedia of Physics, Heidelberg, Germany: Springer-Verlag, 1965, vol. III/3.
[45] M. Uribe, “Hysteresis in a simple model of currency substitution,” J. Monetary Econ., pp. 185–202, Sept. 1997.
[46] F. Vajda and E. D. Torre, “Remanence loop asymmetry and moving-type Preisach models,” IEEE Trans. Magn., vol. 30, pp. 3658–3660, Nov. 1994.
[47] A. Visintin, Differential Models of Hysteresis. New York: Springer-Verlag, 1994.
[48] S. Wadhwa, R. Caprihan, and S. Kumar, “Performance of a hysteresis based control strategy for a flexible machine operating under a periodic status monitoring policy,” Comput. Ind. Eng., pp. 557–574, July 1997. [49] C. J. Watkins and P. Dayan, “Technical note:Q-learning,” Mach. Learn.,
vol. 8, pp. 9–44, May 1992.
[50] P. J. Werbos, “A menu of designs for reinforcement learning over time,” in Neural Networks for Control, W. T. Miller, R. S. Sutton, and P. J. Werbos, Eds. Cambridge, MA: MIT Press, 1990, ch. 3.
[51] , “Backpropagation through time: What it does and how to do it,” Proc. IEEE, vol. 78, pp. 1550–1560, Oct. 1990.
[52] , The Roots of Backpropagation. New York: Wiley-Interscience, 1994.
[53] R. J. Williams, “Adaptive state representation and estimation using re-current connectionist networks,” in Neural Networks for Control, W. T. Miller, R. S. Sutton, and P. J. Werbos, Eds. Cambridge, MA: MIT Press, 1990, ch. 4.
[54] L. Wu and J. Moody, “A smoothing regularizer for feedforward and re-current neural networks,” Neural Comput., vol. 8, pp. 461–489, Apr. 1996.
[55] H. Yanai and Y. Sawada, “Associative memory network composed of neurons with hysteretic property,” Neural Networks, vol. 3, pp. 223–228, 1990.
[56] M. A. Zohdy, M. Karam, and H. S. A. A. Zohdy, “A recurrent dynamic neural network for noisy signal representation,” Neurocomputing, vol. 17, pp. 77–97, Oct. 1997.
