A Neuromorphic Chip That Imitates the ON Brisk
Transient Ganglion Cell Set in the Retinas of Rabbits
Li-Ju Lin, Student Member, IEEE, Chung-Yu Wu, Fellow, IEEE, Botond Roska, Frank Werblin, David Bálya, and
Támas Roska, Fellow, IEEE
Abstract—In this paper, a new CMOS design methodology is proposed to implement CMOS neuromorphic chips which imitate the ON brisk transient ganglion cell (GC) set of rabbits’ retinas. The measurement results on the fabricated CMOS neuromorphic chip are consistent with the biological measurement results. Thus, the biological functions of the chip have been successfully verified. It can be used to understand more biological behaviors and visual language of retinas under different input optical images which have not yet been tested in biological experiments. Based on the results, the full GC sets of retina can be designed. Thus, many potential applications of retinal chips on motion sensors, computer vision, retinal prosthesis, and biomedical devices are feasible.
Index Terms—Neuromorphic sensor, retina, retinal processing circuit, smart vision sensor.
I. INTRODUCTION
T
HE RETINA IS a unique organ common to all organisms with the faculty of sight and is used to communicate with the visual world. It has superior performance in such areas as visual perception, image detection, and preprocessing: for example, in covering a wide range of light intensities; in responding to very low levels of contrast ( 1%); and in integrating for short periods of times ( 0.04 s) [23]. As a conse-quence, much research has been devoted to implementations of retinal functions on silicon chips, such as the partial functions of photoreceptors and horizontal cells with part knowledge of bipolar cells [1]–[4]. Various kinds of applications have also been developed using the designed chips, such as motion de-tection sensors [5]–[16], and high-performance image sensors [17]–[21]. Neuroscientists see the retina as a part of the brain that develops within the eye to provide neural processing for photoreceptor signals [23]. With the latest advances in neuro-science, it has been found that the mammalian visual system analyzes the world through a set of separate spatiotemporal channels [23], [26]. These channels are constructed by different retinal cells, later by different cells of the lateral geniculate nucleus (LGN), and also the 36 retinotopic cortical channels.Manuscript received December 6, 2006; revised February 8, 2007; accepted February 8, 2007. This work was supported in part by the Chip Implementation Center (CIC), R.O.C., and in part by the National Science Council (NSC), R.O.C., under Research Project 94-2215-E-009-042. The associate editor coordinating the review of this paper and approving it for publication was Prof. Fabien Josse.
The authors are with the Nanoelectronics and Gigascale Systems Labora-tory, Department of Electronics Engineering and the Institute of Electronics, National Chiao Tung University, Hsinchu 300, Taiwan, R.O.C. (e-mail: m8711575@alab.ee.nctu.edu.tw; cywu@alab.ee.nctu.edu.tw).
Color versions of one or more of the figures in this paper are available online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/JSEN.2007.901194
In the retina, the output ganglion cells (GCs) can translate photoreceptor signals into neural spikes and send the spikes to the brain. Each channel extracts a unique spatiotemporal feature of the visual world. About 1/2 of the cerebral cortex is used to analyze the data from these channels [23], [25]. These features comprise the necessary visual information needed for the brain to analyze the visual world, thus they constitute visual language for the brain [27], [36]. This visual language is very complicated and much of it is still unknown or not understood by neuroscientists.
Our detailed view of the operation of the mammalian retina, however, changed dramatically by the discovery of the opera-tion of the inner plexiform layer [26], [36]. This led to the un-derstanding of the roles of different amacrine cells, as well as the dual path in each channel. These channels have different re-sponses to light input patterns from the visual world. Therefore, both the morphological and electrophysiological characteristics of the retina should be considered before designing a retinal chip in seeking to understand visual language. Moreover, the neuro-morphic multilayer CNN model [22], [27] shed light to under-stand the role of different circuits and layer parameters.
It is important to successfully duplicate the retinal functions, channels, and visual language on silicon chips because of the key advantages this might provide. First, it could help neuro-scientists to understand retinal functions and visual language. Since biological experiments can only be performed on a very limited number of cells, it is very difficult to see the global spa-tiotemporal features of retinal cells using this method alone. Second, it could provide valuable clues concerning neural activ-ities in the visual cortex and, thus, move a few more steps toward the discovery of the visual processing of the brain. Third and finally, duplication could enable important applications in the areas of intelligent visual sensor systems and retinal prostheses. To date some CMOS neuromorphic chips have been designed to imitate the retinal channels [28]–[32]. In these chips, the retinal channels are simply divided into ON/OFF and transient/ sustained channels. To implement these channels, a general cir-cuit is used to generate the channel signals that are similar to the measurement results of retinal cells. Some of the function-alities have been implemented using programmable cellular vi-sual microprocessors. After the major discovery in retinal oper-ation mentioned above [26], the first real-time, programmable, multichannel, neuromorphic physical implementation was con-structed using the Bi-i camera computer [35].
In our research, a design methodology has been developed to implement specific retinal channel in CMOS technology. Since each retinal channel is represented by an array of the corre-sponding GC types, it is suitable to define the channel with the corresponding GC. In the proposed methodology, each kind of 1530-437X/$25.00 © 2007 IEEE
Fig. 1. (a) The neuromorphic model and (b) the RC equivalent circuit of the ON brisk transient GC set of rabbits’ retinas. The parametert denotes the time constant in milliseconds of the LPF. The parameterD denotes the space constant, which is defined by the laterally diffusing range in cell number in a 180-cell array.
GC set is implemented in a compact CMOS circuit. Different kinds of GC sets can be implemented similarly and integrated together.
In this paper, the ON brisk transient GC set of rabbits’ retinas is adopted and implemented on a CMOS neuromorphic chip. The circuitry and cellular responses of this GC set has been completely revealed. Therefore, the design methodology can be verified well with the chosen GC set. The neuromorphic model of the ON brisk transient GC set, which is directly derived from the biological measurements is considered. The model-building approach is to incorporate the available knowledge concerning morphology, electrophysiology, and pharmacology and by only using elementary building blocks [22]. Then, the model is transformed into a RC equivalent circuit which consists of gain blocks, resistors, and capacitors. Based on the RC equiv-alent circuit, a suitable macromodel is developed for CMOS circuit implementation. The resultant chip has been fabricated and measured, and its retinal functions have been verified successfully.
In Section II of this paper, the retinal model of the ON brisk transient GC set is presented. In Section III, the detailed CMOS circuits and the chip architecture are described. The experimental results are presented in Section IV, where the various spatial, temporal, and spatiotemporal characteristics of the neuromorphic model and the implemented chip are shown. Finally, the summary is given in Section V.
II. RETINAMODEL
A. Neuromorphic Model of the Retinal Cell Set
The block diagram of the neuromorphic model of the ON brisk transient GC set is shown in Fig. 1(a), where each block has a specific name and is defined as the abstract neuron in [22]. An abstract neuron is a low-pass filter (LPF) with laterally dif-fusive capability. The parameter denotes the time constant in millisecond of the LPF. The parameter D denotes the space con-stant, which is defined by the laterally diffusing range in cell number in a 180-cell array. Its physical meaning is defined by
the spatial range where the signals decrease to 10% of its highest value. The small block with the parameter G is the gain factor of each interconnection between two blocks. The black block de-noted as R means the signals are positively rectified through it. Since biological cells do not work very precisely, relationships between both time and space constants affect the spatiotemporal features more than their absolute values. Even if scaling down these parameters by similar factors, similar spatiotemporal fea-tures are still obtained after normalization.
The space constants are obtained from morphological and electrophysiological data derived from living neural tissue, which may contain information concerning both cell sizes and lateral coupling between cells. However, it has been found that there are only three levels of lateral inhibition mediated by the horizontal cell, the OFF bipolar cell, and the amacrine cell [33]. The lateral inhibitions compress the spatial representation of the stimulus, thus they have some key effects on the spatiotem-poral characteristics of the GC channels. Therefore, only the space constants of the horizontal cell, the OFF bipolar cell, and the amacrine cell contain the information of lateral coupling between cells and, as a consequence, they play more important roles than the other space constants.
Corresponding to biological cells, the blocks PH1 and PH2 together comprise the photoreceptor, where PH1 performs the sensing function of a photoreceptor and PH2 performs the tem-poral function of the photoreceptor feedback. The photoreceptor receives photo stimuli and transduces them into electrical sig-nals. The block H provides functions of a horizontal cell, which performs lateral wide-range diffusion operation on signals of the photoreceptor and sends inhibitory feedback to the photore-ceptor. The feedbacks from both PH2 and H are subtracted from the photo-stimuli at the block PH1 and then sent to the bipolar cells. The PH1 signals are amplified to four times their existing level before they enter the bipolar cells.
There are two bipolar cells, namely, the ON and OFF bipolar cells, in the proposed cell set and both the ON and OFF bipolar cells are of a transient type. The transient-type bipolar cell acts like a bandpass filter (BPF). Since the neuromorphic model is built by using LPF blocks, two LPF blocks are needed to realize the function of a bipolar cell. As can be seen in Fig. 1(a), blocks HELP1 and ONBIP comprise the ON bipolar cell, whereas HELP2 and OFFBIP together comprise the OFF bipolar cell. Blocks HELP1 and HELP2 are not real neurons, but are assistant blocks to achieve the BPF function of the ON and OFF bipolar cells, respectively. Since the block ONBIP has a smaller time constant ms than that of the block HELP1 ms , subtracting the output of HELP1 from that of ONBIP can generate BPF signals with the up 3 dB frequency at 4 Hz and the down 3 dB frequency at 33.33 Hz in the ON bipolar cell. The same principle can be applied to
blocks HELP2 ms and OFFBIP ms . The
up and down 3 dB frequencies of the BPF are 5 and 12.5 Hz, respectively, in the OFF bipolar cell. Moreover, the OFF bipolar cell performs a very narrow lateral diffusion, while the ON bipolar cell lacks this function. Subsequently, the ON and OFF bipolar cell signals are positively rectified before they enter the amacrine and GCs.
The block named Ama provides the functions of an amacrine cell in the proposed cell set. The amacrine cell receives
posi-tively rectified signals from the ON and OFF bipolar cells with different gain factors: 0.2 and 1, respectively. It performs a lat-eral diffusion in a small range, and then sends inhibitory sig-nals to the GC. The block named GC performs the functions of the GC. It receives the positively rectified signal from the ON bipolar cell with a gain factor of 2 and the signal from the amacrine cell with a gain factor of 2.2. Finally, the GC signal is positively rectified to achieve the complete function of the GC before generating spikes. This signal can be used to gen-erate neural spikes necessary to communicate with the brain.
The actual RC equivalent circuit of the electrical model is shown in Fig. 1(b), where is realized by a RC LPF and D is realized by a 2-D laterally resistive network. To understand the behavior and spatiotemporal pattern of every cell, the RC equiv-alent circuit is simulated and analyzed. The simulated results are presented in Section IV. Since the neuromorphic model is based on the available neuroscientic knowledge of the retina, the sim-ulated spatiotemporal patterns of the RC equivalent circuit are similar to the biologically measured results.
B. Macromodel for Chip Implementation
To facilitate the integrated circuit implementation, the neuro-morphic model in Fig. 1(a) is transformed into the macromodel in Fig. 2. This macromodel is used to integrate the cell set into a single pixel. A sensory chip containing 32 32 pixels is em-ployed to verify the function of the macromodel. The macro-model consists of a photoreceptor (PH1 and PH2), a horizontal cell (H), an ON and OFF bipolar cells (ONBIP and OFFBIP), an amacrine cell (Ama), and a GC. Block R ensures that the sig-nals are positively rectified as they pass through it. The and signs within a circle indicate the signals which are added and subtracted, respectively, at that node. Due to the variations of biological cells, the constants in space and time for each cell have their varying degrees of tolerance. This allows inevitable process variations in CMOS circuit realization of these space and time constants.
There were some modifications when constructing the macro-model from the neuromorphic macro-model of Fig. 1(a). First, a gain stage with a value 8 was added in front of the pixel circuit. This is used to enlarge the input photocurrent to facilitate the tracking of the circuit operations. Second, space constants of PH1, ONBIP, and GC were removed. Only the space constants of the horizontal cell, the OFF bipolar cell, and the amacrine cell are considered in order to simplify the complex interconnection, while retaining the functions of the three levels of lateral inhi-bition. Furthermore, since the pixel array shrinks from 180 180 to 32 32, the space constants of the horizontal cell, OFF bipolar cell, and amacrine cell are proportionally scaled down to 27, 2, and 6, respectively. Third, there are better methods to implement a BPF with CMOS circuit rather than subtracting a LPF from the other LPF. Therefore, at the ON bipolar cell, the two blocks HELP1 and ONBIP are merged into the one block ONBIP in the macromodel, as can be seen in Fig. 2. The time constants of HELP1 and ONBIP are also merged into a single time constant, . The two values in denote the up and down 3 dB time constants. Similarly, the blocks HELP2 and OFFBIP are merged into the block OFFBIP. Fourth, the time constants of the blocks PH1 and H are modified to and that of block PH2 is modified to to facilitate chip
implemen-Fig. 2. The macromodel of a single pixel of the implemented chip.
Fig. 3. The block diagram of a single pixel.
tation. For the same reason, the down 3 dB time constants of blocks ONBIP and OFFBIP are changed to and the up 3 dB time constants of blocks ONBIP and OFFBIP are changed to and , respectively. Finally, since the time constants of the amacrine cell and the GC are smaller than those of the ON and OFF bipolar cells, their neglect had no effect on the signal flow. Therefore, the time constants of the blocks Ama and GC were removed in the macromodel.
III. CIRCUIT ANDCHIPARCHITECTURE
Fig. 3 is the block diagram corresponding to the macro-model in Fig. 2. A vertically parasitic p+-n-well-p-substrate BJT with a floating base is used as a photo-BJT. The photo-BJT traduces light stimuli into electrical currents. Then the photocurrent is read out, inversed, amplified by eight times, and sent to the block PH1. The PH1 signal is sent to PH2 and H to perform temporal delay and spatial diffusion, respectively. Then, the signals from PH2 and H are again sent to PH1. Since the input of this circuit is a photocurrent, all signals in PH1, PH2, and H are in current mode.
The output current of PH1 directly enters the OFF bipolar cell (OFFBIP). On the other hand, an additional cas-code current mirror (denoted by subblock CM) is used to inverse it and send it to the ON bipolar cell (ONBIP) in order to achieve the inverse bandpass-filtering function. Since very low-frequency BPFs (denoted by subblock BPF) operating in voltage-mode are needed for the bipolar cells, the current-mode signals from PH1 and CM are converted into voltage-mode signals by Rm amplifiers (denoted by the subblock Rm Amp.) before entering the BPF subblocks. The output signals of the subblocks BPF are converted to current-mode signals again using the subblocks named V-I converter.
TABLE I
TRANSISTORS’ SIZES OFCIRCUITS INFIG. 4(a)–(c)
The absolute-value circuits (denoted by the subblock ABS) are used to rectify the bipolar cell signals. At the amacrine cell (Ama), the positively rectified signals of both ONBIP and OFFBIP are summed with different gain factors. The GC subtracts the Ama signal from the positively rectified ONBIP signal. Another subblock ABS is used to rectify the GC signal to realize the function of the ON brisk transient GC set.
Fig. 4. (a) The circuit of the photoreceptor and horizontal cell (PH1, PH2, H). There are three space constants in blocks H, OFFBIP,
and Ama. The space constants are realized by using tunable NMOSFET smoothing networks [4]. Such a technique can adjust the space constants by controlling the gate biases of the NMOSFETs, thus allowing greater flexibility in designing the space constants. Moreover, since the horizontal cell has a relatively large space constant compared with the other cells, the NMOSFETs used in its smoothing network have a larger W/L ratio than the other two cells. The detailed transistor-level circuit of a pixel is shown in Fig. 4(a)–(c), and the transistor sizes are listed in Table I.
A. Photoreceptor and Horizontal Cell
Fig. 4(a) represents the circuit of the photoreceptor (PH1 and PH2) and horizontal cell (H). Because the light-induced pho-tocurrent is around 100-pA to several hundred-nA, it can ben-efit from the good linearity of cascode CMs. Moreover, since the photocurrent might flow in or out of the circuit, complementary CMs are used to enable both current directions. The transistors are used to bias the emitter of at and to di-rect the photocurrent into the circuit. The transistors
provide a current gain of 8, as described in Fig. 3.
The CMOS transmission gates composed of transistors
and in Fig. 4(a) and (b) are used
as resistors. Their resistances are controlled by and
for NMOSFET and PMOSFET, respectively. The resistances and the gate capacitances compose the LPF time constants of the blocks PH1, PH2, H, ONBIP, and OFFBIP. The unit LPF time constant can be controlled by and simultaneously. On the other hand, the ratios of time constants can be set by designing the MOSFET’s sizes to get the suitable gate capacitances.
The CM composed of transistors and
is designed to have a current gain of four times. Thus, the current of block PH1 is enlarged to enter block H. The enlarged current is spread by a diffusion network controlled by , and then it is sent to the block PH1 by transistors . On the other hand, the current of the block PH1 is directly repeated
to enter the block PH2 via transistors and
. Transistors are used to
per-form the temporal delay and to send feedback to the block PH1. Therefore, the channel length and width of transistors , , , and , are both twice as large as those
of , , , and . Thus, the current gain
is kept as unity because the W/L ratio does not change. How-ever, the time constant of the block PH2 is four times larger than that of the block PH1 because the gate capacitances of
and are four times larger than those of and .
The output current of block PH1 is enlarged four times by
Fig. 4. (Continued.) (b) The circuit of ON and OFF bipolar cells (ONBIP, OFFBIP). (c) The circuit of Ama and GCs.
, while transistors serving as
the CM master. Transistors generate ,
which is sent to the block OFFBIP. The drain current of
tran-sistors and is inversed using a complementary
CM to generate , which is sent to the block ONBIP. At the output stage, the transistors and , and
, and and are used to repeat the output currents of the blocks PH1, H, and PH2, respectively, to enable chip mea-surement. Since there are external biases applied to measure the
chip, simple structure instead of the cascode type slaves are used to save the pixel area.
B. ON and OFF Bipolar Cells
The circuit of the bipolar cell stage is shown in Fig. 4(b). The circuit can be divided into two parallel paths including blocks ONBIP and OFFBIP, separately. At the OFFBIP path,
used to transduce the current into voltage-mode. After-ward, a poly capacitor and a CMOS transmission gate, including transistors and , are used to form a high-pass filter (HPF). This HPF and the LPF composed of , , and form the subblock BPF. The bandwidth of the HPF, which is also the up 3 dB time constant of the BPF , can be tuned by the biases and at gates of and , respectively. The transmission gate is connected to the reference bias , which is set to 1/2 of the supply voltage to balance both upward and downward signal swings.
The voltage-mode signal of the subblock BPF is transduced into the current-mode signal again by using a V-I converter com-posed of transistors [34]. The V-I converter is a differential amplifier. Its positive input node connects to the output of the subblock BPF, and its negative input node con-nects to the reference bias . Its output current is passed to a modified absolute value circuit (ABS) composed of the
tran-sistors . Transistors form a
simple CM, and transistors and play the roles
of adjustable switches that are controlled by and , respectively. When are biased suitably, the output of subblock BPF, which is smaller than , will be converted into current-mode and mirrored only by , , and , while that larger than will be converted and
mir-rored only by and . The transistors and
repeat the complete signal of the block OFFBIP and send it out of the chip as . In the mean time, the tran-sistor sends the positively rectified signal of the block OFFBIP to the block Ama as .
At the ONBIP path, the circuit operations are mostly the same as at the OFFBIP path. The main differences of the ONBIP path from the OFFBIP path are as follows. First, the ONBIP path receives the inversed signal from the block PH1 as [Fig. 4(a) and (b)]. Second, is designed to be 1 pF and
is designed to be 312.5 fF in order to achieve the ratio of time constants of the blocks ONBIP and OFFBIP. Third, there is a smoothing network at the output node of the V-I converter of the OFFBIP path to perform the lateral diffusion, but this is lack at the ONBIP path. Finally, the transistors and re-peat the signal of the block ONBIP when the output of the sub-block BPF is smaller than , and send it to blocks Ama and
GC as and , respectively.
C. Amacrine (Ama) and Ganglion Cells (GCs)
The circuit of the Ama and the GC is shown in Fig. 4(c). The block Ama is constructed using transistors .
The transistor receives the summation of and
, shown in Fig. 4(b), and transistors repeat the summed current. The repeated current is laterally diffused by a smoothing network at the drain node of transistors and . Then, this diffused current is inversely repeated by transistors and is sent out of the chip via as . At the same time, this current is inversely repeated again to enter the block GC by the cascode CM composed of
tran-sistors , , , and . This CM is also designed
to implement the gain block after the block Ama, as shown in Fig. 2. The block GC receives , as shown in Fig. 4(b), and subtracts it from the current mirrored using transistors
Fig. 5. (a) The whole chip architecture of the implemented chip. (b) The im-plementation of smoothing networks.
, , , and . The subtracted current is sent to the subblock ABS composed of the transistors to be positively rectified. Thus, only the signal on the positive side is repeated and sent out of the chip by as .
D. Whole Chip Architecture
The architecture of the fabricated chip is shown in Fig. 5(a). The pixel circuit mentioned above is arranged in a 32 32 array. A row decoder and a column decoder are used to select the pixel that is to be observed. Every time there is only one pixel that can be observed with seven observable signals: , , ,
, , , and , as shown in Fig. 4(a)–(c).
Using this method, the behavior of each pixel, as well as that of the whole array can be observed by setting different controlling strategies using the row and column decoders.
The technique to implement the smoothing networks should be mentioned next. Since there are three lateral diffusion layers at the horizontal cell, the OFF bipolar cell, and the amacrine cell, three diffusion networks are needed in the designed chip. As can be seen in Fig. 5(b), there are three NMOSFETs which connect two neighboring pixels. These NMOSFETs correspond to the smoothing networks in Figs. 3 and 4. On average, each pixel has six NMOSFETs to perform the lateral diffusion. Thus, three two-dimensional smoothing networks are achieved.
IV. EXPERIMENTALRESULTS
A. Neuromorphic Model Simulation
The RC equivalent circuit in Fig. 1(b) is used to construct a 32 32 array which is simulated by HSPICE. Since the array
Fig. 6. The HSPICE simulated spatiotemporal patterns of the neuromorphic model of the ON brisk transient cell set in a 322 32 array for (a) photoreceptor, (b) horizontal cell, (c) ON bipolar cell, (d) OFF bipolar cell, (e) Ama cell, and (f) GC. These patterns are recorded from the 17th row of the array. Thex axis is normalized time and they axis is the pixel location which denotes space. The stimulus is applied to the 15th to the 20th pixel at time point 1001 to 2000. The waveform at the right of each pattern is the spatial domain waveform(s) obtained at the time marked by the vertical arrow(s). The waveform at the bottom of each pattern is the temporal domain waveform obtained at the location marked by the horizontal arrow.
size differs from the original definition of the space constants D, all the space constants in the simulation are shrunk with the same factor, 32/180. The simulation results are shown in Fig. 6. In this simulation, a voltage pulse is applied to the 6 6 cells in the center of the array to imitate the 1 Hz flashing light stim-ulus and the spatiotemporal patterns of the 17th row are ob-served. In Fig. 6, the axis is time and the axis is the pixel location which denotes space. The black and white bars denote the spatial and temporal region of the input stimulus. The stim-ulus is applied to the 15th to the 20th pixel from 1001 ms to 2000 ms. The waveform at the right of each pattern is the spatial domain waveform(s) obtained at the time marked by the vertical arrow(s). The waveform at the bottom of each pattern is the tem-poral domain waveform obtained at the location marked by the horizontal arrow.
Fig. 6(a) and (b) represent the spatiotemporal patterns of the photoreceptor and horizontal cell, respectively. It can be seen from Fig. 6(a) that the photoreceptor’s signal level drops when there is a stimulus, and it returns to its original level when the stimulus disappears. Slight undershooting and overshooting in temporal domain can be expected in the periphery as it reacts to the stimulus directly. In the spatial domain, there is strong contrast at the edge of the stimulus. At the edge pixels inside the stimulus, the signal level is higher than the other stimulated pixels. Contrarily, at the edge pixels where the stimulus is just absent, the signal level is higher than the other silent pixels. Therefore, the spatial range of the stimulus can be well defined. In the temporal domain, the horizontal cell has a similar reaction to the photoreceptor, as can be seen in Fig. 6(b). However, since the horizontal cell performs lateral diffusion in space, its spatial domain waveform spreads wider than that of the photoreceptor.
Fig. 7. Photographs of (a) the whole chip and (b) a single cell.
Fig. 6(c) and (d) represent the spatiotemporal patterns of the ON and OFF bipolar cells, respectively. The OFF bipolar cell performs bandpass-filtering on the signals from the photore-ceptor, whereas the ON bipolar cell performs bandpass-filtering on the same signals but with an opposite polarization. In the Ama cell, the rectified signals from both bipolar cells are added with different gain factors, as shown in Fig. 6(e). Moreover, the response to the appearance of the stimulus is weaker than the response to the disappearance of the stimulus. Thus, the Ama cell provides a strong inhibition to the GC when the stimulus disappears in time. Therefore, the GC exhibits clear turned-ON reaction, as can be seen in Fig. 6(f).
B. The Experimental Chip and Setup
The chip is designed using the TSMC 0.35 double-poly-quadruple-metal standard CMOS technology. The photograph of the whole chip is shown in Fig. 7(a) and that of a single cell is shown in Fig. 7(b). The whole chip area is 4.3 mm 4.41 mm, and the cell pitch is 105 . There is a photo-BJT in the center to transduce light stimuli into photocurrents, as shown in Fig. 7(b). The fill factor of the photo-BJT is 7.44%. It can be seen from Fig. 7(a) that the pixel array occupies most of the chip area. The row decoder and column decoder are on the left and top of the chip. The ESD pads are used to protect the chip from electrostatic damages.
Since the designed chip sends pixel signals in current format, external current-to-voltage converters are needed to facilitate measurements. The readout circuit of each output is shown in Fig. 8. Since there are seven output signals generated by this chip, as shown in Fig. 5(a), there are seven similar readout cir-cuits. The circuit contains one operational amplifier (OP), a re-sistor, and a LPF composed of a resistor and a capacitor. The op-erational amplifier with a negative feedback provides a virtual
Fig. 8. The readout circuit to translate the output current into voltage.
TABLE II
(a) THEBIAS CONDITION TOMEASURE THE
SPATIOTEMPORAL PATTERNS OF FIG. 9. (b) THEBIASCONDITION TOMEASURE THETEMPORAL
DOMAINWAVEFORMS OFFIG. 10
bias at the negative input node, which is connected to the chip’s output node. The value of the virtual bias is determined by . The resistor converts the current into voltage. Therefore, the current flowing out from the chip will be transduced into through the readout circuit. Assuming the voltage gain of the operational amplifier A is infinite. The relationship
between and is
(1) The voltage-mode signal is actually observed and recorded.
The measured data are presented and analyzed below. Con-sidering the different signal swings of the output currents, the
for , , , and are set as 3 , while
the for , , and are set as 1 to
op-timize experimental conditions. The optimization of the exper-imental condition will yield the largest but unsaturated signal levels.
A white light LED is used to generate flashing light to stimu-late the test chip. The LED is controlled by a function generator, so that the amplitude and frequency of the incidental light can be monitored. However, the input stimuli are always repetitive because of the inherent nature of the function generator. An op-tical lens is used to concentrate the light from the LED onto the test chip, thus only a small part (around 5 5 pixels) of the chip is stimulated while the other part remains relatively dark.
Fig. 9. The measured spatiotemporal patterns for (a) photoreceptor, (b) horizontal cell, (c) ON bipolar cell, (d) OFF bipolar cell, (e) Ama cell, and (f) GC. These patterns are recorded from the 17th row of the array. Thex axis is normalized time and the y axis is the pixel location which denotes space. The stimulus is applied to the 19th to the 23rd pixel at time point 1001 to 2000. The waveform at the right of each pattern is the spatial domain waveform(s) obtained at the time marked by the vertical arrow(s). The waveform at the bottom of each pattern is the temporal domain waveform obtained at the location marked by the horizontal arrow.
The biases are controlled by a group of regulators adjusted with precisely variable resistors.
C. The Measured Spatiotemporal Patterns
The bias condition for measuring the spatiotemporal patterns is listed in Table II(a). The normalized spatiotemporal patterns are shown in Fig. 9. The light stimulated region is from the 19th pixel to the 23rd pixel in space, and from the 1001 point to the 2000 point in time. At the bottom of each pattern is the temporal domain waveform recorded at the 21st pixel, and at the right is
the spatial domain waveform(s) recorded at the time point(s) marked by the vertical arrow(s). In Fig. 9(a) and (b), the spa-tial domain waveforms are recorded at the time point 1500. In Fig. 9(c)–(e), the spatial domain waveforms are recorded at the time points 1020 and 2020. In Fig. 9(f), the spatial domain wave-forms are recorded at the time point 1020. The effects of the spatial diffusions can be seen in Fig. 9(b)–(e).
Fig. 9(a) illustrates the spatiotemporal pattern of the photore-ceptor. The pattern is not as perfect as that in Fig. 6(a) because of the interpixel variation and imperfection of the light stimulus.
Fig. 10. The normalized waveforms of the electrical model and chip measurements of (a) the ON bipolar cell, (b) the OFF bipolar cell, (c) the Ama cell, and (d) the GC. The light stimulus starts at time point 1001 and ends at time point 2000.
However, the spatial edge of light stimulus still can be recog-nized by the contrast between grey and white rows. The spa-tiotemporal pattern of the horizontal cell in Fig. 9(b) covers a wide area of space. The phenomenon can be seen clearly from its spatial waveform represented at the right of Fig. 9(b). Be-cause of this phenomenon, the effects of interpixel variation and imperfection of the light stimulus are not as strong as those pre-sented in Fig. 9(a).
The spatiotemporal patterns of the ON and OFF bipolar cells are shown in Fig. 9(c) and (d). The two patterns are roughly complementary, except that the pattern of the OFF bipolar cells is narrower in time but wider in space than the pattern of the ON bipolar cells. The pattern of the ON bipolar cell is noisier than the pattern of the OFF bipolar cell. Both cells are affected by the photoreceptor, but the lateral diffusion of the OFF bipolar cell reduces the effects of interpixel variation and imperfection of the light stimulus. The pattern of the Ama cell, as shown in Fig. 9(e), also covers a wide area of space. It responds to both appearance and disappearance of the light stimulus in time, as can be seen at the bottom of Fig. 9(e). It can be seen from Fig. 9(b) and (e) that their temporal domain waveforms are noisier that the other cells because the wide-range lateral diffusion introduces noise into the cells. This phenomenon is not very obvious in Fig. 9(d) because the range of lateral diffu-sion of the OFF bipolar cell is very narrow. The spatiotemporal pattern of the GC in Fig. 9(f) only responds to the appearance of the light stimulus in time, and has a sharp edge in space.
Through this experiment, it is found that the cells without lateral diffusion have good immunity to temporal noise, but have bad immunity to spatial noise. Contrarily, the cells with lateral diffusion have bad immunity to temporal noise, but have good immunity to spatial noise. Thus, a good spatiotemporal filter can be implemented through combining two kinds of cells described above.
Fig. 11. (a) The relationship between the time constant and the controlling biases. The biasV is set asVdd 0 1V and V is set as1V. (b) The relationship between the time constant and the controlling biases. The bias V is set asVdd 0 1V and V is set as1V. The relationships are ob-tained by varying the value of1V.
D. Temporal Domain Analyses
In order to demonstrate the degree of coincidence between the neuromorphic model and the measured results, the
normal-waveforms of measurements (solid line) are the temporal do-main waveforms of Fig. 9(c)–(f). The light stimulus starts at the time point 1001, and ends at the time point 2000. It can be seen from Fig. 10(a)–(d) that the shape of each waveform in the same comparison is similar. The slight deviations are caused by process and interpixel variations. However, this level of vari-ation is acceptable because biological cells also have intercell variations.
The variation of time constants is shown in Fig. 11. In Fig. 11(a), the relationship between and and is presented, and Fig. 11(b) is the relationship between and
and . In this experiment, a parameter is used to represent the x-axis. In Fig. 11(a), the bias voltage is
equal to , and bias voltage is equal to .
Thus, while increasing , is increased, while is decreased. An assumption is applied that the two biases and affect the equivalent resistance of the transmission gate symmetrically. Similarly, the bias voltage is set to
, and bias voltage is set to for finding
the relationship between and and . The
max-imum value of is 0.85 V in Fig. 11(a), and is 0.9 V in Fig. 11(b). The functions of the corresponding HPF and LPF are completely distorted if exceeding these values. This experiment provides the available range of time constants of the HPF, LPF, and BPF used in the circuit.
V. SUMMARY
A neuromorphic chip that imitates the ON brisk transient GC set of rabbits’ retinas was implemented and observed. Most parts of the chip deal with current-mode signals. The chip con-tains 32 32 pixels, where each pixel imitates one cell set of the ON brisk transient GC. The total power consumption is 1.675 W under 0.45 mW light stimuli. The power consumption can be further reduced by simplifying the cell circuit according to spe-cific applications. The measured results under flashing lights have the same spatiotemporal characteristics as those from bio-logical measurements obtained using similar stimuli. Such con-sistency strongly suggests that the chip, in extracting the fea-tures of the visual world, behaves in a way which is similar to that of real retinal cells. Therefore, this study provides the ex-perimental evidence needed by neuroscientists to understand the performance of all other retinal cells.
Furthermore, the verifications of the implemented chip es-tablish the success of the proposed design methodology. Thus, every kind of GC sets can be implemented and integrated in the same way. With the development of biological circuitries and spatiotemporal features, the implementation of this design methodology should make possible the realization and analysis
ACKNOWLEDGMENT
The authors would like to thank Prof. M.-D. Ker and J.-J. Peng for sharing with us their expertise and extensive knowledge involving ESD protection.
REFERENCES
[1] C. A. Mead and M. A. Mahowald, “A silicon model of early visual processing,” Neural Netw., vol. 1, pp. 91–97, 1988.
[2] C. Y. Wu and C. F. Chiu, “A new structure of 2-D silicon retina,” IEEE
J. Solid-State Circuits, vol. 30, no. 8, pp. 890–897, Aug. 1995.
[3] C. F. Chiu and C. Y. Wu, “A new structure for the silicon retina,” in
Proc. IEDM, 1992, pp. 439–442.
[4] C. Y. Wu and H. C. Jiang, “An improved BJT-based silicon retina with tunable image smoothing capability,” IEEE Trans. Very Large Scale
(VLSI) Integr. Syst., vol. 7, no. 2, pp. 241–248, Jun. 1999.
[5] T. Delbruck and C. A. Mead, “Analog VLSI phototransduction by con-tinuous-time, adaptive, logarithmic photoreceptor circuits,” Comput. Neural Syst. Program, Calif. Inst. Technol., Pasadena, CA, Memo No. 30, Jul. 1994.
[6] A. B. Torralba and J. Hérault, “An efficient neuromorphic analog net-work for motion estimation,” IEEE Trans. Circuits Syst. I, vol. 46, no. 2, pp. 269–280, Feb. 1999.
[7] T. Delbruck, “Silicon retina with correlation-based velocity-tuned pixels,” IEEE Trans. Neural Netw., vol. 4, pp. 529–541, May 1993. [8] R. Benson and T. Delbruck, “Direction selective silicon retina that uses
null inhibition,” in Advances in Neural Information Processing
Sys-tems, J. Moody, S. Hanson, and R. Lippmann, Eds. San Mateo, CA: Morgan Kaufman, 1992, vol. 4, pp. 756–763.
[9] S. C. Liu, “A neuromorphic aVLSI model of global motion processing in the fly,” IEEE Trans. Circuits Syst. II, vol. 47, no. 12, pp. 1458–1467, Dec. 2000.
[10] R. C. Meitzler, K. Strohbehn, and A. G. Andreou, “A silicon retina for 2-D position and motion computation,” in Proc. IEEE Int. Symp.
Circuits Syst., 1995, vol. 3, pp. 2096–2099.
[11] A. G. Andreou, K. Strohbehn, and R. E. Jenkins, “Silicon retina for motion computation,” in Proc. IEEE Int. Symp. Circuits Syst., Singa-pore, Jun. 1991, pp. 1373–1376.
[12] P.-F. Ruedi, “Motion detection silicon retina based on event correla-tion,” in Proc. 5th Int. Conf. Microelectron. Neural, Fuzzy, Bio-Inspired
Syst., 1996, pp. 23–29.
[13] R. Etienne-Cummings, J. Van der Spiegel, P. Mueller, and M.-Z. Zhang, “A foveated silicon retina for two-dimensional tracking,” IEEE
Trans. Circuits Syst. II, vol. 47, no. 6, pp. 504–516, Jun. 2000.
[14] H. C. Jiang and C. Y. Wu, “A 2-D velocity- and direction-selective sensor with BJT-based silicon retina and temporal zero-crossing de-tector,” IEEE J. Solid-State Circuits, vol. 34, no. 2, pp. 241–247, Feb. 1999.
[15] C. Y. Wu and K. H. Huang, “A CMOS focal-plane motion sensor with BJT-based retinal smoothing network and modified correlation-based algorithm,” IEEE Sensors J., 2, no. 6, pp. 549–558, Dec. 2002. [16] K. H. Huang, L. J. Lin, and C. Y. Wu, “The analysis and design of a
CMOS angular velocity- and directional-selective rotation sensor with retinal processing circuit,” IEEE Sensors J., vol. 4, no. 6, pp. 845–856, Dec. 2004.
[17] K. A. Boahen and A. G. Andreou, “A contrast sensitive silicon retina with reciprocal synapse,” Neural Inf. Process. Syst., vol. 4, pp. 764–772, 1992.
[18] A. G. Andreou and K. A. Boahen, “A 48,000 pixel, 590,000 transistor silicon retina in current-mode subtreshold CMOS,” in Proc. 37th
[19] A. G. Andreou, K. A. Boahen, P. O. Pouliquen, A. Pavasovic, R. E. Jenkins, and K. Strohbehn, “Current-mode subthreshold MOS circuits for analog VLSI neural systems,” IEEE Trans. Neural Netw., vol. 2, no. 2, pp. 205–213, 1991.
[20] C. Y. Wu and C. T. Chiang, “A low-photocurrent CMOS retinal focal-plane sensor with a pseudo-BJT smoothing network and an adaptive current Schmitt trigger for scanner applications,” IEEE Sensors J., vol. 4, no. 4, pp. 410–418, Aug. 2004.
[21] A. G. Andreou, “Low power analog VLSI systems for sensory informa-tion processing,” in Microsystems Technology for Multimedia
Applica-tions, B. J. Sheu, M. Ismail, E. Sanchez, and T. H. Wu, Eds. Piscat-away, NJ: IEEE Press, 1995, ch. 7.5.
[22] D. Bálya, B. Roska, T. Roska, and F. S. Werblin, “A CNN framework for modeling parallel processing in a mammalian retina,” Int. J. Circuit
Theory Appl., vol. 30, pp. 363–393, 2002.
[23] P. Sterling, “How retinal circuits optimize the transfer of visual infor-mation,” in The Visual Neurosciences, L. M. Chalupa and J. S. Werner, Eds. Cambridge, MA: MIT Press, 2003, vol. 1, pp. 234–278. [24] H. A. Basler, A. A. Brewer, L. T. Sharpe, A. B. Moreland, H. Jagle,
and B. A. Wandell, “Reorganization of human cortical maps caused by inherited photoreceptor abnormalities,” Nature Neurosci., vol. 5, pp. 364–370, 2002.
[25] D. C. van Essen, C. H. Anderson, and D. J. Felleman, “Information processing in the primate visual system: An integrated systems per-spective,” Science, vol. 255, pp. 419–423, 1992.
[26] B. Roska and F. Werblin, “Vertical interactions across ten parallel, stacked representations in the mammalian retina,” Nature, vol. 410, pp. 583–587, Mar. 2001.
[27] F. Werblin, B. Roska, D. Balya, C. Rekeczky, and T. Roska, “Imple-menting a retinal visual language in CNN: A neuromorphic study,” in
Proc. IEEE Int. Symp. Circuits Syst., 2001, vol. 2, pp. 333–336.
[28] K. A. Boahen, “A retinomorphic vision system,” IEEE Micro, vol. 16, no. 5, pp. 30–39, Oct. 1996.
[29] K. Boahen, “Retinomorphic chips that see quadruple images,” in Proc.
MicroNeuro, 1999, pp. 12–20.
[30] K. A. Zaghloul and K. Boahen, “Optic nerve signals in a neuromorphic chip I: Outer and inner retina models,” IEEE Trans. Biomed. Eng., vol. 51, no. 4, pp. 657–666, Apr. 2004.
[31] K. A. Zaghloul and K. Boahen, “Optic nerve signals in a neuromorphic chip II: Testing and results,” IEEE Trans. Biomed. Eng., vol. 51, no. 4, pp. 667–675, Apr. 2004.
[32] S. Kameda and T. Yagi, “An analog VLSI chip emulating sustained and transient response channels of the vertebrate retina,” IEEE Trans.
Neural Netw., vol. 14, no. 5, pp. 1405–1412, Sep. 2003.
[33] B. Roska, E. Nemeth, L. Orzo, and F. S. Werblin, “Three levels of lateral inhibition: A space-time study of the retina of the tiger sala-mander,” J. Neurosci., vol. 20, no. 5, pp. 1941–1951, Mar. 2000. [34] C.-Y. Wu and C.-H. Cheng, “A learnable cellular neural network
struc-ture with ratio memory for image processing,” IEEE Trans. Circuits
Syst. I, vol. 49, no. 12, pp. 1713–1723, Dec. 2002.
[35] D. Bálya and B. Roska, “Retina model with real-time implementa-tion,” in Proc. IEEE Int. Symp. Circuits Syst., Kobe, Japan, 2005, pp. 5222–5225, (Special Session C3LC).
[36] F. Werblin and B. Roska, “The movies in our eyes,” Scientific
Amer-ican, pp. 73–79, Apr. 2007.
Li-Ju Lin (S’00) was born in Taiwan, R.O.C., in 1976. She received the B.S. degree in electronics engineering from the National Chiao Tung Univer-sity, Hsinchu, Taiwan, R.O.C., in 1998. She is now working towards the Ph.D. degree at the Institute of Electronics, National Chiao Tung University.
Her major research interests are silicon retina chip design and corresponding applications, and cellular neural/nonlinear network.
Chung-Yu Wu (S’76–M’76–SM’96–F’98) was born in 1950. He received the M.S. and Ph.D. degrees from the Department of Electronics Engineering, National Chiao Tung University, Hsinchu, Taiwan, R.O.C., in 1976 and 1980, respectively.
In addition, he conducted Postdoctoral research at the University of California at Berkeley (UC Berkeley) in the summer of 2002. Since 1980, he has served as a consultant to high-tech industry and research organizations and has built up strong research collaborations with high-tech industries. From 1980 to 1983, he was an Associate Professor at the National Chiao Tung University. From 1984 to 1986, he was a Visiting Associate Professor in the Department of Electrical Engineering, Portland State University, Portland, OR. Since 1987, he has been a Professor at the National Chiao Tung University. From 1991 to 1995, he was rotated to serve as the Director of the Division of Engineering and Applied Science on the National Science Council, Taiwan. From 1996 to 1998, he was honored as the Centennial Honorary Chair Professor at the National Chiao Tung University. Currently, he is the President and Chair Professor of the National Chiao Tung University. He has published more than 250 technical papers in international journals and conferences. He also has 19 patents including nine U.S. patents. His research interests are nanoelectronics, low-power/low-voltage mixed-signal VLSI design, biochips, neural vision sensors, RF circuits, and CAD analysis.
Dr. Wu was a recipient of the IEEE Fellow Award in 1998 and Third Millen-nium Medal in 2000. In Taiwan, he received numerous research awards from the Ministry of Education, National Science Council, and professional founda-tions. He is a member of Eta Kappa Nu and Phi Tau Phi Honorary Scholastic Societies.
Botond Roska received the Ph.D. degree from the University of California at Berkeley (UC Berkeley).
He was a Harvard Society Fellow at Harvard Uni-versity and Harvard Medical School. Currently, he heads a laboratory at the Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland. His group is interested in understanding the structure and function of retinal circuits.
Frank Werblin is a Professor of Neuroscience at the University of California at Berkeley (UC Berkeley), where he heads a retina research laboratory that analyzes the physiological processing of visual signals by the retina and is also involved in the development of a variety of approaches to prosthetic vision.
David Bálya received the Computer Science degree from the Faculty of Electrical Engineering and Computer Science (EECS), Technical University of Budapest (TUB), Budapest, Hungary, and the Ph.D. degree from the TUB, while working in the Analogic and Neural Computing Laboratory, CARI, HAS, Hungary.
From 1999 to 2000, he was a Visiting Scholar in Frank Werblin’s Laboratory, University of California at Berkeley, cooperating with neurobiologists to develop a multichannel mammalian retina model based on a complex-cell cellular nonlinear network. In 2004, he visited Harvard University, Cambridge, MA, to develop a real-time multichannel retina model. Since the end of 2005, he has been a Postdoctoral Fellow in Botond Roska’s Laboratory, Friedrich Miescher Institute for Biomedical Research, Basel, Switzerland, and measures mouse retina using multielectrode array. His research interest includes neuromorphic studying of the visual system, bioinspired algorithm design, application of adaptive learning systems, and applying CS in social sciences.
been the Head of the Analogic and Neural Computing Research Laboratory. He is also a Professor and had been a founder Dean of the Faculty of Information Technology, Pázmány P. Catholic University, Budapest. He has taught several courses. Presently, he is teaching graduate courses on “Nonlinear Dynamics and Emergent Computations” and “Cellular Wave Computers and Visual Microprocessors.” For many years, he has been directing a Multidisciplinary Doctoral School. In 1974 and since 1989 in each year, he has been a Visiting Scholar at the University of California at Berkeley (UC Berkeley). He has pub-lished more than 100 research papers and four books (partly as a coauthor). His seminal paper on the CNN Universal Machine, coauthored by L. O. Chua, has received close to 500 citations. All together, he has more than 2000 citations. His research interests include cellular wave computing, infobionics, cellular neural networks, nonlinear circuit and systems, neural electronic circuits, and analogic spatial-temporal supercomputing and computational complexity.
fornia at Berkeley (UC Berkeley) . He is an elected member of four Academies of Sciences in Europe. In 2002 and 2003 he had been serving as Editor-in-Chief of the IEEE TRANSACTIONS ONCIRCUITS ANDSYSTEMSHe is a member of the Editorial Board of the International Journal of Circuit Theory and Applications, the International Journal on Bifurcation and Chaos, the Journal of the Franklin
Institute, and the Neural Processing Letters. He has been a founding Chair of
the Technical Committee on Cellular Neural Networks and Array Computing in the IEEE Circuits and Systems Society. For four years, in Hungary, he had been the Advisory Chair of the National R&D Program on Information and Com-munication Technology, and since 2005, he has been serving on the Advisory Committee of the EU Commissioner (Minister) in the Commission of Infor-mation Society and Media Technologies in Brussels.
