Eco-Conductor - 利用虛擬生態系統之互動式音景

國

立

交

通

大

學

工學院聲音與音樂創意科技碩士

學位學程

碩

士

論

文

Eco-Conductor - 利用虛擬生態系統之互動式音景

Eco-Conductor – Interactive Soundscape with Virtual Ecosystem

研 究 生：楊禮瑋

指導教授：冀泰石 教授

Eco-Conductor - 利用虛擬生態系統之互動式音景

Eco-Conductor – Interactive Soundscape with virtual ecosystem

研 究 生：楊禮瑋 Student：Li-Wei Yang

指導教授：冀泰石 Advisor：Tai-Shih Chi

國 立 交 通 大 學

工 學 院 聲 音 與 音 樂 創 意 科 技 碩 士 學 位 學 程

碩 士 論 文

Submitted to Master Program of Sound and Music Innovative Technologies College of Engineering

National Chiao Tung University in partial Fulfillment of the Requirements

for the Degree of Master

in Engineering March 2011

Hsinchu, Taiwan, Republic of China

Eco-Conductor: 利用虛擬生態系統之互動式音景系統

學生：楊禮瑋

指導教授

冀泰石

國立交通大學工學院聲音與音樂創意科技碩士學位學程

摘

要

Eco-Conductor: the Interactive Soundscape with Virtual Ecosystem

Student: Li-Wei Yang

Advisors: Dr. Tai-Shih Chi

Master Program of Sound and Music Innovative Technologies

National Chiao Tung University

ABSTRACT

In this thesis, we present the “Eco-Conductor”, an interactive system offering a

soundscape where users can conduct. In the soundscape, continuums of properties of sounds

were designed. Instead of direct mappings from users’ gestures to sounds or features of

sounds, a virtual ecosystem is used as the mapping platform. By affecting the ecosystem, the

soundscape is indirectly varied. The ecosystem was also visualized so that users can feel the

variation in both the soundscape and the visual presentation. We also designed a

human-machine interface which detects users’ gestures. These gestures are interpreted into

the arousal-valence plane and the corresponding attributes affect the ecosystem. Therefore,

Acknowledgement

Table of Contents

Chinese Abstract

………

i

English Abstract

………

ii

Acknowledgement

………

iii

Table of Contents

………

iv

List of Tables

………

v

List of Figures

………

vi

I.

Introduction………

1

II.

Background………

3

2.1

Soundscape………

3

2.1.1

Soundscape Compostion………

4

2.2

Artificial Life………

6

2.3

Ecosystem………

8

2.4

Related Work………

8

III.

Eco-Conductor………

10

IV.

Soundscape Design………

12

4.1

Nature to Noise………

12

4.2

Weather………

17

4.3

Overall Soundscape………

18

V.

Virtual Ecosystem………

21

5.1

Environment………

21

5.2

Agents………

21

5.2.1

Fitness Evaluation………

22

5.2.2

Attributes………

23

5.2.3

Actions and decision flow………

24

5.3

Dominator………

30

5.4

Game Strategy………

31

5.5

Ecosystem to Sound………

31

5.6

Visual Presentation………

33

VI.

User Interface and System Architecture………

40

VII.

Conclusion………

43

List of Tables

Table 4.1

Natural sounds………

15

Table 4.2

Ocean waves………

17

List of Figures

Figure 3.1

The framework of Eco-Conductor………

10

Figure 4.1

The read ratio of samples decreases when approaching

machine-like noise………

14

Figure 4.2

The fixed duration and silence duration extend when

the soundscape gets passive………

15

Figure 5.1

The mate finding diagram………

25

Figure 5.2

The decision diagram of agents……… 26

Figure 5.3

The diagram of mating process………

28

Figure 5.4

The diagram of reproduction process……… 29

Figure 5.5

The visualized ecosystem………

33

Figure 5.6

The color mapping………

34

Figure 5.7

Agents with different arousal values have different

speed of rotation. The left side shows slower rotation

for lower arousal value. The right side shows faster

rotation for higher arousal value………

35

Figure 5.8

Different types of the transformation………

36

Figure 6.1

The user interface, Scepter. The switch button is at the

bottom. The FSR is on the surface. The accelerometer

and the Arduino microcontroller are inside the scepter

41

Figure 6.2

The system architecture……… 42

Chapter 1

INTRODUCTION

The interactive music has been a popular field with more and more related works and studies for years. Different from most of these works which are based on music theories and structures, our design is to address the soundscape composition where environmental sounds are substituted for musical notes and arrangements of the sounds are emphasized than music structures.

The soundscape composition was first proposed at Simon Fraser University in 1970s [1], and some artists have been dedicated to this area for a long time. However, only few attempts of making soundscape composition via interactive ways have been demonstrated so far. The goal of our work is to let users control and experience the movement of the soundscape modified by their gestures. Our soundscape is designed to vary in the arousal-valence plane [2] with continuums of properties of sounds which are created either by synthesizing or processing methods.

In some works of interactive music and art, the artificial life was used in several ways such as the swarm [3], ant algorithms [4], and ecosystems [5], etc. In our work, we design an ecosystem as the mapping platform. Unlike direct mappings from gestures to sounds or to features of sounds shown in many other works, our virtual ecosystem provides an indirect and more elaborate way of mappings which increase the variety and complexity of structures of composed sounds. Our ecosystem is also visualized on the screen for users, who act as dominators in the ecosystem, so that they can observe the varying ecosystem and listen to the corresponding movement of the soundscape at the sametime.

We combine these concepts in our “Eco-Conductor” system to bring users fresh new experiences in the soundscape composition with the indirect mappings by a visualized ecosystem. Concepts of our proposed system and its underlying modules are thoroughly described in following sections.

Chapter 2

BACKGROUND

2.1 Soundscape

In the famous book which of the soundscape research field “Our Sonic Environment and The Soundscape: the Tuning of the World”, the soundscape is defined as:

The sonic environment. Technically, and portion of the sonic environment regarded as a field for study. The term may refer to actual environments, or to abstract constructions such as musical compositions and tape montages, particularly when considered as an environment [6].

Soundscape is composed of keynote sounds, signals and soundmarks. Keynotes can be regarded as background sounds which are often not listened to consciously. They outline the state and the characteristic of a community. People can acquire the basic information of t they live in such as the weather and the skeleton of geography based on keynotes. Signals can be regarded as foreground sounds which are listened to consciously, such as warning and notifying sounds. Soundmarks are similar to landmark, which are referred to unique sounds in a community.

The soundscape research has started since 1970s. World Soundscape Project (WSP) at Simon Fraser University was founded by R. Murrary Schafer. Its main purpose was to document acoustic environment and increase public awareness of the importance of the soundscape [7]. The first systematic study was The Vancouver Sounscape published in 1973 [1]. Soundscape composition appears afterwards.

2.1.1 Soundscape composition

A form of electroacoustic music which is characterized by recognizable environmental sounds and contexts is termed soundscape composition by Barry Traux [1]. Soundscape composition grew out of the World Soundscape Project. Composers processed recorded sounds to make compositions. The recorded sounds could be simply arranged in the composition or could be transformed to abstract sounds using electroacoustic techniques. Barry Traux wrote four soundscape composition principles [7]:

(1) Listener’s recognizability of the source material is maintained;

(2) Listener's knowledge of the environmental and psychological context is invoked; (3) Composer's knowledge of the environmental and psychological context

influences the shape of the composition at every level;

(4) The work enhances our understanding of the world and its influence carries over into everyday perceptual habits.

Barry Traux also classified the soundscape composition into three structural approaches [1]:

(1) Fixed perspective: emphasizing the flow of time. Sound events in time determine the structure of the piece. This flow is created by the relationship between the sounds that are heard. The auditory scenes are essentially static, the listeners are free to concentrate how on how the flow of original and transformed sounds creates its own mood and symbolism.

The motion can be at the level of vibration, of pattern, and of longer term change. The motion can be made using cross-fade technique to move away from one source toward another. The motion also can refer to the transition from a realistic sound to an abstracted one, or vice versa. Granular synthesis can be used to create smooth transition from an original sound to the transformed one. The moving perspective offers the feeling of a journey.

(3) Variable perspective: emphasizing a discontinuous space/time flow. Multiple auditory scenes are presented simultaneously. Complex imaginary soundscape is created instead of a single image of a soundscape. Listeners are to decode the embeddings of sounds that originated at widely varying times and places as long as there appears to be a logical order to their combination. By using multi-track editing tools, these complex compositions can be created.

Hildegard Westerkamp also mentioned [8]: “A soundscape composition is always rooted in themes of the sound environment. It is never abstract. Recorded environmental sounds are its’ ‘instruments’, and they may be heard both unprocessed and processed.”

Environment sounds to the soundscape composition are as musical notes or phrases to the music composition. Soundscape composition also has its structures as mentioned above. We can regard soundscape composition as music, and also, regard music as soundscape composition. The listeners interpret soundscape and try to grab and investigate the meaning and information encoded in it by invoking their imaginations.

2.2 Artificial life

The term ‘Artificial life’ is defined by Christopher Langton [9]. He defined the discipline of artificial life as “the study of man-made systems that exhibit behaviors characteristic of natural living systems.” The goal of the field is to emulate the real living phenomena of the real world. Furthermore, it is also extended from life-as-we-know-it to life-as-it-could-be.

J. Doyne Farmer and Alletta d’A. Belin defined the properties of life [10]: (1) Life is a pattern in spacetime, rather than a specific material object.

(2) Self-reproduction, if not in the organism itself, at least in some related organism.

(3) Information storage of a self-representation.

(4) A metabolism which converts matter and energy from the environment into the pattern and activities of the organism.

(5) Functional interaction with the environment. (6) Interdependence of parts.

(7) Stability under perturbations and insensitivity to small changes. (8) The ability to evolve.

Artificial evolution is an important element in an artificial life system. Arantza Etxeberria proposed three main tendencies of artificial evolution [11]:

(1) Evolutionary theory is used as a problem-solving strategy.

model or theory has been developed so far.

(3) Boundaries between the artificial and the biological are diffuse, and instances of alternative universes are constructed.

The artificial life is a field of study that is not a part of the artificial intelligence. Jean-Claude Heudin listed four differences between the artificial life and the artificial intelligence. In particular, the artificial intelligence can be regarded as an engineering side of the theoretical psychology while the artificial life can be regarded as an engineering side of the theoretical biology. Furthermore, the artificial intelligence is based on top-down analytical approach for designing systems while artificial life is based on bottom-up synthetic approach [12].

The evolutionary methods can be nature rules defined in the systems and operating internally. These nature rules establish the relationship between the environment and the living organisms. It can also be human selection that the evolution is intervened by users in interactive ways. We can just let the system run and observe the varying output while the state of the artificial world varies.

Artificial life methodologies can offer new directions and new points of view for the music and art creation, especially generative music and art. The features of the artificial life meet the need for generative creation. Furthermore, the use of artificial life strategies provides more complex and various outputs. There are already various artificial life applications in the music and art areas. Various techniques have been used such as intelligence swarm, ant algorithm, boids, and ecosystems.

2.3 Ecosystem

The virtual ecosystem is a bio-inspired and nature-inspired computational system. An ecosystem typically is composed of an environment and agents. It has a set of rules defining the behavior of the virtual world, the interaction between the environment and the agents and the interaction among the agents. The environment has its conditions or contains resources. The agents are capable of evolution via artificial evolution.

2.4 Related work

Some of ecosystem applications are related to the sonic performance. In Living Melodies [13], creatures have a procedure genome called “sing”. The Living Melodies generates MIDI notes by letting creatures sing. Every creature has its “listening pleasure”, which is mapped to the amplitude of the note. If a singing creature has not heard any interesting notes for a while, its amplitude drops to 0. The listening pleasure is also important to reproduction. The creatures have to hear some music they like so they can be happy enough to mate.

In the Jon McCormack’s installation “Eden” [14], the ecosystem is visualized and projected on the screen. Audience can move in front of the screen, and the movements affect the biomass growth rate and genotype mutation rate. Therefore, the activities of agents are affected by the audience indirectly. The agents are able to sing and have different frequency and volume. If audience is attracted by a particular sound and move to the location of sound source, then the agent in that location has higher chance to survive. It is because the location has higher biomass growth rate caused by the audience.

General surveys and overviews can be found in [5] and [11]. These artificial life algorithms imitate behaviors of real lives in the real world. Usually in these algorithms,

there are agents that are capable of some designed actions and a two-dimensional environment (arrays) for agents to live. There are also some mechanisms for the evolutionary procedures. In general, in each application, there are some unique designs and modifications of the algorithms. The algorithms are not necessary to obey the real situations precisely in every aspect. Creators can add their imaginations in the system to build an alternative world, and create life-as-it-could-be as mentioned above. The behaviors and outputs of applications can be totally different because of these different designs based on the creators’ idea.

Chapter 3

ECO-CONDUCTOR

The Eco-Conductor is composed of three parts: the human interface, the virtual ecosystem, and the soundscape system. Arousal and valence are key attributes for the whole system. Users’ gestures are interpreted into emotions represented in the arousal-valence plane via the designed interface. These two attributes affect the state of the ecosystem. As the state changes, the soundscape moves accordingly. In other words, the movement of the soundscape depends on the changing of these two attributes indirectly. Figure 3.1 shows the framework of the Eco-Conductor.

Soundscape User interface

Ecosystem User＇s gestures

In chapter 4, soundscape design is introduced, including the description of the synthesis and arrangement design. The controllable features are specified. The structure of the designed ecosystem and the way how the ecosystem controls the soundscape are described in chapter 5. The detail of the visualized ecosystem is depicted in this chapter. In chapter 6, the specification of the user interface and the patterns of interaction will be described. In chapter 7, the framework of Eco-Conductor is reviewed and discussed.

Chapter 4

SOUNDSCAPE DESIGN

The soundscape system is implemented using Max/Msp. The soundscape is designed to vary in the arousal-valence plane, which is comprised of continuums of sounds from active to passive as well as from positive to negative. For our purpose, we create environmental sounds by synthesizing methods and alter their properties to express different sensations.

In our system, environmental sounds of nature and industrial noises are considered to express different sound textures and reside from positive to negative along the valence axis. In addition, sounds of the “weather” are also represented from positive to negative along the valence axis. The soundscape may offer the abstract images of an urban place in rainy day or countryside in sunny day, but does not offer any specific ideas of a specific city or countryside.

4.1 Nature to Noise

In our design, natural sounds such as the chirp of birds and insects are regarded as positive ones along the valence axis. On the other hand, industrial noises from machines or vehicles are regarded as negative ones. However, not directly using industrial noises in the system, the attributes of natural sounds are modified such that they are gradually transformed into machine-like noises. In this way, the continuum of sounds can be established by continuous changing of the attribute instead of using discrete sound sample files. This design could be useful for various applications. In video games, huge

amount of environmental sounds are needed to simulate the real environment as background sounds. If the games just play discrete sound files in a loop, the sound would become annoying after a while due to the lack of variations. If huge amount of sound files are used to increase the variety of background sounds, the cost and the required storage would be increased drastically. By using the design of the continuums of sounds, few sound files are needed, and variations of background sounds can be increased without much additional cost of storage, time, and money.

To create the positive feeling of the soundscape, we first choose the natural sounds of birds, frogs, and crickets. The major reason for our choices is that they are commonly considered as keynotes in real natural environment. By listening to these keynotes, users would have the feeling of being in a natural soundscape. Another reason for our choices is that the transient behaviors of their temporal envelopes are similar to ones from certain machine sounds. Therefore, we can transform these natural sounds to noisy sounds more effectively.

The transformation to noisy sounds is done by replicating first couple samples from natural sounds to a constant duration. The number of samples from natural sounds is coded by a “read ratio”. While the read ratio decreases, the number of repetitions of natural sounds increases in order to constitute a fixed-duration transformed sound (see figure 4.1).

Machine-like noises are often monotonic and similar to repetitions of transient sounds. Therefore, the smaller the read ratio is, the more similar our produced transformed sounds are to sounds from factories. At the most negative end, the read ratio is reduced to 1/256 of the original sample length

.

away from original natural sounds when the read ratio is decreased. Each transformation creates a sound event in our soundscape.

Figure 4.1: The read ratio of samples decreases when approaching machine-like noise.

This part of soundscape is made of three independent sound tracks along the timeline. In each track, sound events, which are initially chosen from multiple sound files (such as birds’, frogs’ and crickets’ chirping), are periodically played with silences in between. The starting time for these three sound tracks is randomized. The duration of silence in each track gradually increases while the soundscape is moving toward the passive end along the arousal axis (see figure 4.2). The fixed duration ranges from 1 to 5 times of the original sample length. The silence duration ranges from 200 to 3800 milliseconds. A random number from 0 to 500 milliseconds is added to silence duration to avoid regular periods which are not practical in the real situation. Furthermore, the overall amplitude is also changing proportionally to the arousal value. Table 4.1 summarizes the features in the transition from nature to noise.

Figure 4.2: The fixed duration and silence duration extend when the soundscape gets passive.

Table 4.1: Natural sounds

Feature Description Emotion effect

Read ratio

The number of samples from natural sounds

Valence

Pitch variation

The degree of pitch variation from the original pitch

Valence

Silence

The duration of silence between two events in one track

Arousal

Fixed duration

The duration of a sound event in one time.

Arousal

Another natural sound we choose is sounds of ocean waves. We found that amplitudes of sounds of ocean waves and passing vehicles share similar temporal dynamics which show lots of temporal irregularities. We implement the granular synthetic method by the Sakonda Nobuyasu’s MSP Granular Synthesis patch v2.5 [15] to morph sounds of ocean waves. By using granular synthesis, the pitch and playing rate of the sound can be controlled separately. The pitch of original ocean waves is lowered and raised at the same time to produce two transformed ocean sounds.

The pitch is scaled both from 1 to 0.4 and from 1 to 1.6 times for simulating two passing vehicles with different pitches from the original sound.

The Doppler Effect becomes obvious and the decay rate of its temporal envelope increases to simulate sounds of passing vehicles.

The pitch is dropped when the sound is played past the middle point. The more the pitch is dropped, the faster the speed of the simulated moving-away object is. The pitch is dropped to 0.7 times of the original pitch at the most negative end.

The rise of the decay rate is also for simulating the passing speed. At the most positive end, the decay duration is half of the duration. The decay duration is reduced to 0.3 times of the duration at the most negative end.

In addition, the playing rate of samples is raised when the soundscape getsaroused. The rate is scaled from 0.5 to 2. Table 4.2 summarizes the features of the transition from ocean waves to the sounds of passing vehicles.

Table 4.2: Ocean waves

Feature Description Emotion effect

Pitch The increase and decrease of pitch Valence

Speed The speed of sample playing Arousal

Doppler Effect

The simulation of passing object Valence

Decay rate

The decay rate of the amplitude of sounds

Valence

4.2 Weather

Sounds of wind and rain depict the weather of the environment and are important keynotes for our soundscape. We follow the tutorial in [16] to create the sounds of wind, rain, and thunder by the synthetic method and choose parameters for transforming the sounds. It is due to the fact that these sounds can hardly be recorded without interferences from other sounds in the real environment. Only sounds of breeze are at the positive end of the valence axis. When the soundscape moves toward the negative end, the wind gets stronger, rain appears and gets stronger as well. Sounds of the thunder also appear frequently toward the negative end of the valence axis. Table 4.3 summarizes the transition of the weather.

Table 4.3: The weather

Feature Description Emotion effect

Wind Light breeze to fierce wind Valence

Rain

Starts at middle of valence and gets stronger when getting closer to the

negative end

Valence

Thunder

Appears more frequently while the rain gets stronger

Valence

Amplitude The overall amplitude Arousal

When the soundscape moves to negative end of valence axis, the pitch of wind is raised, the possibility of the “howling” of wind is raised, and the blowing rate of wind is also raised.

The rain consists of three parts: high pitch rain, low pitch rain, and raindrop. From positive to negative end of valence axis, raindrop appears first, then stronger rain with high and low pitches appear successively.

4.3 Overall soundscape

These principles to produce the soundscape are highly pertaining to the arousal and valence axes. The soundscape receives multiple arousal and valence values from the ecosystem. These values are assigned and mapped to each feature of the sounscape separately based on the emotion effect (arousal or valence). The arousal and valence values range from -1.0 to 1.0. Therefore, mappings between emotional values and

features of sounscape are established.

The sound of weather and common environmental sounds provide basic information of the environment. Users cannot tell where the exact location is, but they can a sense of how the location feels like with corresponding states of the location. Our intention to construct a virtual sounscape does not refer to anywhere, but to an imagination.

When the soundscape is positive, users can hear the natural sounds of ocean, birds, frogs, crickets, and also breeze. These sounds are usually heard in a natural environment as keynotes, that are, the background sounds. The faint breeze implies a peaceful weather. The less transformed sounds of ocean wave, birds and insects imply a peaceful and harmonious environment and provide users the feeling of belonging to nature.

When the soundscape is negative, users can hear different kinds of noises that are hugely transformed from the environmental sounds. The fierce wind, rain, and thunder mimic a bad weather. The noises imply an industrialized community. Under the combination of bad weather and noises, users have the feeling of being in a disordered and inharmonic environment. The temporal and spectral features of the environmental sounds are altered by the methods mentioned above to make deviated sounds.

The continuums of sounds are the key idea in the soundscape design. Various states of background can be built by only a few sound and synthetic methods. Furthermore, the smooth variations through this design can hardly be achieved by using discrete sound samples.

The use of soundscape is the main idea of this research. In many researches, music is designed to convey emotions. There are many researches that focus on the relationship between music theories and emotions or between music performance and emotions such as [17]. However, few researches were done on the relationships between soundscape and

emotions. There are a lot of interactive music projects, but, only a few interactive soundscape projects were done. We do experiments on the soundscape and combine the idea of emotional plane (arousal and valence) with it for interaction, in hope of creating different views on interactive sounds.

In the following section, we describe how the soundscape is set by the virtual ecosystem.

Chapter 5

VIRTUAL ECOSYSTEM

The virtual ecosystem we designed contains three roles: the environment, agents and the dominator which is controlled by the user. The ecosystem has its regulations to build the virtual environment similar to ecosystems implemented in other works [5]. In this section, we describe the regulations of our ecosystem and functions and attributes of each role.

5.1 Environment

The environment is a two-dimensional space where agents and the dominator live. We set 256 x 256 matrix as the environment. Every location in this virtual space has its own attributes such as states of arousal and valence. This design is analogous to the real world that every location has its living conditions such as temperature, humidity, and air quality. The attributes are affected by the dominator, and the population density, which encodes the population of agents around that location. These attributes would also affect the activities and the health of near-by agents just like the temperature and humidity would affect local living creatures in the real world.

5.2 Agents

Agents are autonomous entities that are capable of actions. The goals of each agent are simply to survive and to reproduce. Every action taken by agents is for the goals.

5.2.1 Fitness evaluation

The fitness evaluation is an important mechanism in the ecosystem. It calculates the difference of arousal and valence values among agents when they look for mates. It also calculates the difference of arousal and valence values between agents and agents’ locations, and the population density around their locations. The higher the difference and the density of the location are, the lower the fitness value is. The mutation possibility is inversely proportional to the fitness value. The environment affects the health of agents judged by the fitness value. If the fitness value of a certain location is lower than the fitness threshold, the health of the agent at that location decreases. Therefore agents always look for the fittest locations.

The fitness evaluation between agents can be expressed by the equation :

1 2 1 2

(

(

[0] -

[0])

(

[1] -

[1])) * 25

(100 -

)

-[0] :

'

[1] :

'

differnce

abs Agent

Agent

abs Agent

Agent

fitness

difference

densitypenalty

Agent

agnet s arousal value

Agent

agent s valence value

=

+

=

The fitness ranges from 0 to 100. Therefore, the range of the sum of the differences is scaled to match the range of the emotional plane. When the difference between agents is lower, the fitness value is higher. When considering the fitness between an agent and the location of the environment, the above equation is still used by replacing an agent with the location. The density is also added in the evaluation:

, 1 , 1

(

(

[0] -

[0])

(

[1] -

[1])) * 25

30

0

(100 -

)

differnce

abs Loc

Agent

abs Loc

Agent

if density exceed the threshold of endurance

densitypenalty

else

densitypenalty

fitness

difference

densitypenalty

=

+

=

=

=

The density of the location is to calculate the number of agents around the location. A 5 x 5 matrix around the location is examined to calculate the density. We set 5 agents in the 5 x 5 matrix as the threshold of endurance. If the density exceeds the threshold, a density penalty is added to the calculation of fitness.

5.2.2 Attributes

The attributes of agents are age, health, energy, arousal and valence. The intrinsic arousal and valence values of each agent are inherited from his two parents via crossover with certain mutation. Every action costs the energy of the agent. If the energy drops to zero, the agent has to rest to recover its energy.

(1) Health: The health point is 100 at the beginning. When the fitness value between an agent and its location are lower than the fitness threshold, the health of the agent is decreased by 1 periodically. If the situation is not changed, the agent would die eventually with zero health point. Therefore, agents must find a way out to survive. On the other hand, if the fitness is higher than the fitness threshold, then the health point is increased by 1 periodically until the health recovers to 100.

(2) Age: the age of the agent is increased by 1 periodically. The limit of an agent’s age is set when the agent is born. Agents die when they reach their limits of age no matter their health conditions. Every agent has different limit of age so that they will not die at the same time.

(3) Energy: The energy point is 100 at the beginning. Every time an agent takes an action, its energy is decreased by 1. If the energy is 0, then the agent cannot take any actions and must take a rest to regain its energy.

5.2.3 Actions and decision flow

Agents can move, rest, and mate in the environment. The movement of the agent is decided by its attributes and the attributes of its location. First of all, the agent finds the fittest mate within a distance determined by the difference of arousal and valence values. If it finds a fitted one, it moves closer to the target to mate. Location where they mate must be with a high fitness value for both of them. Every agent can only reproduce once. Every reproduction generates 2 offspring. After the reproduction, their goal is only to survive. Hence, the agents move to the fittest area within a distance from their current locations. Every turn, every agent makes decision and takes an action. The actions consist of:

(1) Finding mate: If an agent have not reproduced and is not ready to reproduce with its mate, then the agent will find a mate under the condition that its energy must be greater than 0 and health must be greater than 20. The agents find the fittest mate through the fitness evaluation mentioned above within their vision range. Every agent has the same vision range of a 60 x 60 matrix around them.

must not have another mate or have reproduced before. The fitness value between the agent and its target must exceed the match threshold. This is first checked in the decision flow. If the agent finds a mate, then it enters the mating process block. Otherwise, it chooses to move to a fitter place. Figure 5.1 shows the mate finding diagram.

(2) Resting: If the agent’s energy is 0, then it has to rest to recover its energy. (3) Moving to the fitter place: The agent finds a fitter location than its current one by

using the fitness evaluation mentioned above provided it does not need to find a mate or it cannot find a mate. Figure 5.2 shows the decision diagram.

Find the agents within the vision range as

the candidates

Choose the fittest agent in the candidate list Filter out the candidates

that have already had a mate or has reproduced

before

Filter out the candidates that the fitness values between the agent and the

candidate do not exceed the match threshold

Not ready to reproduce & Have not reproduced &

Energy >0 & Health>20 Find a mate Yes Energy = 0 No Rest Mate confirmation process Yes

Move to a fitter place

Yes

No No

Figure 5.2: The decision diagram of agents.

(4) Mating process: If an agent finds a mate within the vision range, the decision flow enters mating confirmation process. First, it checks the distance from the mate. Their distance must be lower than 5 so that they can mate and reproduce their offspring. If the distance exceeds 5, they must come closer to each other. The other condition is that the couple has to reproduce in a fitted location. That is to say, the fitness value between the location and both of the couple must exceed the fitness threshold when reproducing. The mating process is shown in figure 5.3.

(5) Reproduction: The reproduction costs 5 turns. After 5 turns, two agents are born and located near their parents. Within the 5 turns, if one of the couple’s fitness values to the location decreases to be lower than the fitness threshold, the reproduction process is terminated. This can happen due to the interaction from the dominator or variation of the density of the location. If one of the couple finds a fitter mate during the reproduction, the reproduction is terminated as well. This can happen because some agents come into their vision range. If the reproduction completes, both of the couple cannot reproduce again. Their goal lasts in their life is to find the best fitted place. The figure 5.4 shows the reproduction process.

The mutation degree is 0.5. If the inherited arousal value is 0.1, then the mutated value can range from -0.4 to 0.6 after mutation. The reproduction process uses the idea of evolutionary biology such as inheritance, mutation, and crossover. The arousal value of the offspring is chosen randomly from one parent and the valence value is chosen from the other parent.

If Fitness(Mate,Loc(Mate)) > fitness threshold

If the distance exceeds 5

Yes

Reproduction process No

If Fitness(Agent, Loc(Mate)) > fitness threshold Move to the mate＇s location Yes Yes IF Fitness(Agent, Loc(Agent)) > fitness threshold And Fitness(Mate, Loc(Agent)) > fitness threshold No

Rest Move to a fitter place Yes No If Fitness(Agent, Loc(Agent)) < fitness threshold Yes No No

The mutation possibility is inversely proportional to the average of the fitness of the parents. The equation computes the mutation possibility:

1 1

2 2

100 (

(

,

(

))

(

,

(

))) / 2

mutation probability

fitness parent Loc parent

fitness parent Loc parent

=

−

+

If find a fitter mate or Fitness(Parent1,Loc(Parent1))

< fitness threshold or Fitness(Parent2,Loc(Parent2))

< fitness threshold or One of the couple＇s energy<5

Yes No If countdown equals 0 Reproduction Yes Decrease by 1 No

Back to the mating process

In the final step of each turn, every agent’s health and age are checked. If the fitness value between the agent and the environment is lower than the fitness threshold, then the health of the agent is decreased by 1. If the health becomes 0 or the agent reaches its limit of age, then it dies.

5.3 Dominator

The user plays as the dominator in the ecosystem. The dominator changes the attributes within its influence range. The dominator can be considered analogous to the whirlwind in the real world. Similar to agents, the dominator also has its attributes of arousal and valence. The attribute values decrease from the center to the boundary of its influence range. If a location is not covered by the dominator’s influence range, the attributes of the location remain neutral. Because the dominator changes the attributes of the environment and agents always look for the fittest locations, the dominator either attracts or repels the agents.

The influence of the dominator decreases to 0 from the center to the boundary, that is, both the arousal and valence values approach to 0. Meanwhile the fitness threshold is 70 % so that each point of the dominator has the influence range that its radius is 0.3 on the arousal-valence plane. Therefore, every point in the influence range of the dominator can be fitted for the agents which Fitness(Agent, Loc(point))>70.

We can find the agents with extreme attributes have less chance to be attracted by the dominator. Furthermore, unaffected area’s attributes are neutral so that agents with attributes that are close to neutral have better chance to survive. Therefore, if the user do not create comfortable environment for these extreme agents, the agents would die faster. This situation is similar to the real situation that some creatures can only live in an

extreme condition.

5.4 Game strategy

This design of the ecosystem can also be considered as a game strategy. In common interactive music system, users control the music using controllers or sensors. The varying music is based on how users move or gesture directly. Users can easily make sounds they want in those systems if they know the direct mapping rules. In our Eco-Conductor, the users have to face the challenge in the ecosystem so that they can have the chance to have the sounds they want. Because not all agents can see the area affected by the dominator, the user has to move the dominator to attract the agents they want. Meanwhile, the user has to make gestures to change the attributes of the dominator to affect the environment indirectly. Therefore, the output does not only depend on the gestures, but also depend on the way the users move the dominator in the ecosystem to conquer the challenge.

5.5 Ecosystem to Sound

The most important feature of our ecosystem different from other ecosystems is its collectivity. The objective of our ecosystem is to map users’ gestures to the soundscape. All the mechanisms of the ecosystem described in previous sections are for the purpose of an indirect and elaborate mapping. We can say the agents within the influence range of the dominator have similar attributes as the dominator because only suitable agents can stay within the influence range. Therefore, the arousal and valence values of agents are mapped to the soundscape system instead of attribute values of the dominator.

possibility is inversely proportional to the distance from its location to the center of the dominator. The shorter the distance is, the more similar the agent is to the dominator. We build a probability distribution table based on those distances between agents and the center of the dominator. One agent is mapped to one feature of the soundscape. The sound generated at any time instant is composed of n parameters {P1, P2, …, Pn} from the arousal/valence plane. Each parameter is extracted from the parameters of a randomly chosen agent within the influence range of the dominator. Therefore, the sound is “collectively” determined by the agents.

The main reason to adopt current strategy instead of using values of the dominator directly is to increase the variety and complexity of the soundscape. Therefore, the overall performance of the soundscape does not merely depend on one pair of arousal and valence values of the dominator. Instead, the agents within the influence range of the dominator collectivelyput up the performance. Meanwhile, the performance reflects the state of users’ gestures because the dominator is affected by users’ gestures. The mechanisms in the virtual ecosystem are not only to emulate the real ecosystem, but also to produce the soundscape based on users’ intentions.

Fitness evaluation can eliminate the agents that users do not want, because the dominator can repel the unfitted agents by the fitness evaluation. The design of reproduction ensures the evolution where agents have the chance to reproduce their offspring to be more fitted for the dominator via the mutation and the crossover.

The fitness threshold is important to the outcome of the soundscape. If the fitness threshold is too low, users will find it hard to have their own ways to control the audio output. It is because the dominator attracts the agents with large range of attributes. It becomes difficult to interpret the user’s gestures, and the variation of the soundscape will get higher. If the fitness threshold is too high, the users can have explicit control of the

soundscape. But it would be hard to attract agents and the variation would become much lower. After tests, we set 70% as the fitness threshold, that is, the agents can have 30% difference from the environment.

5.6 Visual Presentation

One screenshot of the visualized ecosystem is shown in figure 5.5. The small dots represent the agents and the large circle represents the influence range of the dominator.

The color map is based on [18]. The colors of agents and the dominator present different pairs of arousal and valence values. The color is coded in the HSB color mode. HSB stands for hue, saturation, brightness. The color mapping is shown in figure 5.6. The brightness gets higher from low valence to high valence. The saturation gets higher from neutral to the extreme of the emotion. Through colors, users can identify how the agents’ attributes are similar to the dominator’s attributes in the visualized ecosystem.

The motion graphics of agents and the dominator are produced using generative methods instead of the fixed animations. The shape of the dominator varies in a repetitive manner over time. The virtual creatures (agents) have pairs of wings which rotate over time as well. The variation rates of the shape of the dominator and the rotation of agents’ wings are pertaining to their own arousal values. The rotation of the agents’ wings is shown in figure 5.7.

Figure 5.7: Agents with different arousal values have different speed of rotation.

The left side shows slower rotation for lower arousal value. The right side shows faster rotation for higher arousal value.

The shape of the dominator is transformed while moving. Figure 5.8 shows the different types of the transformation. In the figure 5.8, the left column shows that the dominator is moving left or right, and the right column shows that the dominator is moving up or down. The middle shows that the dominator stays still.

By using the generative method, complex motion graphics can be created. Instead of making fixed animations in advance, the motion of the graphics can be defined by lines of program code. Sometimes, the motion is hard to be drawn by hand because the performance is too complex. The motion of the dominator can be defined as the following code:

userrotation+=(int)(user.arousal*10.0+12); // the amount of rotation each turn userrotation=userrotation%360;

radian=radians(userrotation); //covert degree to radian for(int a=1;a<17;a++){ //16 circles

float inc=(float)(TWO_PI/(100.0-a*3)); //interval between curves

fill((int) thecolor, 210 - a * 10, (int) (70 * user.valence + 110), 80); // set color for(float i=0;i<TWO_PI;i+=inc){

beginShape(); //each curve is defined by 4 curve vertex

//*3: the matrix is scaled from 256x256 to the image size 768x768

curveVertex(user.position_X * 3, user.position_Y * 3); curveVertex(user.position_X * 3 + 10 * a * cos(i),user.position_Y * 3 + 10 * a * sin(i));

if (morph == 0) //stay

curveVertex(user.position_X * 3 + 10 * a * cos(i) + 15* cos(i + radian), user.position_Y * 3 + 10 * a* sin(i) + 15 * sin(i + radian));

else if (morph == 1) //up or down

curveVertex(user.position_X * 3 + 10 * a * cos(i + radian)+ 15 * cos(i), user.position_Y * 3 + 10 * a* sin(i) + 15 * sin(i + radian));

else //left or right

curveVertex(user.position_X * 3 + 10 * a * cos(i) + 15* cos(i + radian), user.position_Y * 3 + 10 * a* sin(i + radian) + 15 * sin(i));

curveVertex(user.position_X * 3, user.position_Y * 3); endShape();

} }

The dominator is composed of 16 circles from the center to the edge. Each circle is composed of different number of curves. The outer circle has the least number of curves. Each turn every curve is rotated by little amount of radians to create the motion of the circle. The amount is proportional to the arousal value of the dominator. The type of the transformation can be easily swapped. As shown in the code, the variable ‘morph’ can be easily swapped by replacing one line of program code, to have the totally different motions. The color setting is also inserted in the loop. The saturation gets lower when drawing outer circle as shown in figure 5.8.

The motion of the agents is also created by a generative method. Because the rotation speed of each agent’s wings is different, many fixed animations have to be created in advance if the generative method is not used.

The system is built using Java, and Processing is used for the visual presentation. Initially, the system generates agents with random positions. Their attributes are equally distributed on 4 quadrants of the arousal-valence plane. Then, the arousal and valence values of selected agents are sent to Max/Msp via Open Sound Control.

The user can observe the agents run to or away from the dominator for every action following principles described in previous sections. In this way, users can interactively affect the ecosystem. Because the artificial lives in the mapping platform are visualized on the screen, the mappings are no longer behind the scene but become a vivid visual presentation. The usage of the ecosystem does not only increase the variety and the

complexity of the soundscape, but also provides a gaming experience. The users can learn how to attract agents they want and how their gestures affect the ecosystem by watching the visual display as well as by listening to the audio.

Chapter 6

USER INTERFACE AND SYSTEM ARCHIETECURE

6.1 User interface

To control the dominator in the ecosystem, we design a scepter as the user interface. The user, as the dominator, can conduct the ecosystem using the scepter and his gestures are interpreted into the arousal-valence plane. The scepter is shown in figure 6.1. It contains an Arduino microcontroller, a 3-axis accelerometer, a force sensing resistor (FSR) and a switch button. The accelerometer detects the direction of waving to control the movement of the dominator in the ecosystem. The force sensing resistor detects the pressure from the user. The switch button is placed at the bottom of the scepter. Every time the scepter hits the ground/surface, the switch is trigger once in order to detect the tempo of hitting. The pressure and the tempo are interpreted into the valence and arousal, respectively. The valence gets negative when the pressure gets higher. The arousal gets active when the tempo gets faster. These arousal and valence values are the attributes of the dominator. The mapping is not only for intuitive control, but also complying with the role of the dominator. The actions of pointing the direction and hitting the ground by using the scepter are suggestive of a ceremony where a chief or a king holds the scepter.

Figure 6.1: The user interface, Scepter. The switch button is at the bottom. The FSR is on the

surface. The accelerometer and the Arduino microcontroller are inside the scepter.

6.2 System Architecture

The system architecture is shown in figure 6.2. The system except for sound synthesizer is built using Java. Especially, Processing is used to build the visual presentation. Serial handler reads the values from the sensors and interprets them into arousal/valence values of the dominator. Sound handler sends arousal/valence values acquired from the ecosystem to sound synthesizer via Open Sound Control. The sound synthesizer is made using Max/Msp.

Serial Handler

Ecosystem

Visual

Presentation

using

Processing

Sound Handler

Sensors connected with Arudino

Sound Synthesizer in Max/Msp

Serial

Open Sound Control

in Java

Chapter 7

CONCLUSION

In this work, we produce an interactive soundscape by adopting a virtual ecosystem as the mapping platform. The soundscape does not imitate any places in the real world. Instead, it is a virtual sound space comprised of certain nature sounds and synthetic sounds of the weather. Although the sounds we create are concrete, the overall soundscape generated from our interactive mappings can be imaginary with users’ creativity.

The interaction strategy we adopt is not a direct mapping from users’ gestures to parameters of sounds or music, but through an ecosystem for indirect mappings. The user, the dominator in the ecosystem, indirectly affects the soundscape by the collectivity of features of surrounding agents. In our work, agents in the ecosystem make sounds collectively, not like in other works where every agent or virtual creature makes sound individually.

Our interactive interface is designed to fit in the role of the dominator. It symbolizes the supreme power in the ecosystem. The visual display shows how the power is exerted and affects the environment. The state of the interaction is shown both in visual and in audio outputs.

In the Eco-Conductor, generative methods, which can produce surprising and amazing results are used in both audio and visual demonstrations. Usually, the results can be too complex to be created by hand. While pieces of fixed animations or discrete sound files are looped to produce limited variations, a well-designed generative method can

generate infinite results. The creation of a generative procedure can be regarded as the creation of art in an indirect way. Generative methods stem from a promising field of study as computers become more and more powerful. Many complex methods can be executed in real-time and bring amazing results.

We hope this work can provoke attentions to the soundscape of the environment we live in. In the future, we expect more creative art works involving usages of artificial lives in a virtual world.

REFERENCES

[1] Truax, B., “Genres and techniques of soundscape composition as developed at Simon Fraser University.” Organised Sound 7(01): 5-14, 2002.

[2] Russell, J. A., “A circumplex model of affect.” Journal of Personality and Social Psychology 39(6): 1161-1178, 1980.

[3] Blackwell, T. I. M., “Swarming and Music.” Evolutionary Computer Music: 194-217, 2007.

[4] Guéret, C., N. Monmarché, et al., “Ants Can Play Music.” Ant Colony, Optimization and Swarm Intelligence: 310-317, 2004.

[5] Dorin, A., “A Survey of Virtual Ecosystems in Generative Electronic Art.” The Art of Artificial Evolution: 289-309, 2008.

[6] Schafer, R. M., Our Sonic Environment and The Soundscape: the Tuning of the World.

Rochester, Vermont: Destiny Books, 1994.

[7] Truax, B., “Soundscape Composition as Global Music: Electroacoustic music as soundscape.” Organised Sound 13(02): 103-109, 2008.

[8] Westerkamp, H., “Linking soundscape composition and acoustic ecology.” Organised Sound 7(01): 51-56, 2002.

[9] Langton, C. G., “Artificial Life”. Artificial Life, Volume VI of SFI Studies in the Science of Complexity, edited by Langton, C. G., Redwood City: Addison-Wesley, 1989.

SFI Studies in the Science of Complexity, vol. X, edited by Langton, C. G., Taylor, C. E., Farmer, J. D., Rasmussen, S., Addison-Wesley, 1991.

[11] Etxeberria, A., “Artificial Evolution and Lifelike Creativity.” Leonardo 35(3): 275-281, 2002.

[12] Heudin, J. C., Artificial life and evolutionary computing in machine perception. Computer Architectures for Machine Perception, 1995. Proceedings. CAMP '95, 1995.

[13] Dahlstedt, P. and M. G. Nordahl (2001). “Living Melodies: Coevolution of Sonic Communication.” Leonardo 34(3): 243-248.

[14] McCormack, J., “The Evolution of Sonic Ecosystems.” Artificial Life Models in Software: 393-414, 2009.

[15] Nobuyasu, S., http://web.me.com/nsakonda/sakoweb [16] Farnell, A., Practical synthetic sound design,

http://www.obiwannabe.co.uk/tutorials/html/tutorials_main.html

[17] Juslin, P.N., Sloboda, J.A. (eds.) Music and Emotion: Theory and research, pp. 309–337.

Oxford: University Press, Oxford, 2001.

[18] Fagerberg, P., A. Ståhl, et al., “eMoto: emotionally engaging interaction.” Personal and