Currently, the adaptive Musification prototypes designed for Chinese Classical Poetry and Chinese Calligraphy Painting are proposed. General conclusion is that the sounds produced in each experiment convey the information about the imagery state of mind and the qualitative nature of the data.
For Text-to-Music conversion:
z Not only the arrangement of text but also the pronunciation properties and the syntactic characteristics of the poem are conveyed in the music output.
For Image-to-Music conversion:
z The position-to-pitch mapping is more intuitively responsive to original visual data and easy for gestalt formation than color-to-pitch applied in the two related works. However, color could be mapped into timbre instead.
z Notwithstanding the two parameters are taken into account (i.e., position and intensity), the two-way scanning results in an extra musical effect — the sonority. To sum up, the texture of the image in both horizontally and vertically sequential scanning reflects on the sonority of the music.
Many interesting applications could be realized based on this study, such as an Audible Digital Album (let the photos sing!), or an Immersive Chinese Classical Poetry E-Learning
Platform. Nevertheless, the actual resolution obtainable with human perception of these sound representations remains to be evaluated, and the algorithmic composition throughout the Musification process need more improvements. The involvement of expertise in poetry composition (Chinese Classical Poetry Analysis), image processing (Chinese character Recognition), music composition, and even psychology is critical for its success.
Although this study has systematically investigated the logical and reasonable mappings from the degrees of freedom in the data to the parameters controlling the algorithmic composition or sound synthesis process, there are still few limitations of this study. The most obvious one is the lack of strokes sequential information in the Im2Ms.
The sequential strokes of a Chinese character play a significant role in this kind of specific image as an important feature itself. Consequently, there might be an alternative demand for a real-time and interactive Musification for Chinese Calligraphy Painting. Mouse, write pad, or other related input devices could be used to obtain more image information, such as the sequence of the character, instead of simply horizontal and vertical scanning. Take the following idea for example. Since the writing sequence is based on the “arrow”, the writing segments are then retrieved for sections of music, with “rest” based on the timing between the end of the last segment and the beginning of the next segment. Simply speaking, the scanning sequence is no longer the pure left-to-right or top-to-bottom, but the real-time writing strokes recorded sequentially. In this way, the image content could be mapped into music, where the vertical axis variance determines the pitch in Pentatonic Scale up or down and the horizontal axis variance determines the timbre (see Fig. 30).
Fig. 30 An exemplified Calligraphy Musification algorithm with FM
References
[6] Ballas, J. A. 1994. Delivery of Information Through Sound. In Auditory Display: Sonification, Audification and Auditory Interfaces, ed. Gregory Kramer, 79–94. Reading, MA: Addison-Wesley.
[7] Barrass, S., and Kramer, G. 1999. Using sonification. Multimedia Systems 7, no. 1 (January), http://www.springerlink.com/content/xd19ftjpfe3pb30l/fulltext.pdf (accessed May 9, 2009).
[8] Blattner, M. M., Papp III, A. L., and Glinert, E. P. 1994. Sonic Enhancement of Two-Dimensional Graphics Displays. In Auditory Display: Sonification, Audification and Auditory Interfaces, ed. Gregory Kramer, 447–470. Reading, MA: Addison-Wesley.
[9] Clement, B. J. 1998. Learning Harmonic Progression Using Markov Models.
http://www-lrn.cs.umass.edu/lab-lunch/papers/clement98learning.pdf (accessed May 10, 2009).
[10] Edlund, J. 2004. The Virtues of the Musifier: A Matter of View. InterAmus Music Systems.
http://www.interamus.com/techTalk/musificationAndView.html (accessed May 9, 2009).
[11] Ekman, R. 2003. Coagula – Industrial Strength Color–Note Organ.
http://hem.passagen.se/rasmuse/Coagula.htm (accessed May 9, 2009).
[12] Farbood, M., and Schoner, B. 2001. Analysis and Synthesis of Palestrina-Style Counterpoint Using Markov Chains. Paper presented at the International Computer Music Conference, September 18-22, in Havana, Cuba.
[13] Franz, D. M. 1998. Markov Chains as Tools for Jazz Improvisation Analysis. Master thesis, Virginia Polytechnic Institute and State University.
[14] Kandinsky, W. 1977. Concerning the Spiritual in Art. Trans. M.T.H. Sadler. New York: Dover Publications.
[15] Kaper, H. G., Wiebel, E., and Tipei, S. 1999. Data Sonification and Sound Visualization. Computing in Science and Engineering 1, no. 4 (July/August),
http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=774840&isnumber=16814 (accessed May 9, 2009).
[16] Kramer, G. 1994a. An Introduction to Auditory Display. In Auditory Display: Sonification, Audification and Auditory Interfaces, ed. Gregory Kramer, 1–77. Reading, MA: Addison-Wesley.
[17] Kramer, G. 1994b. Some Organizing Principles for Representing Data with Sound. In Auditory Display:
Sonification, Audification and Auditory Interfaces, ed. Gregory Kramer, 185–221. Reading, MA:
Addison-Wesley.
[18] Kramer, G. et al. 1999. Sonification Report: Status of the Field and Research Agenda. Report prepared for the National Science Foundation by members of the International Community for Auditory Display.
http://www.icad.org/websiteV2.0/References/nsf.html (accessed May 9, 2009).
[19] Marino, G., Serra, M.-H., and Raczinski, J.-M. 1993. The UPIC System: Origins and Innovations.
Perspectives of New Music 31, no. 1 (Winter), http://www.jstor.org/pss/833053 (accessed May 10, 2009).
[20] Meijer, P. B. L. 1992. An Experimental System for Auditory Image Representations. IEEE Transactions on Biomedical Engineering 39, no. 2 (February),
http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=121642&isnumber=3463 (accessed May 9, 2009).
[21] Moore, F. R. 1990. Chapter Five: Composing. In Elements of Computer Music. Englewood Cliffs, NJ:
Prentice-Hall.
[22] Osmanovic, N., Hrustemovic, N., and Myler, H. R. 2003. A Testbed for Auralization of Graphic Art.
Paper presented at the IEEE Region V 2003 Annual Technical Conference, April 11, in New Orleans, Louisiana.
[23] Saue, S. 2000. A Model for Interaction in Exploratory Sonification Displays. Paper presented at the International Conference on Auditory Display, April 2-5, in Atlanta, Georgia.
[24] Scaletti, C. 1994. Sound Synthesis Algorithms for Auditory Data Representations. In Auditory Display:
Sonification, Audification and Auditory Interfaces, ed. Gregory Kramer, 223–251. Reading, MA:
Addison-Wesley.
[25] Winsor, P. 1992. Automated Music Composition. Denton, TX: University of North Texas Press.