Analyzing Multi-Source Social Data for Extracting
and Mining Social Networks
I-Hsien Ting
Department of Information Management National University of Kaohsiung
Kaohsiung City, Taiwan iting@nuk.edu.tw
Hui-Ju Wu
Institute of Human Resource Management National Changhua University of
Education Changhua, Taiwan d94311001@mail.ncue.edu.tw
Pei-Shan Chang
Department of Information Management National University of Kaohsiung
Kaohsiung City, Taiwan purplemio@gmail.com
Abstract—In recent years, social computing has become a very
popular application in the Internet, and therefore large amount of social (communication) data has been collected in different social computing application. This paper will introduce a methodology to collect and analyze multi-source social, and by this for extracting social networks from the data. A system architecture will also be presented in this paper to show how the data can be collected, pre-processed, analyzed. Furthermore, the system will allow the users to use the data as a resource for personal decision support.
Keywords-Social Networking; Instant Messenger; E-mail; Data Mining; Social Network Analysis
I. INTRODUCTION
With the rapid growth of Internet and communication technologies, there are many communication and social activities of people have been transferred to Internet-based platform, e.g. e-mail communication, instant messaging software and social networking websites (such as Blog and web albums), etc. [8] Under this background, large amount of personal communication and social data has been aggregated and stored in different locations [15]. However, these valuable data have not been well organized, treated and used. Thus, it is an interesting research issue about how to use current information techniques to process and analyze these data, such as artificial intelligence, data mining or visualization technique [18].
Social network analysis and construction are originally in the research fields of Sociology. In recent years, many research issues of information science and social networking have been concerned due to the development of information techniques and the requirements of data processing ability [13]. The target of social network analysis and construction is relationship data and it is therefore suitable to process and analyze communication and social data that discussed previously [7].
Since the communication data, such as e-mails and the logs of instant messenger, are very common data in our daily life. However, there is less work focusing on how to organize,
process and analyze these data [31]. In this paper, we therefore will propose method to analyze the common social data and discuss how to extract social networks from multi-sources social data dynamically. Furthermore, this paper will propose a system architecture to use the three techniques of social network analysis, social network construction and visualization to process and analyze those valuable data. The system will allow user to input tasks for dynamic social network analysis and construction and the final results will be presented by visualized mean and interface for decision support.
The structure of this paper is organized as below: In section 1, the background and introduction will be introduced. Some related literatures of social network extraction, social network and data mining and social network analysis will be reviewed in section 2. A system architecture about how to extract dynamic social networks from multi-sources data will be proposed in section 3 as well as the introduction of the components in the system. In section 4, we will focus on how to extract social networks from social data and how to use data mining and AI techniques for decision support. In section 5, this paper will be concluded with the suggestions for future research.
II. LITERATURE REVIEW
In this section, related literatures will be reviewed, including social networks analysis, social networks extractoin and social networking for decision support.
A. Social Networks Analysis
The research methodology of social network analysis is developed to understand the relationship between “actors”, and the term actor can be a person, an organization, an event or an object [4]. In a social network, each actor is presented as a node and each pair of nodes can be connected by lines to show the relationships. The social network structure graph is a graph that formed by those lines and nodes, and social network analysis is therefore a methodology that used to understand the graph and the relationships and actors in the social network [11][34][27].
There are three important elements that included in a social network: actors, ties, and relationships. Actors are the essential elements in the social network to define the people, events or objects. Ties are used to construct the relationship between actors by using a mean of path to establish the relationship directly or indirectly. Ties can also be divided into strong tie and weak tie according to the strength of the relationships; they are also useful for discovering the subgroups of the social network. Relationships are used to illustrate the interactions and relationship between two actors. Furthermore, different relationships may cause the network to reflect different characteristics [32][33].
The most important measurements of SNA include network size, diameter, density, centrality and structure holes [5]. Size is a measurement to measure the amount of nodes or links in a network, and the measurement of diameter is to measure the amount of nodes between two nodes in a network. Density is used to calculate the closeness of a network [23][28]. These measurements are common used in many social network related researches and will be used in this paper as well.
Traditionally, researches about SNA are mainly focus on small group of actors and are process manually in most cases. [6] However, with the rapid growth of Internet and web techniques, more and more data have been collected and it has become a hard task to process these data by only the mean of manually [9]. Therefore, the scholars of information technology and computer science are starting to devote related researches to deal with these research issues [12][26]. Currently, the researches of computer science in SNA can be divided into four main topics, including social networks construction, social networks extraction, social networks analysis and visualization.[24]
B. Social Networks Extraction
In the research field of information technology and computer science in social networking, social networks extraction is a subfield focusing on extract social networks from large amount of communication data. With the rapid growth of Internet and WWW, there are various kinds of data have been generated due to communication purpose. The common used communication data such as email communication data, web usage logs, event logs, instant messenger logs, logs of telecommunication…, etc[22][29].
Currently, there are some researches which are focusing on the extraction of these social data. For example, Bird et al. propose a method to extract social networks from e-mail communications [3]. Agrawal et. al using web mining techniques to understand the behavior of users in newsgroup [2]. Web is considered as the biggest database in the world, so that various social networks can be extracted from this resource, such as Furukawa et al. were trying to identify social networks from blogspace [14][19] Jin et al. and Matsuo et al. developed systems and tried to extract social networks from the web [17] [21]. Adamic and Adar developed a method to discover the relationship of friends and neighbours in the web [1].
Most of the researches that discussed above are focusing on a single source for social network extraction. However, the issue of how to extract social networks from different sources has not been discussed well in related literatures. It is also a hard task about how to integrate multi-source data for social networking extraction. In addition to the problem of multi-source data, instant messenger is a very popular and hot software for people to send message and communication recently. However, it has not been seen in recent research about how to extract social networks from the data. These research issues will be discussed in this paper.
C. Web Mining Techniques for Social Networking
According to different analysis targets and resources, the web mining techniques can be divided into three different types, which are Web Content Mining, Web Structure Mining and Web Usage Mining [30].
Web content mining is a web mining technique to analyze the contents in the web, such as texts, graphs, graphics, etc [2]. Recently, most of web content mining researches are focused on the text data processing and few are focused on other multimedia data. Natural language process is therefore the main technology that used in this area. The concept and techniques of Semantic Web and Ontology also have to be studied [16][ 20].
Web structure mining is a technique that can be used to analyze the links and structure of websites [10]. Graph theory is usually the main concept and theory for web structure mining to analyze and explain the structure of websites. In addition, the extraction of the structure of websites is always essential in this research area [12]. Therefore, it’s usually the concern about how to design and implement a crawler (or spider, bots) to extract and construct the structure of websites, such as the research topic of Deep-web.
Web usage mining is a web mining technique that can be used to analyze how the websites have been used, such as the navigation behavior of users. The server-side Clickstream data (logs file) is the main sources that used for web usage mining. Client-side data (such as client-side logs file, cookies) is sometimes to be used due to some research concerns, such as in order to record more complete behavior of users. Different web usage mining analyses include basic statistical analysis of the navigation behavior of users in a website, such as how many times the website has been browsed, where the users comes from, etc. Furthermore, advanced web usage mining analyses can also be provided, such as more complex analysis for understand the navigation history of users in a website or cross-website analysis [25].
Figure 1. The architecture of the social network extraction system
III. SYSTEM ARCHITECTURE
According to the research background and motivation of this paper, we have designed a system architecture to addressed the raised issues. The system will allowed us to collect social data from different sources, such as e-mail, instant messenger and blog. The multi-source social data will then be pre-process and analyze. The processed data can be used to extract social networks automatically and dynamically. The system then can be further developed to a decision support system. However, this paper will not focus on the decision support system and only the methodology to collect multi-source social data, pre-processing and how to integrate the data for generating social networks, which is the main contribution of this paper. The architecture of the social network extraction system is presented in figure 1.
As shown in figure 1, the system can be divided to two major phases according to the characteristic of processing. The two phases are offline data collection and processing and on-line process. The elements and process of the two phases will be introduced in detail below.
A. Offline data collection and processing
The first phase of the system is mainly working offline, and there are three elements in this phase including multi-source social data collection, data extraction engine and a database.
1) Multi-source social data collection
This is the first step of the system. In this paper, we intend to collect social data which are most related to personal daily communication. Thus, three types of data will be collected including instant messenger data, email data and blog browsing data. The messaging history of MSN or other messengers will be recorded in a structural format, such as xml. The detail of the message contents and communication target and time will be stored in the file. In the paper, the history file of MSN messenger is used as the social data of instant messenger.
About data collection, a data collection system will be introduced in section 4 of the paper. The system is developed by web-based concept and it allows the users to upload email and the history file of MSN messenger by either automatically or manually mean. About the collection of blog browsing data, we use a client side agent to collect the navigation history of a use when using particular blog. With the ability of client side logging, the complete browsing data will be recorded without missing, such as the behavior of browsing, posting a message and responding to a message.
2) Data extraction engine
The second step of the system is data extraction engine. The engine will firstly used to process the data that collected from previous step. Then, useful data will be extracted and filtered out from the raw data. The detail of how the data will be processed and extracted will be discussed in section 4 of the paper.
3) Database
After data collection and extraction, the output of the data extraction enginer will then be stored in a database. The database is designed according to the characteristics of different sources of social data.
B. Online processing
The second phase of the system is a possible application of the paper in the future. The works in this phase are mainly processed online according to the data that collected offline. The elements of this phase will be introduced as follow even they are not the main focus of the paper.
4) Ontology-base
The second phase of the system is an possible application of the paper. In the system, the user can use the collected social data for personal decision support. It will allow the user to input a keyword and other parameters to ask for decision support. A social network for dealing with the problem will then be generated, which provides possible solutions for the user.
The ontology base in the system is used to illustrate the semantic of the keyword that use input. It is also a very important element of the system for social network extraction.
5) User input:
There are two essential user input of the system, including a keyword and other parameters. The parameters are limitation and condition for the system to scale down the extracted social network. For example, if the input keyword is “BBQ”, it means the user may want to create a network about BBQ from the social data to find out previous communications about the issue of BBQ in email, MSN or blog browsing. Then, the other parameters could be “red wine” and “relationship >2”. It means that the user want to find friends to attend the BBQ party. However, the participatns must attened to BBQ party before more then twice and bring red wine with them.
6) SNA engine:
After the inputting of keywords and parameters, the system will use SNA methods to get information for social networks construction, such as the nodes, relationships, closeness, etc. The detail of how to calculate the relationship value from the multi-source social data will be discussed in next section.
7) SN Construction engine:
The SN construction will use the results of SNA to prepare the visualization of the SN. For example, how much nodes will be used in the SN, the characteristics of the links, the detail of the network, etc. The gathered information will be used for SN construction and visualization.
8) Visualization engine:
In this research,the library of OpenGL will be used to visualize the network based on the information that provided by SNA engine and SN construction engine.
9) Dynamic and task-oriented social network:
Finally, the system will generate a social network based on the input of user, which is considered as a dynamic and task-oriented social network.
IV. DATA COLLECTION SYSTEM AND EXTRACTION
EXTRACTION ENGINE
In this section, the paper will focus on three main tasks of the system architecture that introduced previously. The three tasks include the data collection system, data extraction methodology and relationship calculation methods for social networks construction.
A. The data collection system
Data collection is the first step of the system, and therefore we design a sub-system for uploading related data. The system is a web-based system, and it allows the user to upload email file (in *.eml or text format), MSN history data (in *.xml format) and client side logging data (in *.log or text format). The format and file sample of email and MSN history data will be introduced below.
About email file, the system allows users to upload email file manually or automatically. Although different email agents
or servers may produce various email file, the system will accept any email file and there are some common fields in different email file format. Fields extraction of the uploaded email file will be discussed later in this paper to show which fields are useful for the research. Figure 2 shows a sample email file which is saved by an email agent and figure 3 shows the collected mail in the system.
Figure 2. A sample email file
Figure 3. The email collection system
In this research, only the instant messenger history file of MSN is accepted. The MSN history file is stored in a .xml file and based on the format of xml. Each contactor in the MSN contactor list has an independent history file, and the information that stored in the file include session ID, Date and Time, from, to and message content. Figure 4 shows a sample MSN history file that used in the paper.
Figure 4. A sample MSN history file
Return-path: <eri@xx.xx.xx.xx> Envelope-to: RSs@xx.xxxx.xx.xx
Received: from funnelweb.cs.york.ac.uk ([144.32.161.232] Message-ID: <47552CF4.70806@xx.xxx.xx.xx>
Date: Tue, 04 Dec 2007 10:33:24 +0000 From: E Rid <eri@xx.xxx.xx.xx> Reply-To: eri@xx.xx.xx.xx
User-Agent: Thunderbird 2.0.0.9 (Windows/20071031) MIME-Version: 1.0
To: RSs@xx.xxx.xx.xx
Subject: java versus C benchmarks
Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit
Status: RO
<?xml version="1.0"?>
<?xml-stylesheet type='text/xsl' href='MessageLog.xsl'?> <Log FirstSessionID="1" LastSessionID="12">
<Message Date="2009/7/4" Time="上午 12:50:55" DateTime="2009-07-03T16:50:55.390Z" SessionID="31">
<From><User FriendlyName="Want SAP, Netweaver, J2EE consultant"/></From>
<To><User FriendlyName="Derrick-"/></To><Text Style="font-family:; color:#000000; ">
ok...I got it... </Text> </Message> </Log>
B. Data extraction methodology
In order to filter-out unnecessary data from the collected email and MSN history file, we developed a data extraction methodology to extract useful data for constructing social networks. In section 4.B, we have introduced a sample email file format. However, different email agents and servers may have different email format. Thus, we have selected some important fields in the email file which are useful for us to calculate the social relationship between the communicators of the collected emails and MSN history.
From the email file, some necessary fields will be extracted, including “deliver-to”, “receive-id”, “date”, “to”, “from”, “subject”, “msg-id”, “priority”, “reply-to”, “mailer (agent)”, “encode”. “content-type”, “content”, “cc”. These fields will be extracted from the original. Some of the extracted fields are used to identify emails and some are important for relationship measurement.
In addition to the extraction of email fields, we also extract useful fields from the MSN history file. The fields will be extracted from the file, including from”, to”, content”, datetime”, id”, sessionid”, “msn-totage”. Among all of the extracted fields, the “msn-sessionid” field is used to record the session number and “msn-totage” field is used to identify a communication with multi-users. C. From social data to social networking
After collecting and extracting useful information from the muti-source social data. The relationship can then be calculated. When user inputting keywords and parameters, the system will match the keyword and the ontology based to find related email records, MSN messages and blog content.
Figure 5. Communication frequency table
The relationship (communication frequency) is the most important element to form a social network. Thus, a series of formulations for measure the relationship for different kinds of social communication will be discussed in this section. The data collection system that introduced in section 4.A also has the ability to measure the frequency for the input keyword as presented in figure 5.
The relationship of two nodes can be measured by formulation (1). Ri means the relationship from a specific node (a keyword with parameters) to a node i in a social network. Ei mean the email relationship for node i, Mi means the MSN relationship for node i, and Bi means the Blog relationship for
node i. W1, W2, W3 are three difference weight value to measure the importance of each relationship.
Ri W 1 Ei W2 Mi W3 Bi (1) The details of relationship E, M and B can be measured by formulation 2, 3, 4, and 5. In formulation 2, Esend means the frequency about how many how many mail send to node i, Ereceive means how many mail receive from i, Eforward is how many main forward from i, Ecc denotes how many mail received as cc from i, and Eco-receiver means how many mail received as co-receiver from i. Furthermore, W1, W2, W3, W4, W5 are the weight values for each email relationship.
Ei W1 Esend W2 Ereceive W3 Eforward W4 Ecc W5 Ecoreceiver (2) Formulation 4 is the complete formulation for MSN relationship measurement, Msend means a message send to node i, Mreceive is a message received from i, Mmulti denotes a message send for multi-communicators at the same time and Minteraction is an advanced interaction between node i such as video conference, file sharing, etc. However, only the measurements Msend , Mreceive , Mmulti are used in the paper as the formulation 3. In formulation 3 and 4, W1, W2, W3, W4 are weight values for each MSN relationship.
Mi W1 Msend W2 Mreceive W3 Mmulti (3) Mi W1 Msend W2 Mreceive W3 Mmulti W4 Minteraction (4) For Blog relationship, the measurement is shown in formulation 5. Bbrowsing means browsing frequency to a Blog of node i, Bbookmarking is the frequency of adding a Blog of node i to bookmark and Binteractoin is the frequency of interaction in the Blog of node i, such as response to a Blog entry.
Bi W1 Bbrowsing W2 Bbookmarking W3 Binteraction (5)
The formulations above are helpful for most cases to measure the relationships of the nodes to a specific node in a social network. Then the system can use the analyze results to generate and visualize the social network.
V. CONCLUSION AND FUTURE RESEARCH
Social and communication data are very common data in our daily life; however these data have not been used well for use to make decision. In this paper, we firstly provide an overview about the characteristics of these data and to illustrate how to use the concept of social networking and web mining to analyze the data. A system architecture is then provide to give the reader a picture about how to use the multi-source social data to generate dynamic and task-oriented social networks and by this to assist the decision making. More detail process of data collection, data extraction and relationship measurement in the system are also provided.
In the future, we will try to move our research focus to the second part (online processing) of the system and to implement the decision support system. Furthermore, we will try to study how to use the techniques of web mining to get better analysis
results and to enhance the accuracy of the decision support system. In addition, we will also try to understand more social data sources which are useful for including in the system and our future research.
ACKNOWLEDGMENT
This work is partially supported by a NSC research grant, TAIWAN (NSC 97-2410-H-390-022).
REFERENCES
[1] Adamic, L. A., and Adar, E. (2007) “Friends and Neighbors on the Web” Social Networks, Vol. 25, 2007, pp. 211-230
[2] Agrawal, R., Rajagopalan, S., Srikant, R., and Xu, Y. (2003) “Mining Newsgroup Using Networks Arising From Social Behavior” In Proceedings of World Wide Web 2003 Conference, Budapest, Hungary, pp. 529-535
[3] Bird, C., Gourley, A., Devanbu, P., Gertz, M. and Swaminathan, A. (2006) “Mining Email Social Networks” In Proceedings of MSR 2006, May 22-23, 2006, Shanghai, China.
[4] Borgatti, S. P., and Everett, M.G. (2002) “Ucinet for Windows: Software for Social Network Analysis”, Harvard: Analytic Technologies. [5] Burt, R.S., (1992). “Structural Holes, ” Harvard University Press,
Cambridge,MA.
[6] Cooley, R. Mobasher, B. and Srivastave, J. (1997) “Web Mining: Information and Pattern Discovery on the World Wide Web” In Proceedings of the 9th IEEE International Conference on Tool with Artificial Intelligence, 1997, pp. 558-567, Newport Beach, CA, USA [7] Cross, R. and Parker, A (2004). “The Hidden Power of Social
Networks, ” Harvard University Press
[8] Chin, A. and Chignell, M. (2006) “Finding Evidence of Community from Blogging Co-Citations: A Social Network Analytic Approach” In Proceedings of the IADIS International Conference on Web Based Communities 2006, San Sebastian, Spain, February 26-28, 2006 [9] Churchill, E. F., and Halverson, C. A. (2005) “Social Networks and
Social Networking” IEEE Internet Computing, September/October 2005, pp.14-19
[10] Dingt C. H. Q., Zha, H., Husbands, P., and Simont, H. D. (2004) “Link Analysis: Hubs and Authorities on the World Wide Web ” SIAM Review, Vol. 46, No. 2, pp. 256-268.
[11] Freeman, L., “Centrality in Social Networks: Conceptual Clarification,”
Social Networks, 1979.
[12] Fu, F., Chen, X., Liu, L., and Wang, L. (2007) “Social Dilemmas in An Online Social Network: The Structure and Evolution of Cooperation” Physics Letters A, Vol 371, 2007, pp. 58-64
[13] Fu, F., Liu, L., Wang, L. (2008) “Empirical Analysis of Online Social Networks in the age of Web 2.0” Physics Letters A , Vol. 387, 2008, pp. 675-684
[14] Furukawa, T., Matsuo, Y., Ohmukai, I., Uchiyama, K., Ishizuka, M. (2007) “Social Networks and Reading Behavior in the Blogosphere” In Proceedings of ICWSM 2007, Boulder, Colorado, USA, pp. 51-58 [15] Garton, L., Haythornthwaite, C., and Wellman, B.( 1997) “Studying
Online Social Networks,” Journal of Computer Mediated Communication (3:1).
[16] Godbole, N., Srinivasaiah, M., Skiena, S.: Large-Scale Sentiment Analysis for News and Blogs. In: Proceedings of ICWSM 2007, Boulder, Colorado, USA (2007)
[17] Jin, Y. Z., Matsuo, Y., and Ishizuka, M. (2007) “Extracting Social Networks among Various Entities on the Web” In Proceedings of the Fourth European Semantic Web Conference, 2007
[18] Kumar, R., Novak, J., and Tomkins, A. (2006) “Structure and Evolution of Online Social Networks” In Proceedings of KDD 2006 Conference, August 20-23 2006, Philadelphia, Pennsylvania, USA, pp. 611-617 [19] Lento, T., Welser, H. T., Gu, L., and Smith M. (2006) “The Ties that
Blog: Examining the Relationship Between Social Ties and Continued Participation in the Wallop Weblogging System” In Proceedings of the 15th International World Wide Web Conference, May 23-26 2006, Edinburgh, Scotland
[20] Mika, P.: Flink: Semantic Web Technology for the Extraction and Analysis of Social Networks. Web Semantics 3(2-3), 211–223 (2005) [21] Matsuo, Y., Mori, J., Hamasaki, M. (2006) “POLYPHONET: An
Advanced Social Network Extraction System from the Web” In Proceedings of 2006 Internet World Wide Web Conference, May 23-26, Edinburgh, Scotland.
[22] Mislove, A., Marcon, M., Gummadi, K. P., Druschel, P. and Bhattacharjee, B. (2007) “Measurement and Analysis of Online Social Networks” In Proceedings of 2007 Internet Measurement Conference, October 24-26, 2007, San Diego, California, USA, pp. 29-42
[23] Mitchell, J. C. (1969) “Social Networks and Urban Situations” Manchester University Press, England
[24] Mutton, P. (2004) “Inferring and Visualizing Social Netwokrs on Internet Relay Chat” In Proceedings of the Eighth International Conference on Information Visualization, pp. 25-43
[25] Pierrakos, D., Paliouras, G., Papatheodorou, C., Spyropoulos, C.D.: Web Usage Mining As A Tool for Personalization: A Survey. User Modelling and User Adapted Interaction 13(4), 311–372 (2003)
[26] Sarkar, P., and Moore, A.W. “Dynamic Social Network Analysis Using Latent Space Models” SIGKDD Explorations, Vol. 7, Issue 2, pp. 31-40. [27] Scott, J. (2000) “Social Network Analysis: A Hand Book (2nd ed.)”,
SAGE publication, 2000
[28] Scott, J. (2002) “Social Network Analysis: Critical Concepts in Sociology” Routledge, New York, USA
[29] Tang, J., Zhang, D. and Yao, L. (2007) “Social Networking Extraction of Academic Researchers” In Proceedings of the Seventh IEEE International Conference on Data Mining, pp.292-301.
[30] Ting, I. H. (2008) “Web Mining Techniques for On-line Social Networks Analysis” In Proceedings of the 5th International Conference on Service Systems and Service Management, Melbourne, Australia, 30 June-2 July 2008, pp. 696-700
[31] Turoff, M., Hiltz. S. R., Cho, H. K., Li, Z., and Wang, Y. (2002) “Social Decision Support Systems (SDSS)” In Proceedings of the 35th Hawaii International Conference on System Sciences, pp. 1-10.
[32] Wasserman, S., and Faust, K. (2003) “Social Network Analysis: Method and Applications” Cambridge University Press, Great Britain 2003 [33] Wellman, B. and Berkowitz, S. D. (ed.), (1988) “Social structures: A
network approach”, Cambridge University Press, pp. 19-61
[34] Wasserman, B., and Faust, K. “Social Network Analysis : Methods and Applications.”New York: Cambridge University Press, 1994.
