142
S
ERIES
E
DITORIAL
results. The third category accounts for the highest num-bers, where their acceptance depends mostly on whether they have “new” methodology or observations that are unknown to the community. The fourth, fifth, and seventh categories have only one submission each, the sixth catego-ry has none. We believe that these three categories are of interest to most of the academic and industry communities. Only one of the 17 submissions has authors working in the industry, which implies low involvement in research publi-cation from the industry. Soliciting submissions in the fourth, sixth, and seventh categories from academia and perhaps test laboratories might be more feasible than solic-iting from manufacturers.
The first article in this issue, “Measuring BitTorrent Ecosystem: Techniques, Tips and Tricks,” is a comprehen-sive survey on the measurement techniques for the most successful peer-to-peer application, BitTorrent. It classifies the reported techniques into macroscopic (portal crawling and tracker crawling), microscopic (peers crawling and clients/plugins), and complementary. The main issue here is how to “sniff” or “crawl” the global-scale BitTorrent ecosystem by crawling through portals (servers with meta-data files), trackers (servers maintaining a list of clients downloading a given content), peers, or one’s own client. How detailed one can crawl depends on from whom and how one crawls. Through the portals and trackers, the data is macroscopic (e.g., content popularity distribution and publishing rate per day through portals, demographics information with IP addresses through trackers). Through peers and one’s own client, it is possible to retrieve micro-scopic data such as peer type and download/upload rate. The article also raises unresolved issues along with their possible solutions, including peer identification, a banned crawler’s IP address, and upload rate estimation. It is a valuable reference for peer-to-peer researchers/developers as well as content providers who are eager to understand how their contents are distributed “freely.”
The second and third articles are both on combining emulation and simulation. The second article, “OEFMON: his is the second issue of the still “new” Network
Testing Series. Compared to two accepted out of nine submissions in the first call, this second call received eight submissions, and we accepted three of them, and one submission that received the decision of “major revisions” in the first call got accepted in the second call. Yes, this series allows a second round of reviews if the work has merits but needs further reworking. Among the three accepted articles, one is on peer-to-peer measurements on the prevailing BitTorrent ecosystem, while the other two are on combining emulation and simulation for massive service evaluation and multimedia over networks. The for-mer is Internet-scale measurement, while the latter two belong to testbed construction. All of them belong to “larger-scale” issues in terms of network testing, compared to the “smaller-scale” issues at the “device-level” and the “component-level.” It appears that the academic submis-sions often fall in the larger-scale issues. In this series we continue to solicit submissions that address the smaller-scale issues, which would interest the industry community in addition to the academic community.
The selection criteria include how relevant to network testing the work is, how new the method or result is, and, as it is for other IEEE magazines and journals, how rigid the writing is. All submissions received two to four reviews. Among the 17 submissions to the first two calls, we could divide them into seven categories:
• Research with simulation (four submissions) • Research with testbeds (two submissions)
• Testbed construction (seven submissions; one at the global scale, three at the local scale, two at the labora-tory scale, and one on tool development)
• Measurement (one submission) • Test framework (one submission) • Device-level testing (no submissions) • Component-level testing (one submission)
The first category does not fall in the scope of this series. The second category might or might not fall in it, depending on the percentage on test methodology and
IEEE Communications Magazine • September 2011
T
N
ETWORK
T
ESTING
S
ERIES
Ying-Dar Lin Erica Johnson Eduardo Joo
IEEE Communications Magazine • September 2011 143 An Open Evaluation Framework for Multimedia over
Net-works,” integrates a multimedia emulator with a network simulator in order to evaluate how various multimedia codecs “adapt” to network conditions. The authors provide OEFMON as an open source tool that could integrate DirectShow (a multimedia framework) and Qualnet (a net-work simulator) for the above-mentioned evaluation. The third article, “In Pursuit of Massive Service Emulation: A Methodology for Testbed Building,” also combines emula-tion and simulaemula-tion, although in this case study the core and access networks and the majority of users are simulat-ed, but the video servers and some users are real. Quality of experience (QoE) metrics such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and mean opin-ion score (MOS) are measured. It also presents a procedu-ral methodology with the phases of service specification, testbed requirements, test resource allocation, and experi-ment analysis to “resize” the services and resource alloca-tions when the QoE metrics are not satisfactory.
B
IOGRAPHIESYING-DARLINis a professor of computer science at National Chiao Tung University (NCTU), Taiwan. He received his Ph.D. in computer science from the University of California of Los Angeles in 1993. He spent his sabbatical year as a visiting scholar at Cisco Systems in San Jose in 2007–2008. Since 2002, he has been the founder and director of Net-work Benchmarking Lab (NBL, www.nbl.org.tw), which reviews netNet-work products with real traffic. He also cofounded L7 Networks Inc. in 2002, which was later acquired by D-Link Corp. His research interests include design, analysis, implementation, and benchmarking of network
proto-cols and algorithms, quality of services, network security, deep packet inspection, P2P and mesh networking, and embedded hardware/software co-design. His work on multihop cellular has been cited over 500 times. He is currently on the editorial boards of IEEE Transactions on
Comput-ers, IEEE Network, Network Testing Series of IEEE Communications Mag-azine, IEEE Communications Surveys and Tutorials, IEEE Communications Letters, Computer Communications, and Computer Networks. He
recent-ly published a textbook, Computer Networks: An Open Source Approach (www.mhhe.com/lin), with Ren-Hung Hwang and Fred Baker (McGraw-Hill, 2011).
ERICAJOHNSONis the director of the University of New Hampshire InterOper-ability Laboratory. In this role, she manages and oversees over 20 different data networking and storage technologies providing all aspects of adminis-tration, including coordination of high profile testing events, coordination with different consortia, and working with various industry fora. She is also a prominent member of organizations both internally and externally, and enjoys a powerful mix of technology and business related activities. At the University of New Hampshire she participates in the UNH Steering Commit-tee for Information Technology, the Senior Vice Provost for Research Work-ing Group, and Computer Science Advisory Board. In the industry, she was appointed the technical representative of North America for the IPv6 Ready Logo Committee and was also chosen to be an IPv6 Forum Fellow. Passion-ate about the laboratory and its possibilities, she continues to work with many industry fora, commercial service providers, network equipment ven-dors, and other universities in order to further the InterOperability Labora-tory’s mission.
EDUARDOJOOis a software project leader at Empirix, Inc., in Bedford, Mas-sachusetts. He received his Master of Science in computer system engineer-ing, computer communications and networks, from Boston University in 2006. He joined Empirix, Inc., in 2001 and has led the successful develop-ment of network testing and emulation systems, including PacketSphere Network Emulator, PacketSphere RealStreamer, Hammer NxT, and Hammer G5. He is currently leading the development of next-generation mobile broadband data network monitoring and testing tools. His areas of interest include voice and data protocols and wired, wireless, and mobile network communications.
