The proposed client and server systems are extremely complicated because they involve several protocols, require multiple concurrent threads, and have many modules linking together to accomplish the goals mentioned in Section 1.3. Thus, to make the aforementioned concepts of designs more understandable, this section is written to provide a summary about what has been said in this chapter, and more importantly, to describe how the pieces can be gathered to accomplish the proposed goals.
Server’s Expected capabilities
Starting with reviewing what the systems can do, the server is expected to have a GUI that can interact with the program user. The GUI is responsible for collecting the user events such as button-pressing or text-typing, and then transferring these actions into the system by calling corresponding event handling functions. The server has to know how to parse MPEG-4 files in order to transport the data in a timely fashion. Moreover, the server must be able to acquire audiovisual contents from various types of cameras such as wireless or USB cameras. If these acquired contents are not in MPEG-4 format, it has to be able to transcode them into MPEG-4. In the proposed system, the transcoding is carried out with the uses of MJPEG decoder and MPEG-4 encoder. Before starting a streaming session, the server has to be able to listen and accept incoming connections.
Upon establishing the connection with the client, they can start to exchange RTSP messages. Then the server can find out what media files to send and how to send them.
Regarding to RTSP, the server has to manage a state machine for each of its connected clients. Upon receiving an RTSP request from one of the clients, the server has to find out the corresponding state machine, perform the state change, react to it if necessary, and reply to the client with a response indicating the result of this request. If the streaming process is started, the server has to packetize the desired media data into fixed-size RTP packets, and transmit them via UDP or TCP with the rate controlled by the traffic smoothing function that computes the transmission plan according to packet-based MVBA.
Server’s Architecture
As can be observed, the server has to handle so many tasks at the same time. Thus, it is not hard to think of that several modules are needed to accomplish these tasks.
Essentially, the server uses five modules: 3GP Parser, 3GP Creator, Sub Server, Server Dialog, and SP_encore.lib modules. The first two and the last modules mainly deal with MPEG-4 related tasks, such as parsing, creating virtual files for live-captured media files being accessible by the clients, and encoding raw media data into MPEG-4 format. Sub Server module is the most important one. It provides RTP and RTSP services to the system. Server Dialog module is the GUI, which interacts with the user.
Client’s Expected capabilities
Having talked about the server’s capabilities and architecture, it is time to turn the focus to the client. The client, compared with the server, has fewer capabilities. It has to have a GUI just like that of the server in order to interact with the user and react to the user inputs by calling corresponding event handlers. In addition, it must know how to connect to the server to establish the streaming session. After the streaming session is established, it has to prepare and mange an RTSP state machine, just like the server, and be ready to exchange RTSP messages with the server. After exchanging connection information about RTP transmission with the server via RTSP means, the client must get prepared to receive RTP packets either from UDP or TCP channel. Obviously, the client has to know how to de-packetize the received RTP packets, and parse and decode the reconstructed MPEG-4 frames in order to render the desired audiovisual contents.
Client’s Architecture
Again, several modules are required to perform these jobs. There are three modules present: Sub Client, Client Dialog, and SP_decore.lib modules. The first module takes care of the RTP and RTSP related tasks. In addition, Client Dialog module is the GUI.
Finally, the last module, as implied by its name, is for MPEG-4 decoding.
Operations of Threads on Server Side
After reviewing the capabilities and architectures of the client and server systems, it is time to describe how the threads employed by both systems operate to achieve the final goals. Referring to Figure 27 again, on the server side, there is a GUI thread that awaits the user inputs. When the user presses the “Start” button, several other threads will begin. First, “Transcoder()” will start to acquire audiovisual contents from the connected cameras, transcode or encode them into MPEG-4 format, and then place the resulted bitstreams onto the buffer where the data packetizer can access when needed.
As a side note, these live-captured media data can be rendered to the GUI at the same time if desired. In addition to “Transcoder()” thread, “OverallListenerThread()” in effect will start listening to new incoming connections at port 554 via TCP connection. The listening is done by the listening socket, and whenever a connection is accepted and setup, a
connected socket will be created to handle this connection. The listening socket will still exist to listen to new incoming connections. The created connected socket will be handed over to another thread called “RTSPProcessorThread()”, which is responsible for exchanging RTSP messages with the client and reacting to them when they are valid for the current RTSP state. RTP connection information is obtained by the server when it receives SETUP request. After this information arrives and RTP channels are setup,
“StreamerletIs()” thread will start functioning. Before doing anything, it will check the type of the target media file, if the file is live-captured, it will get the data from the buffer where
“Transcoder()” thread places MPEG-4 bitstreams. On the other hand, if the file is stored media, it will get the data from the storage device. Whether the file is live-captured or obtained from the storage, it will be packetized into RTP packets by “StreamerletIs()” if the transmission is via UDP. The rate that “StreamerletIs()” packetizes is controlled by the token bucket realized by “BandwidthSmoother()” thread. “StreamerletIs()” will place these packets onto the buffer where “PacketScheduler()” can access. “PacketScheduler()” is a thread that inserts packets on the buffer into UDP, which in turn injects them into the network. On the other hand, if TCP is selected as the means of transportation,
“StreamerletIs()” will not packetize media frames into RTP packet format to make the uses of timestamping and sequence-numbering provided by RTP, since TCP guarantees correct transmission (ordered arrivals and re-transmission if loss occurs).
“StreamerletIs()” will hand over these frames to “TunnelMain()” thread, which inserts frames on the buffer into TCP. TCP in turn will inject them into the network.
When and by Whom Threads are Created on Server Side
Having talked about how these threads work together, it is time to see when and by whom they are created. The GUI thread is the primary thread of the process, thus it is started when the program begins. When the user presses the “Start” button, the GUI thread create both “Transcoder()” and ”StatusTimeFunc()” to transcode the acquired frames and display clients’ statuses, respectively. In addition, recall that an object of
“RTSPStreamServer” class is created when “StartServer()” in the streaming service interface block is called by the GUI thread to start the stream server, the construction function of this class will create “OverallListenerThread()” to listen to and accept new
incoming connections from the clients and “PacketScheduler()” to hand over RTP packets to UDP. Whenever “OverallListenerThread()” accepts and establishes a TCP connection with a new client, it will create a brand-new “RTSPProcessorThread()” and pass the control of this connected connection to “RTSPProcessorThread()”, which is in charge of exchanging RTSP messages with this client and performing RTSP state changes when necessary. When “RTSPProcessorThread()” receives a SETUP request while in Init state, it will create “StreamerletIs()” to packetize media frames , and
“TunnelMain()” if the client demands that the RTP packets are to be transmitted over TCP instead of UDP.
Operations of Threads on Client Side
Like the server, the client also has several threads running together to accomplish the final goals. Referring to Figure 29 again, on the top-right, the GUI thread waits for the user input. When the user presses the “Open” button, the GUI thread will try to connect to server’s port 554 (handled by “OverallListenerThread()” on the server side) to establish a TCP connection with the server for RTSP message exchanges. After the connection is setup and ready for data transmission, “RTSPProcessorThread()” will take over the control of this connected socket and start to handle RTSP message exchanges with the server and perform necessary RTSP state changes. At the same time,
“VideoStatusTimeFunc()” and “AudioStatusTimeFunc()” will continuously collect and display the receiving statuses for video and audio on the GUI, respectively, and also decode and render the received media frames on the GUI. After PLAY request is granted by the server and RTSP state is transited to PLAY state, “TunnelMain()” thread will be present to receive RTP packets if transmitted over TCP, or “RecvThreadIsK()” thread will be there to receive RTP packets if transmitted over UDP. If RTP packets indeed are sent via UDP channel, “DePktThreadIsK()” will be responsible for reconstructing the original un-packetized media frames from the received RTP packets. Depending on the data transmission channel selection, either “TunnelMain()” or “DePktThreadIsK()” will place the received frame data on the buffer where “VideoStatusTimeFunc()” and
“AudioStatusTimeFunc()” can access.
When and by Whom Threads are Created on Client Side
After the explanations of how these threads work together are given, it is time to see when and by whom they are created. The GUI thread is the primary thread of the process, thus it is started when the program begins. When the user presses the “Start” button, the GUI thread will connect to port 554 of the server to establish the TCP connection for the RTSP control messages. If this connection is setup successfully,
“RTSPProcessorThread()” will be created by the GUI to take over this connected connection and start to handle all RTSP related tasks. In addition, at this time,
“VideoStatusTimeFunc()” and “AudioStatusTimeFunc()” will also be created by the GUI. If the user desires that the media data are to be delivered via TCP, “TunnelMain()” thread will be created by the GUI to receive them from TCP when SETUP request is issued and sent to the server. Moreover, “RecvThreadIsK()” and “DePktThreadIsK()” will also be created by the GUI to be able to receive RTP packets via UDP and de-packetize those received RTP packets.
Summary
Section 4.7 can be summarized as follows. Referring to Figure 34, when both the client and the server systems first start up, only GUI threads will be up and running since they are the primary threads of the processes. These GUI threads will sit there waiting for the user inputs, such as button pressing or text editing. Referring to Figure 35, when the
“Start” button on the server side is clicked to start the streaming service, four more threads will be created. Their functions are described earlier. In particular,
“OverallListenerThread()”, which is designed to establish RTSP connections with multiple clients, will have a TCP listening port listening and accepting any incoming connection request arriving at port 554. The GUI thread of the client upon being requested by its user, will try to establish a TCP connection with “OverallListenerThread()”. Referring to Figure 36, after this TCP connection is established, the control of the connected socket on the server side will be passed from “OverallListenerThread()” to “RTSPProcessThread()”.
Similarly, on the client side, the control of the connected socket will be passed from GUI thread to “RTSPProcessThread()”. Therefore, “RTSPProcessThread()” threads on both sides can start their RTSP message exchanges via TCP channel.
GUI Thread GUI Thread
Server Client
IP Network
Figure 34 – Threads present when both server and client programs start
Figure 35 – Threads present when server starts its service and client connects to it
Figure 36 – Threads present when RTSP message exchanges are in progress
When SETUP messages are being exchanged by the client and the server, the RTP channel used to transport the RTP packets should be setup. If UDP is selected as the RTP channel, referring to Figure 37, on the client side, “RecvThreadIsK()” and
“DePktThreadIsK()” will be created to receive RTP packets and de-packetize received RTP packets, respectively. On the server side, if live-captured data are to be transmitted, then “StreamerletIs()” will acquire media frames from “Transcoder()” thread, packetize them into RTP packets, and hand over these packets to “PacketScheduler()”, which in turn will transmit these packets to “RecvThreadIsK()” on the client side. If stored media are to be streamed, then “StreamerletIs()” will acquire media frames from the storage device, packetize them into RTP packets with the rate regulated by
“BandwidthSmoother(), and hand over these packets to “PacketScheduler()”, which in
turn will transmit these packets to “RecvThreadIsK()” on the client side. On the other hand, if TCP is preferred, referring to Figure 38, then on both sides, “TunnelMain()”
threads will be created. The TCP connection between them is setup in the way similar to the RTSP channel. On the server side, if live-captured data are to be transmitted, then
“StreamerletIs()” will acquire media frames from “Transcoder()” thread. On the other hand, if stored media are to be streamed, then “StreamerletIs()” will acquire media frames from the storage device. In either case, “StreamerletIs()” will hand over these frames to
“TunnelMain()”, which in turn will transmit these packets to “TunnelMain()” on the client side.
Figure 37 – Threads present when both RTSP and RTP via UDP are in progress
Figure 38 – Threads present when both RTSP and RTP via TCP are in progress
5 Usages of Modules
This chapter gives descriptions about how to use the implemented modules to develop the server and the client systems. In particular, the usages of 3GP Parser and Creator modules, Sub Server module, and Sub Client module will be described in detail.