Component-Based Video Update

This section describes how component-based video update is handled in the modified graphics architecture. Figure 4.15 illustrates an example of component-based video update and Figure 4.16 shows the pseudo code of the process. Like the previous two examples in Section 4.2.3 and Section 4.2.4, there are three components on the screen; a background video in the back, a component-based video in the middle, and an oval AWT component in the front.

An AWT update event is posted when the next frame of the component-based video is decoded. The update region is the bounds of the component-based video. The update process is similar to that described in section 4.2.3, except that the AWT update event is originated from component-based video update instead of normal AWT component update.

(a) Clips and scales decoded component-based video

(b) Clears the update region of the off-screen buffer

(c) Draws the component-based video to the off-screen buffer

(d) Draws the AWT component to the off-screen buffer

(e) Clears the update region of X Pixmap

(f) Draws the background video to X Pixmap

(g) Alpha-blends the off-screen buffer over X Pixmap

(h) Blits X Pixmap to X Window

Figure 4.16 An example of component-based video update

update() {

The update region is the bounds of the component-based video;

Calls clearRect ();

paint(); }

paint() {

For each child component { for example, draws a oval;

}

clearRect() {

Clears the update region //

of the off-screen buffer //

by filling it with completely //

transparent color; // Figure 4.16(d) }

fillOval() {

Draws a filled oval to the //

off-screen buffer clipped //

by the update region; // Figure 4.16(d) }

start() {

While next video frame available {

Decodes next frame;

Clips and scales the decoded //

video; // Figure 4.16(a)

If this is a component-based Player {

Draws component-based video //

to the off-screen buffer; // Figure 4.16(c) }

updateGraphics() {

Clears the update region //

of X Pixmap by filling it //

with background color; // Figure 4.16(e) Draws background video to //

X Pixmap clipped by the update //

region; // Figure 4.16(f)

Alpha-blends the off-screen //

buffer over X Pixmap clipped //

by the update region, and put //

the result to X Pixmap; // Figure 4.16(g) Blits the update region of //

X Pixmap to X Window; // Figure 4.16(h) }

Video and graphics integrator

Figure 4.17 Pseudo code of component-based video update

4.2.6 Performance Profiling

Because there are no dedicated profiling tools on the ML310 platform to evaluate the execution time performance, several scenarios are chosen and the execute time of a specific function is measured instead. These scenarios simulate typical DTV video and graphics presentations. A complicated scenario would be that a user navigates available services while watching a program. On the video plane, the video content of the service that users are currently watching is presented. At the same time, users select the EPG to navigate other services available. The GUI components of the EPG are presented on the graphics plane, including several menu items and a component-based video to preview the video content of the service that users want to navigate. This scenario is illustrated as Figure 4.18.

Figure 4.18 Scenario of typical EPG usage

To simulate this scenario, the EPG GUI components are simplified to a component-based video and a full-screen-sized translucent AWT component as illustrated in Figure 4.19. The decoded video has a dimension of QCIF. The background video presented on the video plane is stretched to have a dimension of 640480. The component-based video presented in the graphics plane has a dimension of the original decoded video, and therefore no video scaling operation is needed. Both the component-based video and the background video have 2418 frames. The following profiling results are based on the combinations of the three elements:

the component-based video, the full-screen translucent AWT component, and the full-screen background video.

Figure 4.19 Scenario of profilings

The complete video and graphics pipeline consists of four stages: video decoding, video clipping and scaling, video and graphics composition, and screen blitting. The profiling results listed below are to investigate the execution time of video scaling and composition of video and graphics. The screen blitting stage is first excluded from the pipeline because it is handled by the TinyX server. Finally, a number of profilings are performed to investigate the time taken up by the screen blitting stage.

To exclude the contribution of the time spent on application initialization and cleanup, a set of profilings are performed based on background video with different frame numbers. The results are listed in Table 4.2. The initialization and cleanup time can be neglected when the number of frames is large enough.

Table 4.2 Profiling results of the execution time of different frame numbers.

Number of frames Execution time (sec)

5 5

221 58

859 210

2418 589

The first set of profilings is to evaluate the time spent on full-screen background video scaling. The results are summarized in Table 4.3. The video scaling takes up a major portion of the total time from 51% to 70%.

Table 4.3 Profiling results of the video scaling time.

Condition Scaling

Translucent AWT component 734 316 56.9

Background video

Component-based video 793 282 64.4

Background video 589 176 70.1

The second set of profilings is to determine the time spent on the video and graphics composition. The results are summarized in Table 4.4. The composition does not take up as much time as scaling does. It can also be found that whether composing or not does not make significant difference when there is only background video. The reason is that the graphics plane is completely transparent because there are no AWT components on it. The only work to do is to check that each pixel on the graphics plane is transparent, and no composition computations are performed.

Table 4.4 Profilings of the compostion time

Translucent AWT component 734 537 26.8

Background video

Component-based video 793 715 9.8

Background video 589 543 7.8

The third set of profilings is to evaluate the time spent on X Pixmap to X Window blitting. The results are summarized in Table 4.5. The blitting takes up more time than scaling or composition does. In fact, this portion of time has little relationship with the architecture or algorithm of the proposed MHP video and graphics subsystem. The inefficient framebuffer manipulation of the TinyX server contributes most of the performance loss in this case.

Table 4.5 Profilings of the screen blitting time

Condition Blitting

Translucent AWT component 1239 734 505

Background video

Component-based video 1448 793 45.2

Background video 1096 589 46.3

Chapter 5

Conclusion and Future Work

In this thesis, a simplified video and graphics subsystem of MHP is proposed and implemented. This proposed system conforms to the MHP graphics model as closely as possible. To achieve this goal, several important steps were gone through:

 A comprehensive understanding of the MHP specifications is the first necessary step, especially on the sections about the graphics model and the composition pipeline.

 An in-depth knowledge of the target environment -- the Xilinx ML310 platform, including the Linux operating system and the underlying X11 graphics system.

 Choose the appropriate Java platform and its reference implementation -- Java ME, CDC, and PBP. The Sun PBP 1.0.2 RI is a good starting point.

 The JMF Lite reference implementation is used as the media framework of the video and graphics subsystem.

 FFmpeg libraries are studied and the MPEG-2 demultiplexer and MPEG-2 video decoder are extracted.

 Extended functionalities of the JMF Lite are implemented to support for clipping and scaling of an MPEG-2 video.

 Integration of the modified JMF implementation and Microwindows-based Java AWT. A video can be presented on the video plane or on the graphics plane. The final screen is the composition of the background plane, video plane and graphics plane.

However, a few points are worth to continue to work on:

 Implement all the HAVi APIs. The MHP specification suggests that applications should use HAVi APIs rather than AWT APIs.

 Replace the X Window System by a hardware framebuffer and its associated graphics rendering operations.

We can partition the video and graphics MHP subsystem into two parts. The first part does not consume much processing power, such as the JMF Player state management. Thus it can be implemented by pure software. The second part requires extensive processing power, such as video scaling, video and graphics composition, and so on. This part is a good target for software-hardware-coworking.

Bibliography

[1] ETSI TS 101 812 V1.3.1, “Digital Video Broadcasting (DVB); Multimedia Home Platform (MHP) Specification 1.0.3”, June 2003.

[2] Steve Morris,http://www.interactivetvweb.org/tutorial/javatv/xletintro.shtml, “An Introduction to Xlets”, 2004

[3] Steve Morris,http://www.interactivetvweb.org/tutorial/mhp/mhp_background.shtml,

“Some Background to MHP”, 2004

[4] Xilinx, Inc., “ML310 User Guide. Virtex-II Embedded Development Platform”, January 2005

[5] Pierre-Jean Turpeau,http://nobug.ifrance.com/nobug2/article1/babyloon/tut_xwin.htm,

“Introduction to X Window Programming”, April 1998

[6] Christophe Tronche,http://tronche.com/gui/x/xlib/, “The Xlib Manual”, October 2005

[7] X-Oz Technologies,http://www.x-oz.com/tinyx.html, “Integrated TinyX”, 2003

[8] Sun Microsystems, Inc., “CDC: Java Platform Technology for Connected Devices. Java Platform, Micro Edition, White Paper”, June, 2005

[9] Sun Microsystems, Inc., “JSR-000129 Personal Basis Profile 1.0b (Maintenance Release)”, December 2005

[10] Greg Haerr,

http://www.ocera.org/archive/upvlc/public/components/sa-rtl/sa-rtl-2.2-pre2/Documentat ion/MicroWindows/microwindows_architecture.html,“The Nano-X Window System Architecture”, March 2000

[11] Sun Microsystems, Inc., “Java TV API Reference Implementation Porting Guide Version 1.0”, November 2000

[12] http://ffmpeg.mplayerhq.hu/index.html, “FFmpeg Multimedia System”

[13] Martin Böhme,http://www.inb.uni-luebeck.de/~boehme/using_libavcodec.html, “Using libavformat and libavcodec”, February 2004

自傳

陳旻弘，男，民國七十一年九月十七日生於台灣省彰化縣彰化 市。民國九十三年六月畢業於交通大學電機與控制工程學系，並 於同年直升交通大學電子研究所碩士班，從事數位電視中介層軟 體平台之相關研究，指導教授為杭學鳴博士。民國九十五年六月 取得碩士學位，畢業論文題目為『MHP 視訊及圖形子系統於 Xilinx ML310 平台上之設計與實現』。研究興趣為 Java 平台、嵌入式系 統上之軟體系統及數位電視上之軟體平台。