Objective quality evaluation

The aim of objective perceptual measurements is to predict the basic audio quality by using objective measurements incorporating psychoacoustic principles. For objective quality evaluation, we mainly adopt the PEAQ (Perceptual Evaluation of Audio Quality) system which is the recommendation scheme by ITU-R Tack Group 10/4. While PEAQ is based on a refinement of generally accepted psychoacoustic models, it also includes new cognitive components to account for higher-level processes that come to play a role in the judgment of audio quality. The objective difference grade (ODG) is the output variable from the objective measurement method via an artificial neural network. The ODG value should range from 0 to –4, where 0 corresponds to an imperceptible impairment and –4 to impairment judged as very annoying. The improvement up to 0.1 is usually perceptually audible. The PEAQ has been widely used to measure the compression technique due to the capability to detect perceptual difference sensible by human hearing systems. The following experiments are based on this PEAQ system [27].

We use NCTU-MP3 [20] as our code base for quality evaluation in MP3. Both CBR mode and ABR mode are conducted in following experiments. Other bit reservoir methods are also listed for comparison with our proposed design.

4.5.1.1 Objective quality evaluation for MP3 CBR mode

For CBR mode, there are another 4 bit reservoir methods for comparison. The first one is NoReservoir, which just uses the allocated mean bits without accumulating the bits left. The second one is Simple, which only preserves remaining bits from previous one frame to current frame without any managing scheme. The third one is ISO, which is the method proposed in MP3 standard [1]. The fourth one is LAME-3.88, which is the bit reservoir scheme used in [14]. We also consider coding environment with or without other modules, e.g. window switch and M/S coding. The results shown in Table 3 and Table 4 illustrate that the new bit reservoir design could gain 0.2384 (Long/Short window, M/S coding) and 0.3375 (Long window only, without M/S coding) improvement in average than those without any reservoir controlling. With comparing to other reservoir schemes, our new design is also superior, especially in si02, sm02 and sm03.

Table 3: Objective measurements through the ODGs for different bit reservoir designs in MP3 CBR mode (Long/Short window, M/S coding).

Coding Methods 1 2 3 4 5

Stereo Modes M/S M/S M/S M/S M/S Allow Short Yes Yes Yes Yes Yes

es01 -0.41 -0.34 -0.34 -0.33 -0.31 es02 -0.23 -0.19 -0.19 -0.22 -0.2 es03 -0.29 -0.28 -0.28 -0.24 -0.26 sc01 -0.62 -0.55 -0.55 -0.59 -0.56 sc02 -1.08 -0.99 -0.99 -1.07 -0.99 sc03 -0.93 -0.79 -0.79 -0.83 -0.79 si01 -1.05 -0.9 -0.9 -0.92 -0.86 si02 -2.03 -1.75 -1.75 -1.33 -1.09 si03 -1.72 -1.48 -1.48 -1.63 -1.48 sm01 -2.06 -1.73 -1.73 -1.91 -1.74 sm02 -0.75 -0.68 -0.68 -0.46 -0.49 sm03 -1.6 -1.37 -1.37 -1.27 -1.14 Average -1.0642 -0.9208 -0.9208 -0.9 -0.8258 Bit Rate : 128kbps

Sample Rate : 44100 Hz Coding Method :

1 : NoReservoir; 2 : Simple; 3 : ISO; 4 : LAME-3.88; 5 : Our New Design

Comparison of Objective Quality

-2.5 -2 -1.5 -1 -0.5 0

1 2 3 4 5

ODG

Max Min Average

Figure 35: The ODG range comparison of Table 3. The top arrow represents the maximum ODG value, the down cross represents the minimum ODG value, and the middle square represents average ODG value among the twelve test tracks.

Table 4: Objective measurements through the ODGs for different bit reservoir designs in MP3 CBR mode (Long window, without M/S coding).

Coding Methods 1 2 3 4 5

Stereo Modes L/R L/R L/R L/R L/R

Allow Short No No No No No

es01 -1.72 -1.46 -1.46 -1.35 -1.3 es02 -1.47 -1.33 -1.33 -1.29 -1.34 es03 -1.6 -1.35 -1.35 -1.28 -1.34 sc01 -0.84 -0.76 -0.76 -0.8 -0.76 sc02 -1.21 -1.06 -1.06 -1.17 -1.06 sc03 -1.3 -1.11 -1.11 -1.18 -1.1

si01 -1.19 -0.99 -0.99 -1.02 -0.99 si02 -2.97 -2.57 -2.31 -2.1 -1.7 si03 -1.79 -1.54 -1.54 -1.68 -1.54 sm01 -2.06 -1.75 -1.75 -1.91 -1.76 sm02 -1.09 -0.9 -0.9 -0.72 -0.71 sm03 -1.56 -1.33 -1.33 -1.26 -1.15 Average -1.5667 -1.3458 -1.3242 -1.3133 -1.2292 Bit Rate : 128kbps

Sample Rate : 44100 Hz Coding Method :

1 : NoReservoir; 2 : Simple; 3 : ISO; 4 : LAME-3.88; 5 : Our New Design

Comparison of Objective Quality

-3.5 -3 -2.5 -2 -1.5 -1 -0.5 0

1 2 3 4 5

ODG

Max Min Average

Figure 36: The ODG range comparison of Table 4. The top arrow represents the

4.5.1.2 Objective quality evaluation for MP3 ABR mode

For ABR mode, there are other two bit reservoir methods for comparison. The first one is NoReservoir, which just uses the allocated mean bits without accumulating the bits left. The second one is LAME-3.88, which is the bit reservoir scheme for ABR mode in [14]. We also consider coding environment with or without other modules, e.g. window switch and M/S coding. The results shown in Table 5 illustrate that the new bit reservoir design could gain 0.3309 (Long/Short window, M/S coding) and 0.4659 (Long window only, without M/S coding) improvement in average than those without any reservoir controlling. With comparing to reservoir schemes in LAME-3.88, our new design is also superior, especially in si01, si02 and sm01 with other assistant modules (window switch and M/S coding) or si01, si02, and natural vocal serious (es01, es02, and es03) without other assistant modules.

Table 5: Objective measurements through the ODGs for different bit reservoir designs in MP3 ABR mode.

Coding Methods 1 2 3 4 5 6

Stereo Modes M/S M/S M/S L/R L/R L/R Allow Short Yes Yes Yes No No No

es01 -0.41 -0.36 -0.29 -1.72 -1.62 -1.09 es02 -0.23 -0.24 -0.16 -1.47 -1.43 -1.06 es03 -0.29 -0.27 -0.26 -1.6 -1.6 -1.09 sc01 -0.62 -0.63 -0.47 -0.84 -0.88 -0.68 sc02 -1.08 -1.12 -0.95 -1.21 -1.27 -1.07 sc03 -0.93 -0.97 -0.68 -1.3 -1.36 -1.04 si01 -1.05 -1.14 -0.71 -1.19 -1.27 -0.86 si02 -2.03 -1.47 -0.99 -2.97 -2.36 -1.48 si03 -1.72 -1.7 -1.42 -1.79 -1.75 -1.52 sm01 -2.06 -2.09 -1.45 -2.06 -2.07 -1.76 sm02 -0.75 -0.59 -0.32 -1.09 -0.82 -0.49 sm03 -1.6 -1.41 -1.1 -1.56 -1.39 -1.07 Average -1.0642 -0.9992 -0.7333 -1.5667 -1.485 -1.1008 Bit Rate : 128kbps

Sample Rate : 44100 Hz Coding Method :

1, 4 : NoReservoir; 2, 5 : LAME-3.88; 3, 6 : Our New Design;

Comparison of Objective Quality

-3.5 -3 -2.5 -2 -1.5 -1 -0.5 0

1 2 3 4 5 6

ODG

Max Min Average

Figure 37: The ODG range comparison of Table 5. The top arrow represents the maximum ODG value, the down cross represents the minimum ODG value, and the middle square represents average ODG value among the twelve test tracks.

4.5.1.3 Objective quality evaluation for AAC

For AAC, there are another 2 bit reservoir methods for comparison. The first one is NoReservoir, which just uses the allocated mean bits without accumulating the bits left. The second one is Simple, which only preserves remaining bits from previous one frame to current frame without any managing scheme. We also consider coding environment with or without other modules, e.g. Temporal Noise Shaping, window switch and M/S coding. The results shown in Table 6 illustrate that the new bit reservoir design could gain 0.1875 (TNS, Long/Short window, and M/S coding) and 0.4975 (Long window only, without M/S coding and TNS) improvement in average than those without any reservoir controlling. With comparing to other reservoir schemes, our new design is superior in si01, sm01 and sm02 with other assistant modules (TNS, window switch and M/S coding) or si02, sm02, sm03 and natural vocal serious (es01, es02, and es03) without other assistant modules.

Table 6: Objective measurements through the ODGs for different bit reservoir designs in AAC.

Coding Methods 1 2 3 4 5 6

Stereo Modes M/S M/S M/S L/R L/R L/R Allow Short

& TNS Yes Yes Yes No No No

es03 -0.27 -0.27 -0.22 -2.24 -2 -1.43 sc01 -0.54 -0.5 -0.45 -0.7 -0.65 -0.59 sc02 -0.82 -0.67 -0.63 -0.98 -0.79 -0.74 sc03 -0.44 -0.4 -0.37 -0.6 -0.52 -0.45 si01 -0.79 -0.68 -0.55 -1.08 -0.91 -0.7 si02 -0.63 -0.56 -0.52 -3.28 -3 -1.97 si03 -1.08 -0.94 -0.92 -1.21 -1.06 -1.04 sm01 -0.84 -0.69 -0.51 -0.81 -0.64 -0.46 sm02 -1.2 -1.08 -0.5 -1.54 -1.4 -0.67 sm03 -0.7 -0.61 -0.54 -1.2 -1.03 -0.63 Average -0.6592 -0.5758 -0.4717 -1.4317 -1.2642 -0.9342 Bit Rate : 128kbps

Sample Rate : 44100 Hz Coding Method :

1, 4 : NoReservoir; 2, 5 : Simple; 3, 6 : Our New Design;

Comparison of Objective Quality

-3.5 -3 -2.5 -2 -1.5 -1 -0.5 0

1 2 3 4 5 6

ODG

Max Min Average

Figure 38: The ODG range comparison of Table 6. The top arrow represents the maximum ODG value, the down cross represents the minimum ODG value, and the middle square represents average ODG value among the twelve test tracks.