Noisy Video Super-Resoluti on
Feng Liu, JinjunWang,ShenghuoZhu (MM’08)
University of Wisconsin-Madison, NEC Laboratories America, Inc.
第一組: 資訊四 B95902105 黃彥達 資訊碩一 R98922046 蔡旻光 網媒碩二 R97944012 鄒志鴻
Outline
Introduction
Goal
File Format
Noise Reduced Image
Proposed Approach
Motion Estimation & Estimated Super-Resol ution Result
Implementation
Result
Conclusion
2
Introduction
Low-quality videos often not only have lim ited resolution but also suffer from noise
In fact, the requirements of de-noising & super- resolution is quite similar
This paper present a unified framework wh ich achieves simultaneous video de-noisin g and super-resolution algorithm by some m easurements of visual quality
3
Goal
Refine low-quality videos from YouTub e, and make the video better effects, which has better quality by human eye s.
Input is low-quality and noise-includ
ed (block effects or somewhat noise)
videos
File Format
.3gp file
Frame rate: 15(our video) or 25
Frame size: 176(w) * 144(h)
MPEG-4 Part 12
It is used on 3G mobile phones also can be played on
2G and 4G phones.
Our video: 867 KB/ 98 sec
5
Noise-Reduced Image
mv-SAD Gaussian- space
Gaussian- time
| p(I,j) – p(i’, j’) | >
threshold
Gaussian Space
Frame t
Pixel(I ,j)
Standard deviation Set Mean
= 0
Motion Vector
Frame t Pixel ( i , j , t)
Frame Pixel ( i + mv_i , j + mv_j t+1 , t+1)
(mv_i , mv_j)
Gaussian Time
Frame t - 2
Frame t - 1
Frame t
Space Gaussian
Time Gaussian
Pixel(I ,j)
Frame t+1
Frame t+2
Frame t
Shot Detecti
on
Noise-Reduced Image
Before After
Proposed Approach – 1 / 4
Consider the visual quality with respect t o the following 3 aspects:
Fidelity Preserving
▪ To achieve similar high-resolution result
Detail Preserving
▪ Enhanced details (edge)
Spatial-Temporal Smoothness
▪ Remove undesirable high-frequency contents (e.g. jitter)
11
Proposed Approach – 2 / 4
Fidelity Preserving
Conventional metrics:
▪ Measure fidelity by the difference between Ih & Il would be problematic & waste useful time-space information in video
Proposed metrics:
▪ Estimate an approximation of super-resolution results from space-time neighboring pixels
▪ The fidelity measurement:
see next page for details noised
12
Proposed Approach – 3 / 4
Detail Preserving
Enhanced details (edge)
Contrast preserving
▪ Human visual system is more sensitive to contrast than pixel values
▪ Gradient fields of Ih & should be close
,where Wk is one or zero if the patchk with/o edges (canny detector)
13
Proposed Approach – 4 / 4
(Spatial-Temporal) Smoothness
Smooth results are often favored by the human system
Encourage to minimize:
A 2-D Laplace filter may be
Spatial-temporal Laplacian
OR
14
An Optimization Problem
Proposed Measurements
A quadratic minimization problem to solve ( AX = b):
Contrast Similarity
Detail Information(edge)
Spatial-Temporal Smoothness
15
Implementation – 1 / 3
Inputlow h
I~
Ih~
Gh~
Ih~
) 1 , ( )
1 ,
(I~ t w I~ t
wmv h mv h
X =
I
6 -1 … -1 -1 6 -1 … -1
-1 6 -1 … -1Laplacian
Gradient
-1 0 1 … 1 -1 0 1 … 1 -1 0 1 … 1
Minimize Edge
Motion Estimation Motion Estimation
++
Result (X)
Fidelity
Gaussian filter
wmv
fidv
fidg
dt
sm
16
Implementation – 2 / 3
....
,
1
500
1
11
x b s too slow
A
nn n
n
17
Ih~
Gh~
Ih~
) 1 , ( )
1 ,
(I~ t w I~ t
wmv h mv h
X =
I
6 -1 … -1 -1 6 -1 … -1
-1 6 -1 … -1Laplacian
Gradient
-1 0 1 … 1 -1 0 1 … 1 -1 0 1 … 1
Minimize Edge
Fidelity
fidv
fidg
dt
sm
!
!
! 2
1 A ' x A b s
x A
A t n n n t
by sparse least square solver
Implementation – 3 / 3
Adjustments for the weight terms
The measurement term is more emphasized if the w eight is larger
By iteratively experiments for our test data, we took
However, we found that the best weight set may b e different for different videos
3 . 0 ,
1 . 0 ,
1 ,
1
fidg dt smfidv
18
Result
352 x 288 Result
19
Result
352 x 288 Result
20
Result
352 x 288 Result
21
Conclusion
The proposed framework formulates noisy video super-resolution as an optimization problem, aiming to maximize the visual quality
By exploiting the space-temporal information, we can estimate a better baseline than conven tional fidelity measurement
The properties include fidelity-preserving, d etail-preserving and smoothness are considere d to achieve the best visual quality results
22
Thank you!!
23