Panoramic Video Stabilization

Panoramic video stabilization intuitive method created the 3D camera path by structure from motion. Traditional structure from motion method has much restricts. Panoramic frames only can be as six traditional photos and are not benefit for this steps stable. Kamali [9] proposed a new structure from motion method used in frames stitching to create the 3D camera path to video stabilization. However, 3D camera path calculated by features of frames is not reflect the true camera motion because the parallax problem is much serious in panoramic videos camera path generating. Outdoor panoramic videos include the ground plane and the sky at the same time often. The difference between the distance from camera to the ground plane and the distance from camera to the sky is too large to fit a single camera path for panoramic frame.

Traditional 2D video stabilization methods is stable and fast in use. However it is not suitable for handling the panoramic frames sequence. The features motion in different part of the frame must be difference in panoramic frames contents relations in 3D space. [5,12]

propose a method to use Laplacian mesh warping to reduce the parallax effective of the video stabilization. It performed a good result under the promise of the features motion description accurately. It is easy in video frames are not distortion. While the panoramic frame is rectangle size and can considered as a texture for a sphere in space. The texture mapping lead to the panoramic frames vision contents distortion. The features motion in panoramic frames combine the reason of the shaking and mapping. In addition, the distortion of the panoramic also influence the feature extracting in this step.

Another method of panoramic video stabilization is each original video stabilized first.

Then, using all the stabilized original video to generate the panoramic video. Each Syn-chronous frame in different videos need to calibration. We can not accept the flicker taken by the error of calibrations between frames for the panoramic video. That’s why tradi-tional video calibration to generate panoramic video method is choose one or multiple frames for each video synchronized to create a template. All frames of the videos use this template to calibration. It is fast and stable.

Our goal is stabilize the panoramic video robust. We propose a subdivision camera paths for panoramic video stabilization. Split the influence of the image distortion by texture mapping is the core idea of this processing.

While the panoramic frame features mapping is effected by the distortion. We can obtain the features motion by video shaking in original videos not distortion. So, to obtain the right motion of shaking, we obtain the features motion between two frames by optical flow calculation for each original video.

We need to know the mapping of each feature between the original videos and the panoramic video. Fortunately, we apply the traditional panoramic video calibration which use a fixed mapping template. This means the features we extracted in original frames can applied to panoramic frames easily. Note that this mapping succeed in surround the center of original video because of the panoramic frame deforming. A panoramic frame with right motion features generated now.

We separate the influence of panoramic deforming and obtain the right feature motion by original video shaking now. Subdivision camera paths is a good solution for panoramic

video stabilization. We split the whole panoramic frames to multiple areas and generate smooth camera path for each area to stabilization. It fit the observation that different part of panoramic frame must have different pixel motions. For each subdivision camera path, we follow the basic framework to stabilization. We need to use the original features motion results to calculate the homography matrix. The 2D motion trajectory represented the subdivision camera motion generated by the sequence of transformations in feature rich area in panoramic frame. This result is not the final camera position pixel motion in panoramic video. The intensity of panoramic deforming in subdivision camera position effect the final motion represented.

The panoramic deformed frame mapping to a sphere called UV mapping. For any point P (d_x, d_y, d_z) on the sphere , UV normalized coordinate in the range of (0, 1) calculated as follows:

u = 0.5 + arctan 2(d_z, d_x) 2π v = 0.5− arcsin d_y

π

We find the intensity of pixels deforming in panoramic frames is same if they have same latitude. The intensity of pixels deforming in panoramic frames fit the sin func-tion of longitude. The relafunc-tionship of the intensity of each pixels latitude and longitude fit the above normalizations. Using this above relationship of U,V to adjust the calcu-lated intensity of the camera motion perform the right camera points position adjustment in panoramic frames. Defined the camera position point P original motion by shaking motion vector is (x, y) and the adjust result is (x^′, y^′), the adjustment as follows:

x^′ = x· α · sin θ (4.1)

y^′ = y· β · sin θ (4.2)

Note α, β is the ratio of the original frame size and the panoramic frame size in rows

and columns directions. The sinθ function is fit the influence of latitude.

We obtain the right constraint points motion for Laplacian mesh. Then, we use Lapla-cian mesh warping approach generating all subdivision camera trajectory. The challenge is maintain the omnidirectional view in mesh warping. Parallax reducing laplacian mesh warping can not maintain the rectangle shape of the frame. It caused the field of view reduce. We can accept this FOV changing in traditional video stabilization. Panoramic video stabilization must maintain the panoramic view of frame at any time. We propose a subdivision camera paths generating approach maintain the panoramic views by Laplacian mesh warping.

We reform the subdivision camera paths mesh according to the panoramic frame char-acteristic. The left and right mesh edge points have corresponding relationship. We con-sidered pair of points in left and right edge have same latitude as the same points and combine their neighbours influence in calculating. The top edge of the mesh points cor-respond to the same point in space sphere. We considered all the top edge mesh points as a same point and combine all their neighbours influence in mesh warping calculation.

The vertices on bottom edge of mesh handled the same approach of the top edge vertices.

Then, we obtain all the vertices new positions. In image compensation, we define the filled area is same as the original panoramic frame area. Tilling outside the area defined all can draw back to the filled area because the edge extend relations defined above. This guarantee using the original panoramic pixels filling the filled area is enough. We obtain a same size warped panoramic frame to make sure the panoramic views.