VR Project Orientation

23  Download (0)

Full text


VR Project Orientation

Ching-Chi Lin

Institute of Information Science, Academia Sinica

Department of Computer Science and Information Engineering, National Taiwan University




Research Issues

Current Project and Working Items

◦ Real-time Room fusion

◦ Perspective Changing


What is VR

Virtual Reality(VR) replicates an

environment that simulates a physical

presence in places in the real world or an imagined world, allowing the user to

interact in that world.[1]

◦ Artificially create sensory experiences, which can include sight, touch, hearing, and smell.

[1] https://en.wikipedia.org/wiki/Virtual_reality


Side-by-side Image


Augmented Reality(AR)


Mixed Reality

VR: virtual objects + virtual environment

AV: real objects + virtual environment

AR: virtual objects in real environment


VR Devices

Head Mount Device(HMD):

◦ Oculus Rift

◦ Samsung Gear VR

◦ Sony PlayStation VR

◦ HTC Vive

◦ Google Cardboard

◦ ...

Position Tracker, controller, joystick, …etc.


VR Applications


◦ FPS, adventure, music, … etc.


◦ 3D movie, 360 ° videos

◦ VR movies




Research Issues

Improve user experiences

◦ Make the VR/AR effect more realistic.

◦ Reduce the dizziness

◦ Reduce random wobble of the picture.


Research Issues(Cont.)

Mobile VR/AR

◦ Find a good balance between QoE and battery life of a mobile device.

◦ Dynamically scale resolution and refresh rate to reduce mobile GPU power consumption.

◦ Use user head movement information to decide the appropriate QoE metrics.


Research Issues(Cont.)

Cluster-Assisted VR

◦ Move the computations from local device to a server cluster, such that users without high-

end hardware equipments can also experience VR fluently.

◦ How to allocate sufficient computing

resources to different VR applications in the cluster?


Research Issues(Cont.)

Develop new VR/AR applications.


Current Project

Live Reality Fusion

◦ Combine the real-time images from two different locations into one.

◦ ex: fuse two seminar rooms for oversea joint meeting; “wall removing” for Interior design.

◦ Cooperate with Dr. Wang’s group.


Two Approaches




One-360’-Camera Approach

Only one 360’ camera in the remote room.

A server captures and process the stream from camera, then fuse the processed

stream into the video stream captured by

user HMD.


One-360’-Camera Approach(Cont.)

Remote room

Observation Room


Multiple-Camera Approach

Remote room

Observation Room



Break the video captured by 360’ camera into frames.

◦ Panoramic image

Construct a cuboid model of the remote room.

◦ Transform the panoramic frame into cube.

◦ Transform the cube into cuboid.



Compute what an user should see from the current angle.

◦ By projection or other methods.

Replace the wall in observation room

with the projected image of the remote



Short-Term Goal

Fuse R106 and R107 in IIS.

◦ The wall between the two rooms can be removed to achieve real “room fusion”.

◦ Take picture/video as ground-truth.


Current Progress

Dr. Wang’s RA, 祖詒, has completed the

construction of the cuboid model of the

two rooms.


Working Plans

Speedup the model construction process.

◦ The current time required to process one image is a few seconds.

Less than 1/33 seconds in order to achieve real- time.

◦ Study the Matlab functions 祖詒 used in his code.

Continue 禎佑‘s work on multi-camera




Thank you!

Any Questions?




Related subjects :