VR Project Orientation
Ching-Chi Lin
Institute of Information Science, Academia Sinica
Department of Computer Science and Information Engineering, National Taiwan University
Agenda
Introduction
Research Issues
Current Project and Working Items
◦ Real-time Room fusion
◦ Perspective Changing
What is VR
Virtual Reality(VR) replicates an
environment that simulates a physical
presence in places in the real world or an imagined world, allowing the user to
interact in that world.[1]
◦ Artificially create sensory experiences, which can include sight, touch, hearing, and smell.
[1] https://en.wikipedia.org/wiki/Virtual_reality
Side-by-side Image
Augmented Reality(AR)
Mixed Reality
VR: virtual objects + virtual environment
AV: real objects + virtual environment
AR: virtual objects in real environment
VR Devices
Head Mount Device(HMD):
◦ Oculus Rift
◦ Samsung Gear VR
◦ Sony PlayStation VR
◦ HTC Vive
◦ Google Cardboard
◦ ...
Position Tracker, controller, joystick, …etc.
VR Applications
Games
◦ FPS, adventure, music, … etc.
Media
◦ 3D movie, 360 ° videos
◦ VR movies
Art
Architecture
…
Research Issues
Improve user experiences
◦ Make the VR/AR effect more realistic.
◦ Reduce the dizziness
◦ Reduce random wobble of the picture.
Research Issues(Cont.)
Mobile VR/AR
◦ Find a good balance between QoE and battery life of a mobile device.
◦ Dynamically scale resolution and refresh rate to reduce mobile GPU power consumption.
◦ Use user head movement information to decide the appropriate QoE metrics.
Research Issues(Cont.)
Cluster-Assisted VR
◦ Move the computations from local device to a server cluster, such that users without high-
end hardware equipments can also experience VR fluently.
◦ How to allocate sufficient computing
resources to different VR applications in the cluster?
Research Issues(Cont.)
Develop new VR/AR applications.
Current Project
Live Reality Fusion
◦ Combine the real-time images from two different locations into one.
◦ ex: fuse two seminar rooms for oversea joint meeting; “wall removing” for Interior design.
◦ Cooperate with Dr. Wang’s group.
Two Approaches
One-360’-camera
Multiple-camera
One-360’-Camera Approach
Only one 360’ camera in the remote room.
A server captures and process the stream from camera, then fuse the processed
stream into the video stream captured by
user HMD.
One-360’-Camera Approach(Cont.)
Remote room
Observation Room
Multiple-Camera Approach
Remote room
Observation Room
Steps
Break the video captured by 360’ camera into frames.
◦ Panoramic image
Construct a cuboid model of the remote room.
◦ Transform the panoramic frame into cube.
◦ Transform the cube into cuboid.
Steps(Cont.)
Compute what an user should see from the current angle.
◦ By projection or other methods.
Replace the wall in observation room
with the projected image of the remote
room.
Short-Term Goal
Fuse R106 and R107 in IIS.
◦ The wall between the two rooms can be removed to achieve real “room fusion”.
◦ Take picture/video as ground-truth.
Current Progress
Dr. Wang’s RA, 祖詒, has completed the
construction of the cuboid model of the
two rooms.
Working Plans
Speedup the model construction process.
◦ The current time required to process one image is a few seconds.
Less than 1/33 seconds in order to achieve real- time.
◦ Study the Matlab functions 祖詒 used in his code.
Continue 禎佑‘s work on multi-camera
interpolation.
Finish
Thank you!
