Virtual Reality

(1)

Virtual Reality

Course ID: CSIE 7633

Lecturer: Ming Ouhyoung, Professor (歐陽明)

課名 : 虛擬實境

Textbook: Understanding Virtual Reality, by William R.

Sherman, Alan B. Craig,

Morgan Kauffmann Inc.

Reference book: Augmented Reality: Principles and Practice, by Dieter Schmalstieg, Tobias Hollerer; Publisher: Addison Wesley

Communications & Multimedia Lab 1

(2)

Course syllabus

• Part I: Virtual Reality

1. Look real, sound real, feel real, smell real, react realistically and in real-time

2. 3D Sound, directional sound

3. Environment Walkthrough, Distributed Interactive Simulation (DIS)

4. Tracking devices: space tracker, tracking algorithms

5. Immersive display: Head Mounted Display, BOOM, Stereo shutter glasses

6. Force Feedback Devices (Joystick, PHANToM etc.) 7. Trajectory prediction algorithms

(3)

Part II, III

• Part II: Display and Visualization

1. Modeling (Solid modeling, build large models, physically based modeling, motion dynamics)

2. Global illumination algorithms( radiosity, volume rendering, scientific sualization)

3. Texture mapping and advanced animation

4. Graphics packages : OpenGL (X window, Win), DirectX(WinX, 9, 10)

Part III: Hardware and accelerators

1. High performance graphics architectures

(Pixel-Planes, Pixel Machine, SGI reality engine, PC Graphics (nVidia, ATI), Accelerator Chips & Cards)

(4)

Table of contents (course slides)

VR Introduction

ARcore; from Google

VR/AR for Medical Applications Human factors

Virtual Reality TERM PROJECT LISTING VR/game hardware

GPU/Graphics HW

pp.01-55 pp.56-57 pp.58-68 pp.70-83 pp.90-92 pp.94-129 pp.130-150

Space tracker/Optical tracker pp.151-153

pp.185-208

3D sound pp.156-169

Virtual worlds (molecules) pp.158-165

Force feedback devices

Advanced rendering: Radiosity Conclusion

Appendix 1 (Oculus HMD

Appendix 2 (Google Cardboard VR HMD)

pp.209-244 pp.244-265 pp. 267-334

4

(5)

Virtual Reality(虛擬實境)

In Chinese

虛擬實境的基本原理在於利用電腦產生並控制一個虛擬的世界，在此虛擬世界中，可以感受到如同處於一個真實的環境。

Textbook Definition

a medium composed of interactive computer simulations that sense the participant's position and actions and replace or augment the feedback to one or more senses, giving the feeling of being mentally immersed or present in the simulation (a virtual world).

The Ultimate Display

The ultimate display would be a room within which the computer can control the existence of matter. Such a display could literally be the Wonderland into which Alice walked. - Ivan E. Sutherland [1965] Father of CG and VR

Cyberspace

a location that exists only in the minds of the participants, often as a result of technology that enables geographically distant people to interactively communicate.

Augmented Reality

a type of virtual reality in which synthetic stimuli are registered with and superimposed on real-world objects; often used to make information otherwise imperceptible to human senses perceptible.

(6)

Terminologies Used

• Virtual Environment

• Micro Worlds

• Virtual Worlds

• Telepresence

• Virtual Reality

(7)

V.R. Conditions to Meet

I. Sutherland[1965]

• Looks real

• Moves realistically

• Feels real

• Sounds/Smells real

• Real-time Response

DEMO video

History

(8)

2015/5 speech, at UNC CS 50 ^th celebration

(“Sketchpad” 1962, Ph.D thesis, by C. Shannon M. Minsky, S. Coons)

8

Turing Award, 1988 ACM Fellow 1994

Member, United States National Academy of Sciences, 1978

National Academy of

Engineering member, 1973 Associate Professor: Harvard Univ.(1965-68)

Professor, Univ. of Utah (1968-) Co-Founder: CS Dept., Univ of Utah

Evans-Sutherland Computer Graphics Company

Alma mater MIT (Ph.D., 1963) Caltech (M.S., 1960) Carnegie Institute of Technology (B.S., 1959)

Evans & Sutherland is the world's first computer

graphics company and has developed advanced

computer graphics technologies for more than five

decades. founded in 1968

(9)

DEMO video to know the devices in VR

• Virtual Drum (stereo glasses/trackers) DEMO

• Joystick (Force Feedback Display) DEMO

• Dressing Mirror/mirror_system_video DEMO DEMO2

• Robotic Surgery (Davinci Surgery System, Surgical_System_Live)

• The Centrifuge Brain Project (Virtual Roller Coaster) DEMO

• VIDEO_Face_Shift (facial expression) DEMO

• Tantofish_HeadMotionTracking (Kinect) DEMO

• Panorama/Cardboard VR HMD (360 degrees display) DEMO

• Building Walkthrough (HMD + Treadmill) DEMO

• TaiChi Motion Capture (Ming DEMO)

(10)

More Demo

• Augumented_Reality_cars (autostereoscopic display)

• 3D_DDR (3D Dance Dance Revolution, 2006, gesture recognition)

• Mocap_skating (motion capture with cameras and markers)

• 2014_2015_CHI (Robin etc. under 2015_CHI directory)

Nail Display (DEMO), Mike_TouchSense (Users' Finger Pads), Mute Robot (Kinect Apps)

• Birdly Project (Flying Eagles, under

2014_Flying_Eagle_SIGGRAPH_EMERGING_TECH directory)

• Flight_simulator (360 degrees flight simulator by US air force) DEMO2

• 2018 Eye surgery AR system (caTAR, demo1 short, demo2 National Geographic TV )

• I-Ping Hung’s: Dun-huang (敦煌石窟) DEMO

(11)

Expected capabilities from taking this course

• Know VR (and Augmented Reality)

• Know human factors in VR

• Understand the basic instruments/devices in VR

• Implementing a graphics simulation (short story for one minute)

• Implementing a VR system

(12)

Sensorama

• Morton Heilig, 1950s

(13)

Sensorama

• The Sensorama, from U.S. Patent #3050870

• The Sensorama machine

(scriptanime.wordpress.com)

• The Sensorama was a machine that is one of the earliest known examples of

immersive, multi-sensory (now known as multimodal) technology.

(14)

• The Sensorama was a mechanical device,

which includes a stereoscopic color display, fans, odor emitters, stereo‐sound system, and a motional chair. While it still functions today, audiences cannot interact with it.

• DEMO video

(15)

VR and recent developments Wearable devices:

Google glasses

Oculus HMD (Appendix 1) Google Cardboard VR

Google Daydream View VR VR HMD (Appendix 2)

Ming Ouhyoung

(16)

Google Glass: Specification

• The core of Google Glass is its tiny prism

display which sits not in your eyeline, but a little above it. You can see what is on the

display by glancing up. The glasses also have

an embedded camera, microphone, GPS and,

reportedly, use bone induction to give you

sound.

(17)

• the glasses are expected "to cost around the price of current smartphones." So that's

around $750/£500, then, possibly with the

help of a hefty Google subsidy.

(18)

Google Glass 2013

(19)

Google Glass

(20)

UNC See-through

Head-mounted display (1989)

(21)

UNC HMD

(22)

UNC-US AF

Head-mounted display (1989)

(23)

Sony HMD

(24)

Canon Mixed Reality HMD

(25)

Meta Glasses: Meta Inc. and Epson Inc.

• https://www.spaceglasses.com/

(26)

• Head up display: used in pilots of fighter planes.

• No need of stereo display.

• In car business: embed front panel display.

• Must have vs Nice to have

(27)

Why important

• Popularity of smart phones

• Constant information display by Internet explorers, Chrome, Safari, etc.

• Not allowed to use smart phones during driving for the driver.

• However, important short messages are

necessary, including Yes/No quick answers.

• Stereoscopic display is not a must, but nice

to have [* ?]

(28)

New VR/AR glasses design

• See through features

• Light weight,

• Independent of eye sight values (near sight, far sight for elders), so that original

eyeglasses can be used.

• Appears to be at least 40 cm ahead.

• Simple gesture recognition, Yes/No

selection, slider up/down, left/right.

(29)

3D/Stereo Display

Sources from ITRI (工研院, 前瞻顯示計畫)

(30)

Types of 3D/Stereo Displays

for 3D TV and Animation

Left and right eye: about 6 degrees difference in

viewing an object, if placed ahead of our head 60 cm away.

(Human Interpupillary Distance: 55 ~ 71 mm, assuming average 6mm interpupillary distance, then the viewing angle is about 5.72 degrees a circle of radius 60 cm, 6mm means 5.72 degrees)

◇ stereoscopic(眼鏡式技術)

◇ polarizing glasses(偏光眼鏡)

◇ shutter glasses (快門眼鏡)

◇ autostereogram (裸視3d技術)

◇ parrallax barrier (光柵式)

◇ lenticular lens (柱狀透鏡式)

(31)

3D Display: Multi-Projectors

(32)

(33)

(34)

Few Technologies in recent years have evoked such fiery discussions in the technical community, and fewer still have sparked such passionate involvement of the humanities and the cultural sector. --- Carl Machover.

VR: The quality of the experience is crucial. To stimulate creativity and productivity, the virtual experience must be credible. The "reality"

must both react to the human participants in physically and perceptually appropriate ways, and confirm to their personal cognitive representations of the micro world in which they are engrossed. --- Carl Machover CG&A 1994.

Virtual Reality

1994 prediction: Research Directions and Future Trend

Ivan Sutherland at the console of the TX-2- Sketchpad Project, MIT, 1963 In 1963, his Ph.D. thesis, "Sketchpad: A Man-

machine Graphical Communications System," used the lightpen to create engineering drawings directly on the CRT.

(35)

Basic Equipment (video demo)

• visual display

– helmet-mounted display – see-through helmet

mounted display – liquid-shutter glasses

• acoustic display – 3D sound

• haptic display

– force-feedback joystick or arm – force feedback data-glove

• 6D Sensors: 6D tracker & data-glove

• Mechanical sensors: robot arm

• Motion sensor: camera

Ivan Sutherland,

Born: 1938, Hastings, Nebraska

Profession: Engineer, Entrepreneur, Capitalist, Professor Position: V.P. and Fellow, Sun Microsystems, Inc.

Education: Ph.D. Massachusetts Institute of Technology M.S. EE California Institute of Technology

B.S. EE Carnegie Institute of Technology (now Carnegie Mellon University)

Patents & Publications: See list

Honors & Professional Societies (partial list):

Smithsonian Computer World Award, 1996

AM Turing Award, Association for Computing Machinery, 1988 First Zworykin Award, National Academy of Engineering, 1972 Member, National Academy of Sciences, since 1978

Member, National Academy of Engineering, since 1973

Member, Institute of Electrical and Electronic Engineers (IEEE) Fellow, Association for Computing Machinery

Latest Accomplishment: Became a historical relic in the Smithsonian

Proudest Accomplishment: Four Grandchildren

(36)

Scientific Visualization

• Evolution

– 2D display – 3D display

– stereoscopic display

• stereoscopic display with haptic display

• Visualization of planetary surfaces(NASA)

• Virtual wind tunnel (NASA NAS project)

• Molecular synthesis (UNC, GMD, U.York)

(37)

Medical applications

• Virtual stereotactic surgery

– from CTs – from MRIs

– Davinci surgery (Demo)

• Ultrasonic imaging: A virtual environment perspective

• Pathological tremor investigation

• Radiation treatment planning

(38)

radiation treatment planning

• In radiotherapy , radiation treatment planning is the process in which a team

consisting of radiation oncologists, radiation therapist, medical physicists and medical

dosimetristsplan the appropriate external beam radiotherapy or

internal brachytherapy treatment technique for a patient with cancer .

(39)

• Forward planning is a technique used

in external-beam radiotherapy to produce a treatment plan. In forward planning, a

treatment Medical physicist places beams into a radiotherapy treatment

planning system which can deliver sufficient radiation to a tumour while both sparing

critical organs and minimising the dose to healthy tissue.

(40)

radiation treatment planning

(41)

Ming’s brain vessels, MRI, for tumor scan

(42)

Virtual Cockpits

• Virtual Coupled Airborne System Simulator (USAF)

• Virtual Environment configurable training aid (Brough, British Aerospace)

• Real and virtual environment configurable training aid (Brough)

– mix a virtual world with a real world.

– concept: mount a color TV camera onto a HMD and couple the result into HMD via a chroma processing system.

The Stewart Motion Platform, Consists of a fixed base and a moving platform connected by six extensible actuators.

Provide sensation of motion by tracking desired

trajectory. Currently

in Professor Li-Chen

Fu’s lab.

(43)

Flight simulator: Airbus A320

• Video demo (Ming) 360 flight simulator demo

• demo2

(44)

VR game: Motorbike Tour

• VIDEO DEMO (Infinity Rider VR Motorbike)

(45)

Telepresence

• control a robot by remote control, especially in hazardous environments

• Virtual environment remote driving experiment(ARRL)

• Man in virtual space (ESA)

• Astronaut simulation (TNO-FEL, NASA)

• Conduction robots (Fujita)

– control in Japan, action in America

– problem: long-distance communication's delay

45 Communications & Multimedia Lab

(46)

CAD/CAM

• Rover 400 car interior design(Brough)

• 3D CAD shape model(NEC)

• Operations with virtual aircraft(Boeing VSX)

• Positioning clothing in 3D(Switzerland)

• Hair style design(NTU CML)

• Software analysis(Tepco)

• Maintenance systems(Brough)

(47)

Entertainment

• Star Wars, Lucasfilm (USA)

• Sega, Nintendo (Japan)

• W Industries (UK)

• Evans & Sutherland (USA)

• Virtual DOOM (id Software & CML)

2014

(48)

Related areas of technology:

multimedia, HDTV, information superhighway, 3D graphics, video games, arcade, theme parks.

America's entertainment economy

(Business Week, March 14, 1994)

As consumers spend big bucks on fun, companies are plowing billions into theme parks, casinos, sports, and interactive TV.

The result: Entertainment is reshaping the U.S.

economy.

Virtual Reality

1994 prediction: Research Directions and Future Trend

(49)

A prediction made in 1994, Future trends:

• The applications market will dominate the technology and research directions.

• Low cost high performance graphics engine, even reaching 1 million Gouround or Phong shaded polygons per second.

• High precision body suit/sensors, 6D trackers, 3D mouse, data glove.

• Low cost force feedback joystick, tactile glove. Authoring tools for VR packages.

• Light weight HMD, eyeglasses, stereo LCD.

Virtual Reality

Research Directions and Future Trend

(50)

Communications & Multimedia Lab 50 Force Feedback Joystick (1995), patented in Taiwan Molecular visualization

Virtual Drum

(51)

(52)

Disney Future World entertainment:

VR (since 1994)

• Disney Vision Adventure: In Virtual Reality, a show that took place

in Innoventions about how Disney movies are made using CGI technology

featuring Iago from Aladdin.

(53)

In Epcot Center, Disney Worlds, Florida, USA, 1996

In Pasteur Research Center, Strasbourg, France, 2005

(54)

Communications & Multimedia Lab 54 Laparoscope(內視鏡顯微手術)

Large number of medical doctors were trained in this way in France

Remote Robot Surgery

Sewing simulation: tie knots

(55)

Data glove (1995)

(56)

(57)

Google’s ARcore

• ARCore uses three key capabilities to integrate

virtual content with the real world as seen through your phone's camera:

• Motion tracking allows the phone to understand and track its position relative to the world.

• Environmental understanding allows the phone to detect the size and location of all type of surfaces:

horizontal, vertical and angled surfaces like the ground, a coffee table or walls.

• Light estimation allows the phone to estimate the environment's current lighting conditions.

(58)

Light estimation allows the phone to estimate the environment's current lighting conditions ^.

(By Alex Wan-Chun Ma, Google Inc, former Ph.D from NTU)

(59)

VR/AR for Medical Applications

Reference:

https://www.youtube.com/watch?

v=pJnvYjn42fE

(60)

• Walter Greenleaf, PhD

• Walter Greenleaf is a research scientist,

entrepreneur, and medical product developer. He is known internationally as an early pioneer in the

medical application of virtual environment

technology and is viewed as one of the founders of the field. For the last 25 years he has developed

innovative clinical systems for use in physical and

cognitive medicine, with a specific focus in the use of

virtual reality technology.

(61)

Early Useful DEMO

(62)

From Walter Greenleaf talks

(63)

(64)

(65)

(66)

(67)

(68)

From Walter Greenleaf talks

(69)

From Walter Greenleaf talks

(70)

Human Factors

• Visual acuity: eye’s ability to distinguish two points of light is limited to 1.5 – 2.0 mm at a distance of 10 meters. (or 2 microns on the retina)

• Sound: at 0 degree Celsius, travels at 331 meters per second

• Hearing range for a young healthy person, 20Hz to 20KHz

• Tactile: (receptors, Pacinian corpuscles) respond to frequencies 30-700Hz

PS: Pacinian corpuscles (感覺神經末梢一種層狀囊包)

(71)

Human factors part 2

• Sensing equilibrium and head rotation:

(human ears) the semicircular ducts (半規管) are sensitive enough to detect angular acceleration of 1 degree per second squared.

(72)

• The semicircular ducts provide sensory

input for experiences of rotary movements.

They are oriented along the pitch, roll, and yaw axes.

(73)

Short questions

• (a) Is a typical RPG (role playing game) game considered VR technology? Why and why not?

• (b) What is motion parallax? How to estimate the thickness of a brick if front of us, if we have only one good eye and the other eye is blind?

• (c) Please describe three cases where it is easy to cause “motion sickness”, why?

•

(74)

Latency for mobile devices

Smartphones Latency（ms）

PsViewer Google Street

View

HTC M9+ None 83.33

HTC U Play 79.2 85.4

Sharp Z2 60.4 100

Sony XZ 100 97.9

Huawei Mate 8 83.33 68.75

(75)

Latency for mobile devices: Part 2

(76)

Motion sickness

• In boat rides

• Driving in mountain roads

• Inconsistent sensing of eyes and ears (semi- circular ducts), and one is fixed while the

other one is moving.

(77)

Effects of field-of-view (FOV) in a virtual environment, Explanations

• A refined scale designed to assess subjects’

engagement, enjoyment, and immersion (E2I) was developed.

• Simulator sickness (SS)

• SS was examined using the Simulator Sickness Questionnaire (SSQ).

(78)

FOV related hypothesis: (Henry B.L.

Duh, IEEE VR2002)

1. We hypothesized that subjects’ E2I scores would increase with increasing FOV.

2. We also hypothesized that subjects would report more SS with increasing FOV.

(Regarding relationships among the E2I scale, its subscales and SS)

3. we hypothesized that presence and enjoyment would be positively correlated, that presence and SS would be

positively correlated, and that enjoyment and SS would be negatively correlated.

(79)

Experiments

data were collected from 10 subjects at four FOVs (60º, 100º, 140º, and 180º).

(80)

These 9 items and their associated factors were as follows. (Crayolaland is the name of the VE we used.)

1. How much did looking at Crayolaland (the VE) involve you, i.e. how much did the visual scene attract your attention? (SF)

2. To what extent did events such as noise occurring outside Crayolaland distract your attention from Crayolaland? (DF)

3. How compelling was your sense of objects moving through space? (SF, RF)

4. How consistent were experiences in the virtual environment; i.e., to what extent did you feel as though you were actually moving through

Crayolaland? (SF, RF)

(81)

5. How completely were you able to actively survey or search the environment using vision? (SF, RF, CF) 6. Were you involved in the memory task to the extent

that you lost track of time? (DF)

7. How much did you have a sense of “being there” in the virtual environment? (SF, RF)

8. During the time of the experience, which was

strongest on the whole, your sense of being in the

driving simulator room or in Crayolaland? (SF, DF, RF) 9. Structure of memory. (SF, RF, CF)

(82)

Results

1. All the data showed the same trend – with increasing FOV, subjects reported more SS as well as increased E2I cores.

The engagement and immersion components of the E2I scale were designed to capture the concept of presence.

(83)

FOV limitations

The combined horizontal FOV is about 180°.

In this study, when the FOV levels increased beyond 120° (140° and 180°), it is likely that subjects’ responses were reduced because

they were mediated by monocular, non-stereo perception.

(84)

General observations under fixed latency

• Latency  Simulator Sickness (SS)

• Display Resolution: resolution increase (1024-

> 1920 -> 2560 ->4K (3840) )  presence (enjoyment)

but also SS

• Motion velocity (head turning) SS

• Without motion blur  SS

• Stereo Vision  better

Immersion/PresenceSS

• FoV  Better presence  SS

(85)

How to analyze your data in experiment?

1. Student’s t test: A t-test is any statistical hypothesis test in which the test statistic follows a Student's t-

distribution under the null hypothesis. It can be used to determine if two sets of data are significantly different from each other.

2. Analysis of variance (ANOVA) is a collection of statistical models used to analyze the differences among group means and their associated procedures (such as "variation" among and between groups), developed by statistician and

evolutionary biologist Ronald Fisher.

3. Multivariate analysis of variance (MANOVA) is simply an ANOVA with several dependent variables.

(86)

One example in experiments

Example of repeated measures

Number Name Test 1 Test 2

1 Mike 35% 67%

2 Melanie 50% 46%

3 Melissa 90% 86%

4 Mitchell 78% 91%

(87)

Pair Name Age Test

1 John 35 250

1 Jane 36 340

2 Jimmy 22 460

2 Jessy 21 200

Example of matched pairs

(88)

just-noticeable difference or JND is the

amount something must be changed in order for a difference to be noticeable, detectable at least half the time (absolute threshold). ^[1][2]

Just Noticeable Difference

(89)

Just Noticeable Difference

To estimate the Details Visibility Change between two images of signifcantly different dynamic range, knowledge of the hypothetical HVS response to given physical contrasts under given adaptation conditions is required. A reasonable prediction for a full range of contrast values is given by the following transducer function that is derived and approximated by Mantiuk et al. [MMS06]:

T(G) = 54:09288 G0:41850; (6) with the following properties:

T(0) = 0 and T(Gthreshold ) = 1; (7)

The transducer function estimates the HVS response to physical contrast

in Just Noticeable Difference (JND) units. Thus for a given contrast

threshold, Gthreshold , a transducer value equals 1 JND. It is important to

note that this measure holds for suprathreshold measurements, since it

not only estimates the detection, but also the magnitude of change.

(90)

Virtual Reality TERM PROJECT LISTING

Pure VR/AR oriented:

1. A 3D sound Synthesizer + HMD

2. An optical tracker (church’s algorithm) + HMD 3. Virtual objects (molecules, etc.) manipulation

with HMD

4. A force feedback application

5. Virtual design using Cardboard HMD (house construction, interior design, lighting

simulation etc.)

6. Choose your own projects: Human Computer Interfaces, Installation Arts, Games, etc.

(91)

VR Term Project (II): graphics oriented:

1. Rendering/Animation of articulated figures in HMD 2. Rigid body animation (Newton’s laws)

3. Ray tracing or Radiosity method for a room / many objects with different materials

4. Volume rendering for a set of tomography slides (Data set from National Taiwan University

Memorial Hospital etc.)

5. Sketch system for animation (Teddy system) 6. Oil painting and water color effects for images 7. 3D morphing and animation with skeleton

mapping

(92)

VR Term Project (III)

8. Motion retargeting (motion of cats likes that of a human)

9. Hardware GPU/GPGPU acceleration research and applications

10. Beautifying Images (Color harmonization, face beautification)

11. Water Rendering, mud simulation

(93)

Research topics

(94)

VR Game in an Arcade

1.Visual Fighter, SEGA, 3D graphics

2.Rad Mobile, SEGA, 2D graphics, seat feedback 3.Visual Racing, SEGA, 3D graphics, wheel feedback

4.Helicopter, NAMCO, 3D graphics, driver seat back impact 5.Motorcycle, SEGA, 2D graphics, Driver body tilting control 6.Train, SEGA, 3D graphics, marching acceleration/

deacceleration

7.mining car, SEGA, 2D graphics, vibration of seat

8.V.R.Galaxy, SEGA, Surrounding wall 3D graphics, seat tilting

9.Roller ciaster, SEGA, Sphere with 3D graphics windows, body roll &

spin

(95)

SEGA SATURN by SEGA

SEGA Saturn was shipping in Nov. , 1994. It is the next generation of SEGA TV-game machine.

Its existence acclaims the new age of 3DCG, and brought this age into your house ! Let's see what it is :

• CPU :

Sega Saturn uses a pair of SH2 , a kind of 32-bit RISC processor by Hitachi, for computing and video output, and one M68000 for audio output. The total computing ability is about 56 MIPS.

• RAM :

Sega Saturn uses 2MB as main RAM, 1.5 MB as V-RAM, 512 KB as Audio Output and another 512 MB as CD-ROM buffer. Total RAM used is 4 MB!

• 3DCG :

Sega Saturn has build-in hardware supporting 3DCG, of course. It is able to display ( compute ) about 200,000 polygons, including flat-shading, gouraud-shading, and texture-mapping. It uses Z-sort to solve the problem of Z value, and about 3 light sources.

• 2DCG :

Sega Saturn also supports 2D image processing. It can display 1 layer of sprite in front of 5 layers of BG, including scaling, stretching, and rotating (by 3 axis). In addition, SS could support animation of its own format (about 2/3 screen sized).

• CD-ROM :

Double-speed CD-ROM driver. Of course it can be used to display normal CD(usually used as BGM), Photo CD, CD-G, and video CD(MPEG1,optional).

(96)

Sony PLAYSTATION Specifications

Emotion Engine'128bit CPU'

運行速度 300MHz

快取記憶體指令 16Kbyte / 資料 8Kbyte + 16Kbyte (SP)

主記憶體 Direct RDRAM

記憶體容量 32MByte

Memory Bus

Bandwidth 每秒3.2GByte

副處理器 FPU(浮點數乘加算器 x 1, 浮點數減算

器 x 1)

向量處理器 VU0 + VU1 (浮點數乘加算器 x 9, 浮

點數減算器 x 3) 三次元CG座標演

算性能每秒6600萬多邊形

浮點數演算性能每秒6.2GFLOPS

壓縮畫像解碼器 MPEG-2

消耗電力 15W

Graphics Synthesizer

運行速度 150MHz

DRAM Bus

Bandwidth 每秒48GByte DRAM Bus Size 2560bit

畫像構成 RGB:Alpha:Z(24:8:32)

最高描繪性能每秒7500萬多邊形

圖像處理機能貼圖/衝撞測繪,煙霧效果,a混合,雙線性/

三線性濾波器

密封地圖,抗變形裝置,複合傳送,透視圖

音聲處理 (SPU2+CPU)

同時發聲數 ADPCM: 48ch (SPU2) + 軟體音源數

取樣頻率 44.1/48KHz

I/O Processor

CPU核心 PlayStation CPU

運行速度 33.8/37.5MHz

Sub-Bus 32bit

輸出入協定 IEEE1394, USB

通訊埠 PCMCIA格式PC卡對應

儲存媒介

CD-ROM DVD-ROM

(97)

SEGA SATURN 主機性能介紹

CPU 中央處理晶片 SH-2 (32bit RISC) 2 棵,和 SH-1 用來管理 CD-ROM(共 3 顆 32 bit 的 CPU )

CD － ROM 光碟機二點五倍速光碟機

軟體讀取方式 CD 或卡匣

影像處理 LSI VDP1 (3D 影像處理) & VDP2(背景處理)

影像功能擴大,縮小,旋轉,變形機能最多五面畫面(可各別擴

大,縮小)

聲音處理 LSI SCSP ( 128 階 DSP, 32 聲道) MC68EOO -3D特殊音效用)

記憶體 Memory 共 36 Mega bits 的 RAM 和 256K 的快取 Memory 特殊功能

最高1677萬色,放大,縮小,視窗,變形 ,TEXTURE

MAPPING 處理 , FM 音源 44,1KHZ 取樣, PCM 32 音源, 32 聲道, 80 萬多邊形處理

售價約新台幣 4500 元

以發行軟體數目將近 1500 種

(98)

Recent development in game consoles (2006-2014)

• Nintendo,

• Sony

• Microfoft

(99)

Nintendo 64 Specs

實體體積:

‧10.25" x 7.5" x 2.57“

‧2.42 lbs Custom CPU:

‧Custom 64-bit MIPS R4300i-class RISC CPU (93.75 MHz)

。64-bit data path, registers, buffer

。5-stage pipeline

‧CPU Benchmarks

。125 Dhrystone MIPS (93 million operations/sec)

。60 SPECint92

。45 SPECfp92 Co-Processor:

‧Custom 64-bit MIPS RISC "Reality Immersion" RCP (62.5 MHz) Built-in Audio/Video Vector Processor (RSP)

‧RCP Benchmarks

。Over half a billion (500,000,000) vector operations/sec

▪10 times more than some Pentium engines

。Built-in Pixel Drawing Processor (RDP) takes care of:

▪Advanced Texture-Mapping - Detail Texturing

- Tri-linear Mip Map Interpolation - Perspective Correction - Environment Mapping - Depth Buffering

▪Color Combiner

▪Anti-Aliasing and Blending

▪Rasterizing

▪Z-Buffering

▪Automatic LOD Management

▪Vertex positioning and transformations

▪Depth, color and texture clipping

▪Transparency (256 levels max)

▪Gouraud Shading Processor/Co-Processor Engine:

‧Contains Over 4 Million Transistors Total

‧Manufactured by NECBased on .35 Micron Process Memory:

‧4 Megabytes (36 megabits) total RAM

‧Rambus DRAM subsystem

。Transfers up to 562.5 MBytes/sec

‧Custom 9-bit Rambus Bus (to the DRAM)

。Runs at 500 MHz max

‧Internal data bus to the RCP is 128-bit

聲音:

‧Stereo 16-bit

‧ADPCM Compression

‧100 PCM channels possible

。Each PCM channel takes 1% of the CPU time

▪Average 16-24 channels

‧Wavetable Synthesis

‧Sampled at 48 KHz max

‧Internal Special Effects

。Voice (w/ Pitch Shifting)

。Gain and Pan

。Reverb and Chorus

‧External (software) Effects Supported 影像:

‧Video Output

。RF、Stereo A/V、S-Video、HDTV

‧Video and Resolution:

。256 x 224 to 640 x 480

▪Limited by TV Standards

▪Flicker Free Interlace Mode

。21-bit color output

。32-bit RGBA Pixel Color Frame Buffer Controller Ports:

‧Four Controller Ports

‧Three-prong Feed Controllers:

‧Digital joypad at left

‧Analog stick in middle

‧Six buttons on the right

。'B' and 'A' buttons

。Four "C Group" buttons

。'L' and 'R' buttons on top

。One "Z Trigger" button on the bottom

‧Memory card port on back

。Initial controller paks start out at 256k

。Paks (up to 2 MB) will be available

。Supports other 'paks' such as a "Jolt Pak"

擴充槽:

‧Cartridge Slot

‧Controller Ports

‧Extension Port (bottom)

‧Memory Expansion option (top front) N64 Console Games:

‧Games begin at 32-128 Megabits

‧Uses JPEG image format for pre-rendered images

‧Produces polygon graphics on the fly

‧On-board hardware decompression; software optional

‧256 Megabit carts max; (four 64 meg ROMs)Downward Compatible

(100)

Nintendo Wii Specs

• System specs

Central Processing Unit (CPU) IBM Broadway 729MHz Graphics Processing Unit (GPU) ATI Hollywood 243MHz Supported Resolution Up to 480p

System Memory 88MB

Internet Connectivity WiFi 802.11 b/g

• Media

Internal Storage 512MB Flash Memory

Optical Drive 12cm Wii Disc & 8cm GameCube Disc. Discs will self-load into the bay.

12cm Disc Capacity 4.7Gb (or 8.5Gb Dual Layer) Memory Expansion 1 x SD Memory

• Backward Compatibility

Downloadable Games NES, SNES, N64, Genesis, Neo-Geo, TurboGrafix16 (and CD) Disc Compatibility GameCube

• Connectivity

Wii Controller Ports 4 x Wireless GameCube Controller Ports 4 Ports GameCube Memory Expansion 2 Ports USB 2.0 2 Ports

Internet Wireless IEEE802.11b/g or a USB LAN adaptor. WiiConnect 24 persistant connection,

even when powered off.

Output ports AV Multi-output port, allowing Composite, S-Video and Component.

• Controller

Connection method Bluetooth (wireless)

Wiimote Buttons 3 axis motion sensor, + Direction pad, A, B, Minus, Home, Power button, 1 and

2

Nunchuck Buttons 3 axis motion sensor, Analog stick, C and Z

Force feedback (rumble) Yes

Other features Wiimote has a speaker and an expansion portCommunications & Multimedia Lab 100

(101)

Nintendo Wii

• First six weeks of sales volume in Japan (2006/11-2007):

– PS3: 310K, Xbox360: 110K,Wii: 1130K

• Why is it successful?

1) Human Computer Interface breakthrough VS. Graphics rendering power 2) Make the player pie bigger: including the traditional non-game players:

such as grandparents and grandchildren, not just youth groups.

3) Traditional game players are usually lonely in one room, with

networking connected to outside world only, while the Wii includes everyone in one game in one room (a social medium).

4) Single-handed remote control, much less buttons and control.

5) Good to "PLAY" vs. Good to "LOOK“

6) 3D orientation and location control, added force feedback: vibration and sound,

• My own experience: DEMO

• SIGGRAPH2006_EmergingTech: 3D_DDR

• VR: The quality of the experience is crucial. To stimulate creativity

and productivity, the virtual experience must be credible. The "reality"

must bothreact to the human participants in physically and

perceptually appropriate ways, and confirm to their personal cognitive representations of themicroworld inwhich they are engrossed.---

CarlMachoverCG&A 1994.

(102)

Nintendo Wii

(103)

Wii Market performance 2007

• (3) Market share: (2007/2)

• First six weeks of sales volume in Japan:

• PS3: 310K, Xbox360: 110K, Wii: 1130K

• (4) Supports: N64, Game Cube, Wi-Fi wireless network with browser, 512 M RAM, SD reader etc.

• • As a comparison to the prediction in 2005:

• Market Now: (BusinessWeek 2005, February 28) Games: US24.5 billions revenue in 2004, larger than the movie box-office

business Game software sales: 7.3 billion, (1) EA (Electronic Art): $3 billion, and top five game developers accounted for 56% of the revenue. Likely winners: SONY, EA (Consoles, games, movies, only Sony has it all) At risk: Ninendo, Activision (second largest independent game developer), Walt Disney 2004:

game consoles: Sony (56.4%), Microsoft (24.9%), Nitendo(18.7%) 39% of video gamers are women.

(104)

Nintendo: Wii remote Why is it successful?

(1) Human Computer Interface breakthrough VS. Graphics rendering power (2) Make the player pie bigger: including the traditional non-game players:

such as grandparents and grandchildren, not just youth groups.

(3) Traditional game players are usually lonely in one room, with networking connected to outside world only, while the Wii includes everyone in one game in one room (a social medium).

(4) Single-handed remote control, much less buttons and control.

(5) Good to “PLAY” vs. good to “LOOK”

(6) 3D orientation and location control, added force feedback: vibration and sound,

VR: The quality of the experience is crucial. To stimulate creativity and productivity, the virtual experience must be credible. The

"reality" must both react to the human participants in physically and perceptually appropriate ways, and confirm to their personal

cognitive representations of the microworld in which they are engrossed. --- Carl Machover CG&A 1994.

(105)

Trend: Virtual Reality in 1994 and 2005

Research Directions and Future Trend

• Market Now: (BusinessWeek 2005, February 28)

– Games: US24.5 billions revenue in 2004, larger than the movie box-office business

– Game software sales: 7.3 billion,

• EA (Electronic Art): $3 billion, and top five game developers accounted for 56% of the revenue.

– Likely winners: SONY, EA (Consoles, games, movies, only Sony has it all) – At risk: Nintendo, Activision (second largest independent game

developer), Walt Disney

– 2004: game consoles: Sony (56.4%), Microsoft (24.9%), Nitendo(18.7%) 39% of video gamers are women.

• Compared to past predictions:$250 million worth of VR products and services will be shipped in 1994.

• One billion dollars by 1997 (IEEECG&A, p15, Vol 14, No 1, Jan 1994)

– S.E.Tice, President of S.E.Tice Consulting, Inc.

– Carl Machover, President of Machover Associates.

• VR is growing at annual rates on the order of 60% (twice the growth

rate graphics experienced 25 years ago)

(106)

• Three company’s revenue and profits in 1994:

 Nintendo has $3.2B in sales, approximately $850M net(1/4)

 Sun Microsystems has $2.47B in sales, approximately

$110M net(1/24).

 Intel has $3.92B in sales, approximately $650M net

1/6).

(107)

Entertainment

SEGA. David Rosen started SEGA in 1954 in Japan.(4 billion 1993; 2.2 billion 1992; 1.2 billion 1991)

SEGA will exploit the efficiency of electronics over iron and steel to creat a new entertainment form: virtual-reality theme parks. Parks packed with VR will occupy perhaps 3% of the land area of Florida's Disney World, so they can be put in densely populated areas.

Nintendo(任天堂) Market share in Japan 78%, in US 51%, in Europe 41%

Revenue distribution:(SEGA) 1/6 for ARCADES

1/6 for Home-use software

1/3 for Home-use game machines

VR related games are already approaching 1/2 of ARCADES.

Virtual Reality

(1994 prediction)Research Directions and Future Trend

(108)

3D graphics & VR games worth 2/3 of 8 billion dollars business => (Nintendo + SEGA)

Data from Business Week. Feb 21, p38-44

Nintendo & SEGA plus others expect to make 40 million video-game machines over the next five years

Virtual Reality

(1994) Research Directions and Future Trend

Buyers: ChipMaker:

Nintendo (project Reality game machine, due 1995) SEGA(Saturn game machine, due 1994)

Nintendo

Silicon Graphics Hitachi

V800 low-power consumption chip

(109)

New player: Microsoft Xbox, Xbox 360, Xbox one

• Strategy: no profit or negative profit for hardware (game console), but profitable from software game titles.

Two billion US$ loss during the first three years.

• No other OEM/ODM big countries such as

Taiwan and South Korea can enter this market in the future.

Microsoft's 2013 Q2(October to December Quarter):

record $24.52 billion revenue and 3.9 million Xbox One sales

(110)

• For both Microsoft and Sony, their latest-

generation video game console hardware is unprofitable at the time of release, requiring the companies to subsidize it initially.

However, these companies easily can largely compensate for their losses though sales of highly lucrative game titles,

(111)

Huge loss between 2001 to 2005

• 2001: Microsoft could lose as much as $2 billion on Xbox--potentially selling the game console at a loss for three years or more--before breaking even in fiscal 2005, according to a Merrill Lynch report released Tuesday.

• The expected losses represent a standard practice in the video game industry of relying on income from software sales and licensing to subsidize hardware costs. The practice helped drive Sega out of the

hardware business earlier this year because the company could no longer come up with money to cover losses associated with production of its

Dreamcast console.

(112)

Microsoft Xbox 360

(113)

Microsoft Xbox 360 Specs

Custom IBM PowerPC-based CPU

•3 symmetrical cores running at 3.2 GHz each

•2 hardware threads per core; 6 hardware threads total

•1 VMX-128 vector unit per core; 3 total

•128 VMX-128 registers per hardware thread

•1 MB L2 cache CPU Game Math

Performance

•9 billion dot product operations per second

Custom ATI Graphics Processor

•500 MHz

•10 MB embedded DRAM

•48-way parallel floating-point

dynamically-scheduled shader pipelines

•Unified shader architecture Polygon Performance •500 million triangles per second

Pixel Fill Rate •16 gigasamples per second fillrate using 4X MSAA

Shader Performance •48 billion shader operations per second

Memory

•512 MB GDDR3 RAM

•700 MHz DDR

•Unified memory architecture

Memory Bandwidth

•22.4 GB/s memory interface bus bandwidth

•256 GB/s memory bandwidth to EDRAM

•21.6 GB/s front-side bus Overall System

Floating-Point Performance

•1 TFLOP

Storage

•Detachable and upgradeable 20 GB hard drive

•12X dual-layer DVD-ROM

•Memory unit support starting at 64 MB

I/O •Support for up to 4 wireless game controllers

•3 USB 2.0 ports

•2 memory unit slots

Optimized for Online

•Instant, out-of-the-box access to Xbox Live features, including Xbox Live Marketplace for downloadable content, Gamer Profile for digital identity and voice chat to talk to friends while playing games, watching movies or listening to music

•Built in Ethernet Port

•Wi-Fi Ready: 802.11 A, B and G

•Video Camera Ready

Digital Media Support

•Support for DVD-Video, DVD-ROM, DVD-R/RW, DVD+R/RW, CD-DA, CD-ROM, CD-R, CD-RW, WMA CD, MP3 CD, JPEG Photo CD

•Stream media from portable music devices, digital cameras, Windows XP PCs

•Rip music to Xbox 360 hard drive

•Custom playlists in every game

•Windows Media Center Extender built in

•Interactive, full screen 3D visualizers

HD Game Support

•All games supported at 16:9, 720p and 1080i, anti-aliasing

•Standard definition and high definition video output supported

Audio

•Multichannel surround sound output

•Supports 48 KHz 16-bit audio

•320 independent decompression channels

•32-bit audio processing

•Over 256 audio channels

System Orientation •Stands vertically or horizontally Customizable Face

Plates •Interchangeable to personalize the console

(114)

Microsoft Xbox Kinect

Kinect is a motion sensing input device by Microsoft for the Xbox 360 video game console and Windows PCs. Based around a webcam-style add-on peripheral for the Xbox 360 console, it enables users to control and interact with the Xbox 360 without the need to touch a game controller, through a natural user interface using gestures and spoken commands. The project is aimed at broadening the Xbox 360's audience beyond its typical gamer base. Kinect competes with the Wii Remote Plus and

PlayStation Move with PlayStation Eye motion controllers for the Wii and PlayStation 3 home consoles, respectively. A version for Windows was released on February 1, 2012.

Microsoft released a non-commercial Kinect software development kit for Windows 7

on June 16, 2011, with a commercial version following at a later date. This SDK will

allow .NET developers to write Kinecting apps in C++/CLI, C#, or Visual Basic .NET.

(115)

Microsoft Xbox Kinect Specs

Sensor

Colour and depth-sensing lenses Voice microphone array

Tilt motor for sensor adjustment

Fully compatible with existing Xbox 360 consoles Field of View

Horizontal field of view: 57 degrees Vertical field of view: 43 degrees Physical tilt range: ± 27 degrees Depth sensor range: 1.2m - 3.5m Data Streams

320×240 16-bit depth @ 30 frames/sec 640×480 32-bit colour@ 30 frames/sec 16-bit audio @ 16 kHz

Skeletal Tracking System

Tracks up to 6 people, including 2 active players Tracks 20 joints per active player

Ability to map active players to Live Avatars Audio System

Live party chat and in-game voice chat (requires Xbox Live Gold Membership) Echo cancellation system enhances voice input

Speech recognition in multiple

(116)

Microsoft Xbox Kinect

Widely used in researches:

Shahram Izadi, David Kim, Otmar Hilliges, David

Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew

Davison, and Andrew Fitzgibbon,KinectFusion: Real- time 3D Reconstruction and Interaction Using a Moving Depth Camera, ACM Symposium on User Interface Software and Technology, October 2011

REALTIME PERFORMANCE-BASED FACIAL ANIMATION

Thibaut Weise, Sofien Bouaziz, Hao Li, Mark Pauly ACM Transactions on Graphics, Proceedings of the 38th ACM SIGGRAPH Conference and Exhibition 2011, 08/2011 -SIGGRAPH

2011[paper] [video] [fast forward] [bibtex]

(117)

Microsoft Xbox Kinect

(118)

Sony PlayStation 3,4

• Sony has to sell its PC/NB and TV division in 2013/2014

(119)

Sony PlayStation 3

PlayStation 3 slim (PS3 CECH-2000 model) is the revamped version of Sony's seventh generation video game console - PS3, which was launched by Sony Computer Entertainment in 2009. Other than the new sleek design, these PS3 slim console also boasts of potent mix of new (such as the removable hard drive) and old features (such as online gaming, a high-definition optical disc format and Blu-ray Disc technology), which make gaming easier and interesting.

While the first two consoles of these series were armed with 120GB and

250GB HDD, they were eventually discontinued to make way for the two

existing models - the 160GB slim and the 320GB slim in 2010. (The 320GB

PS3 slim though, is only available as a part of the PS3 bundle.) As with

their predecessors, both models come with a Blu-ray drive which can be

used for multimedia purposes as well. Some of the most popular PS3

games include Medal of Honor - Frontline, Far Cry 2, Demon's Souls, MAG,

inFamous, Resident Evil, Kill Zone 2, etc. At the same time, the addition of

PlayStation Move which facilitates motion gaming has also added to the

popularity of PS3 slim console.

(120)

Communications & Multimedia Lab 120 CPU:Cell Processor

PowerPC-base Core @3.2GHz 1 VMX vector unit per core 512KB L2 cache

7 x SPE @3.2GHz

7 x 128b 128 SIMD GPRs 7 x 256KB SRAM for SPE

* 1 of 8 SPEs reserved for redundancy total floating point performance: 218 GFLOPS

GPU:RSX @550MHz

1.8 TFLOPS floating point performance Full HD (up to 1080p) x 2 channels

Multi-way programmable parallel floating point shader pipelines Sound:Dolby 5.1ch, DTS, LPCM, etc. (Cell-base processing) Memory:

256MB XDR Main RAM @3.2GHz 256MB GDDR3 VRAM @700MHz System Bandwidth:

Main RAM: 25.6GB/s VRAM: 22.4GB/s

RSX: 20GB/s (write) + 15GB/s (read) SB: 2.5GB/s (write) + 2.5GB/s (read)

System Floating Point Performance:2 TFLOPS Storage:

HDD

Detachable 2.5" HDD slot x 1 I/O:

USB:Front x 4, Rear x 2 (USB2.0) Memory Stick:standard/Duo, PRO x 1 SD:standard/mini x 1

Sony PlayStation 3 Specs

CompactFlash:(Type I, II) x 1

Communication:Ethernet (10BASE-T, 100BASE-TX, 1000BASE-T) x3 (input x 1 + output x 2)

Wi-Fi:IEEE 802.11 b/g

Bluetooth:Bluetooth 2.0 (EDR) Controller:

Bluetooth (up to 7) USB2.0 (wired) Wi-Fi (PSP® ) Network (over IP) AV Output:

Screen size:480i, 480p, 720p, 1080i, 1080p HDMI:HDMI out x 2

Analog:AV MULTI OUT x 1

Digital audio:DIGITAL OUT (OPTICAL) x 1 CD Disc media (read only):

PlayStation CD-ROM PlayStation 2 CD-ROM CD-DA (ROM), CD-R, CD-RW SACD Hybrid (CD layer), SACD HD

DualDisc (audio side), DualDisc (DVD side) DVD Disc media (read only):

PlayStation 2 DVD-ROM PLAYSTATION 3 DVD-ROM

DVD-Video: DVD-ROM, DVD-R, DVD-RW, DVD+R, DVD+RW Blu-ray Disc media (read only):

PLAYSTATION 3 BD-ROM

BD-Video: BD-ROM, BD-R, BD-RE

(121)

Sony Playstation 4

• The PlayStation 4 uses a processor developed by AMD in cooperation with Sony.

• The CPU consists of two quad-core Jaguar modules totaling 8 x86-64 cores.

• The GPU consists of 18 compute units to produce a theoretical peak performance of 1.84 TFLOPS.

• The system's GDDR5 module contains 8 GB of GDDR5 memory, 16 times the amount of RAM found in the PS3 and is expected to give the console considerable longevity.

(122)

Sony PlayStation Move

The PlayStation Move combines a video camera with a physical controller packed with motion-sensing electronics, making it the technological cross between Kinect and the Nintendo Wii. The Move Motion Controller, or "wand,"

combines a gyroscope, accelerometer, and magnetic sensor (a sort of digital

"compass" that uses the Earth's magnetic field to determine the controller's

orientation) to track the controller in three dimensions, while the glowing ball

at the end gives the PlayStation Eye camera a visual reference for handling

aiming, cursor movement, and other motion. Like Kinect, PlayStation Move

requires room to function; Sony recommends 5 to 9 feet between the player and

the PlayStation Eye, but you can play anywhere from 2 to 10 feet of the camera.

(123)

Sony PlayStation Move Specs

•

PlayStation Move Specifications and Details

•

Features

"The latency for the Playstation Move is under one frame" - Scott Rohde, vice president of product development, SCEA.

•

PlayStation® Move motion controller

Three-axis gyroscope

Three-axis accelerometer

Terrestrial megnetic field sensor

Colour-changing sphere for Playstation Eye tracking Bluetooth® technology

Vibration feedback

•

PlayStation® Move sub-controller

Built-in lithium-ion rechargeable battery Bluetooth® technology

2 DUALSHOCK® or SIXAXIS® Wireless Controller replacement capability.

•

PlayStation® Eye

Built-in four-capsule microphone array Echo cancellation

Background noise suppression

•

Price

"Under $100" (£47)

Virtual Reality

Virtual Reality

Course ID: CSIE 7633

Lecturer: Ming Ouhyoung, Professor (歐陽明)

課名 : 虛擬實境

Textbook: Understanding Virtual Reality, by William R.

Sherman, Alan B. Craig,

Morgan Kauffmann Inc.

Reference book: Augmented Reality: Principles and Practice, by Dieter Schmalstieg, Tobias Hollerer; Publisher: Addison Wesley

Course syllabus

• Part I: Virtual Reality

1. Look real, sound real, feel real, smell real, react realistically and in real-time

2. 3D Sound, directional sound

3. Environment Walkthrough, Distributed Interactive Simulation (DIS)

4. Tracking devices: space tracker, tracking algorithms

5. Immersive display: Head Mounted Display, BOOM, Stereo shutter glasses

6. Force Feedback Devices (Joystick, PHANToM etc.) 7. Trajectory prediction algorithms

Part II, III

• Part II: Display and Visualization

1. Modeling (Solid modeling, build large models, physically based modeling, motion dynamics)

2. Global illumination algorithms( radiosity, volume rendering, scientific sualization)

3. Texture mapping and advanced animation

4. Graphics packages : OpenGL (X window, Win), DirectX(WinX, 9, 10)

Part III: Hardware and accelerators

1. High performance graphics architectures

(Pixel-Planes, Pixel Machine, SGI reality engine, PC Graphics (nVidia, ATI), Accelerator Chips & Cards)

Table of contents (course slides)

VR Introduction

ARcore; from Google

VR/AR for Medical Applications Human factors

Virtual Reality TERM PROJECT LISTING VR/game hardware

GPU/Graphics HW

pp.01-55 pp.56-57 pp.58-68 pp.70-83 pp.90-92 pp.94-129 pp.130-150

Space tracker/Optical tracker pp.151-153

pp.185-208

3D sound pp.156-169

Virtual worlds (molecules) pp.158-165

Force feedback devices

Advanced rendering: Radiosity Conclusion

Appendix 1 (Oculus HMD

Appendix 2 (Google Cardboard VR HMD)

pp.209-244 pp.244-265 pp. 267-334

Virtual Reality(虛擬實境)

In Chinese

Textbook Definition

a medium composed of interactive computer simulations that sense the participant's position and actions and replace or augment the feedback to one or more senses, giving the feeling of being mentally immersed or present in the simulation (a virtual world).

The Ultimate Display

The ultimate display would be a room within which the computer can control the existence of matter. Such a display could literally be the Wonderland into which Alice walked. - Ivan E. Sutherland [1965] Father of CG and VR

Cyberspace

a location that exists only in the minds of the participants, often as a result of technology that enables geographically distant people to interactively communicate.

Augmented Reality

a type of virtual reality in which synthetic stimuli are registered with and superimposed on real-world objects; often used to make information otherwise imperceptible to human senses perceptible.

Terminologies Used

• Virtual Environment

• Micro Worlds

• Virtual Worlds

• Telepresence

• Virtual Reality

V.R. Conditions to Meet

I. Sutherland[1965]

• Looks real

• Moves realistically

• Feels real

• Sounds/Smells real

• Real-time Response

DEMO video

History

2015/5 speech, at UNC CS 50 th celebration

(“Sketchpad” 1962, Ph.D thesis, by C. Shannon M. Minsky, S. Coons)

Turing Award, 1988 ACM Fellow 1994

Member, United States National Academy of Sciences, 1978

National Academy of

Engineering member, 1973 Associate Professor: Harvard Univ.(1965-68)

Professor, Univ. of Utah (1968-) Co-Founder: CS Dept., Univ of Utah

Evans-Sutherland Computer Graphics Company

Alma mater MIT (Ph.D., 1963) Caltech (M.S., 1960) Carnegie Institute of Technology (B.S., 1959)

Evans & Sutherland is the world's first computer

graphics company and has developed advanced

computer graphics technologies for more than five

decades. founded in 1968

2015/5 speech, at UNC CS 50 ^th celebration