Bin Sheng
Introduction
CS337: Introduction to Computer Graphics and Virtual reality
Still from Pixar’s Piper, 2016
© 9/8/2016 1/38
Bin Sheng
Computer graphics generally means creation, storage and manipulation of models and images
Such models come from diverse and expanding set of fields including physical, biological, mathematical, artistic, and conceptual/abstract structures
What is Computer Graphics? (1/2)
Frame from animation by William Latham, shown at SIGGRAPH 1992.
Latham creates his artwork using rules that govern patterns of natural forms.
© 9/8/2016 2/38
Bin Sheng
William Fetter coined term “computer graphics” in 1960 to describe new design methods he was pursuing at
Boeing for cockpit ergonomics
Created a series of widely reproduced images on “pen plotter” exploring cockpit design, using 3D model of human body.
“Perhaps the best way to define computer graphics is to find out what it is not. It is not a machine. It is not a computer, nor a group of computer programs. It is not the know-how of a graphic designer, a programmer, a writer, a motion picture specialist, or a reproduction specialist.
Computer graphics is all these – a consciously managed and documented technology directed toward communicating information accurately and descriptively.”
Computer Graphics, by William A. Fetter, 1966
3/38
© 9/8/2016
Bin Sheng
User controls content, structure, and appearance of objects and their displayed images via rapid visual feedback
Basic components of an interactive graphics system
input (e.g., mouse, stylus, multi-touch, in-air fingers…)
processing (and storage of the underlying representation/model)
display/output (e.g., screen, paper-based printer, video recorder…)
First truly interactive graphics system,
Sketchpad, pioneered by Ivan Sutherland 1963 Ph.D. thesis Sketchpad, A Man-Machine Graphical Communication System
Used TX-2 transistorized “mainframe”
at MIT Lincoln Lab
What is Interactive* Computer Graphics? (1/2)
Note CRT monitor, light pen, and function-key panels – the “organ console” showing bi- manual operation
* Sometimes called real-time computer graphics, and in certain
contexts, real-time rendering 4/38
© 9/8/2016
© 9/8/2016 5/38
Bin Sheng
Almost all key elements of interactive graphics system are expressed in first paragraph of Sutherland’s 1963 Ph.D. thesis
The Sketchpad system uses drawing as a novel communication medium for a computer. The system contains input, output, and computation programs which enable it to interpret information drawn directly on a computer display. Sketchpad has shown the most usefulness as an aid to the understanding of processes, such as the motion of linkages, which can be described with pictures. Sketchpad also makes it easy to draw highly repetitive or highly accurate drawings and to change drawings previously drawn with it…
What is Interactive Computer Graphics? (2/2)
6/38
Bin Sheng
Today, still use non-interactive batch mode (aka offline rendering) for final production- quality video and film (special effects – FX). Rendering a single frame of The Good Dinosaur (a 24 fps movie) averaged 48 hours on a 30,000-core render farm!
Pixar’s Render Farm Still from The Good Dinosaur
Statistics from https://www.fxguide.com/featured/making-the-world- 7/38
of-pixars-the-good-dinosaur/
Bin Sheng
Hardware revolution
Moore’s Law: every 12-18 months, computer power
improves by factor of 2 in price / performance as size shrinks
Newest CPUs are 64-bit with 2, 4, 6, 8, even up to 18 cores
Intel Skylake – consumer processor with 4 cores, 8 threads, and a fully featured graphics chip built in to the processor
Significant advances in commodity graphics chips every 6 months vs. several years for general purpose CPUs
NVIDIA GeForce GTX Titan X… 3072 cores, 12GB memory, and 7 teraflops of processing power in a single chip
Enabling Modern Computer Graphics (1/5)
NVIDIA statistics from http://wccftech.com/nvidia-pascal-gp102-gpu- 8/38
titan-graphics-card
Bin Sheng
Graphic subsystems
Offloads graphics processing from CPU to chip designed for doing graphics operations quickly
nVidia GeForce™, AMD Radeon™, and Intel HD and Iris Pro Graphics
GPUs originally designed to handle special-purpose graphics computations
Increasingly, GPUs used to parallelize other types of computation (known as GPGPU, or General-Purpose Computing on the Graphics Processing Unit)
Hardware show and tell: Dept’s NVIDIA GeForce GTX 460s
1.35 GHz clock, 1GB memory, 37.8 billion pixels/second fill rate
Old cards: GeForce 7300 GT: 350 MHz clock, 256 MB memory, 2.8 billion fill rate
© 9/8/2016 9/38
Bin Sheng
Input Devices
Mouse, tablet & stylus, multi-touch, force feedback, and other game controllers (e.g., Wii U), scanner, digital camera (images, computer vision), etc.
Body as interaction device
http://youtu.be/zXghYjh6Gro
Enabling Modern Computer Graphics (3/5)
Xbox Kinect Leap Motion Nimble UX
10/38
Bin Sheng
Many form factors
Smartphones/laptops/desktops/tablets
Smart watches
Head-mounted displays (HMDs)
Oculus bird simulator video
3D immersive virtual reality spaces
Brown’s old Cave
Microsoft’s first Surface
Apple iPhone Android Phones
Google Cardboard Tablets
Microsoft Hololens
Android Wear Apple Watch
11/38
Vive Oculus Rift
Bin Sheng
Old Cave:
4 1024 x 786 projectors on 8’ x 8’ walls (8-10 pixels per inch)
Too low resolution and brightness for many applications, and got worse (brightness, contrast deteriorated over time)
New Cave:
69 projectors onto cylindrically
curved screen 8’ radius, floor, ceiling
140 million pixels
Powered by a ~69 gpu cluster
No right angles, up to 40 pixels per inch (can’t see individual pixels at normal viewing distance)
Digression: Cave Redesign
Brown’s new Cave, the YURT
12/38
© 9/8/2016
Bin Sheng
Software Improvements
Algorithms and data structures
Modeling of materials
Rendering of natural phenomena
“Acceleration data structures” for ray tracing and other renderers
Parallelization
Most operations are embarrassingly parallel: calculating value of one pixel is often independent of other pixels
Distributed and Cloud computing
Send operations to the cloud, get back results, don’t care how
Rendering even available as internet service!
13/38
© 9/8/2016
Bin Sheng
Character Displays (1960s – now)
Display: text plus alphamosaic pseudo-graphics (ASCII art)
Object and command specification: command-line typing
Control over appearance: coding for text formatting (.p = paragraph, .i 5 = indent 5)
Application control: single task
Environmental Evolution (1/5)
14/38
© 9/8/2016
Bin Sheng
Vector (Calligraphic, Line Drawing)
Displays (1963 – 1980s)
Display: line drawings and stroke text; 2D and 3D transformation hardware
Object and command specification: command-line typing, function keys, menus
Control over appearance: pseudo-WYSIWYG
Application control: single or multitasked, distributed computing pioneered at Brown via mainframe host <-> minicomputer satellite
Term “vector” graphics survives as “scalable vector graphics” SVG library from Adobe and W3C – shapes as transformable objects rather than just bitmaps
15/38
© 9/8/2016
Bin Sheng
2D bitmap raster displays for PCs and workstations (1972 at Xerox PARC - now)
Display: windows, icons, legible text, “flat earth” graphics
Note: late 60’s saw first use of raster graphics, especially for flight simulators
Minimal typing via WIMPGUI (Windows, Icons, Menus, Pointer): point-and-click selection of menu items and objects, direct manipulation (e.g., drag and drop), “messy desktop”
metaphor
Control over appearance: WYSIWYG (which is really
WYSIAYG, What You See Is All You Get – not pixel-accurate or controllable)
Application control: multi-tasking, networked client-server computation and window management (even “X terminals”)
Environmental Evolution (3/5)
Above, a classic WIMP interface. The technology, at its core, remains largely
the same today. Below, a modern WIMP interface.
16/38
© 9/8/2016
Bin Sheng
3D graphics workstations (1984 at SGI – now)
could cost up to $1M for high-end!
Display: real-time, pseudo-realistic images of 3D scenes
Object and command specification: 2D, 3D and N-D input devices (controlling 3+ degrees of freedom) and force feedback haptic devices for point-and-click, widgets, and direct manipulation
Control over appearance: WYSIWYG (still WYSIAYG)
Application control: multi-tasking, networked
(client/server) computation and window management
Graphics workstations such as these have been replaced with commodity hardware (CPU + GPU), e.g., our MaxBuilts + Nvidia cards
17/38
© 9/8/2016
Bin Sheng
High-end PCs with hot graphics cards (nVidia GeForce™, AMD Radeon™) have supplanted graphics workstations
Such PCs are clustered together over high speed buses or LANs to provide
“scalable graphics” to drive tiled PowerWalls, Caves, etc.
Also build GPU-clusters as number crunchers, e e.g., protein folding, weather prediction
Now accessible to consumers via technologies like NVIDIA’s
SLI (Scalable Link Interface) bridge
Environmental Evolution (5/5)
You can put multiple GPUs together in your computer using SLI.
18/38
© 9/8/2016
Bin Sheng
Vector (calligraphic, stroke, random- scan)
Raster (TV, bitmap, pixmap) used in displays and laser printers
Driven by display commands
(move (x, y), char(“A”) , line(x, y)…)
Survives as “scalable vector graphics”
Driven by array of pixels (no semantics, lowest form of representation)
Note “jaggies” (aliasing errors) due to discrete sampling of continuous primitives
Ideal
Drawing Vector
Drawing Outline Filled
19/38
© 9/8/2016
Bin Sheng
Graphics library/package is
intermediarybetween application and display hardware (Graphics System)
Application program maps application objects to views (images) of those objects by calling on graphics library. Application model may contain lots of non-graphical data (e.g., non-geometric object properties)
User interaction results in modification of image and/or model
This hardware and software framework is 5 decades old but is still useful
Conceptual Framework for Interactive Graphics
Graphics System/
Application GPU Model / database
Software Hardware
Application program
Graphics Library
20/38
© 9/8/2016
Bin Sheng
Examples: OpenGL™, DirectX™, Windows Presentation Foundation™
(WPF), RenderMan™, HTML5 + WebGL™
Primitives (characters, lines, polygons, meshes,…)
Attributes
Color, line style, material properties for 3D
Lights
Transformations
Immediate mode vs. retained mode
immediate mode: no stored representation, package holds only attribute state, and application must completely draw each frame
retained mode: library compiles and displays from scenegraph that it maintains, a complex DAG. It is a display-centered extract of the Application Model
21/38
© 9/8/2016
Bin Sheng
Sample-based graphics vs Geometry-based graphics
Application Distinctions: Two Basic Paradigms
22/38
© 9/8/2016
Bin Sheng
Sample-based graphics: Discrete samples are used to describe visual information
pixels can be created by digitizing images, using a sample-based “painting” program, etc.
often some aspect of the physical world is sampled for visualization, e.g., temperature across the US
example programs: Adobe Photoshop™, GIMP™ , Adobe AfterEffects™
23/38
© 9/8/2016
Bin Sheng
Pixels are point locations with associated sample values, usually of light intensities/colors, transparency, and other control information
When we sample an image, we sample the point location along the continuous signal and we cannot treat the pixels as little circles or squares, though they may be displayed as such
Sample-based Graphics (2/3)
Visualization of a
mathematical pixel grid
LCD display CRT beam illumination pattern
Can’t visually resolve adjacent
pixels on CRT 1 pixel light intensity
24/38
© 9/8/2016
Bin Sheng
Samples created directly in Paint-type program, or by sampling of continuous (analog) visual materials (light intensity/color measured at regular intervals) with many devices including:
flatbed and drum scanners (e.g., https://luminous-landscape.com/drum-scans/)
digital still and motion (video) cameras
Sample values can also be input numerically (e.g., with numbers from computed dataset)
Once an image is defined as pixel-array, it can be manipulated
Image editing: changes made by user, such as cutting and pasting sections, brush-type tools, and processing selected areas
Image processing: algorithmic operations that are performed on image (or pre-selected portion of image) without user intervention. Blurring, sharpening, edge-detection, color balancing, rotating, warping. These are front-end processes to Computer Vision.
25/38
© 9/8/2016
Bin Sheng
Lets do some sampling of a building
A color value is measured at every grid point and used to color corresponding grid square
Crude sampling and image reconstruction method creates blocky image
Sampling an Image
3D scene
0 = white, 5 = gray, 10 = black
26/38
© 9/8/2016
Bin Sheng
Once image is defined in terms of colors at (x, y) locations on grid, can change image easily by altering location or color values
E.g., if we reverse our mapping above and make 10 = white and 0 = black, the image would look like this:
Pixel information from one image can be copied and pasted into another, replacing or combining with previously stored pixels
27/38
© 9/8/2016
Bin Sheng
WYSIAYG (What You See Is All You Get): No additional information
no depth information
can’t examine scene from different point of view
at most can play with the individual pixels or groups of pixels to change colors, enhance contrast, find edges, etc.
But increasingly great success in image-based rendering to fake 3D scenes and arbitrary camera positions. New images constructed by interpolation, composition, warping and other operations.
For a computational and cognitive science perspective, Take Thomas Serre’s Computational Vision
What’s the Disadvantage?
“Scene Reconstruction from High Spatio-Angular Resolution Light Fields” by Kim, Zimmer et al., 2013
28/38
© 9/8/2016
Bin Sheng
Geometry-based graphics (also called scalable vector graphics or object-oriented graphics):
geometrical model is created, along with various appearance attributes, and is then sampled for visualization (rendering, a.k.a image synthesis)
often some aspect of physical world is visually simulated, or “synthesized”
examples of 2D apps: Adobe Illustrator™ and Corel CorelDRAW™
examples of 3D apps: Autodesk’s AutoCAD™, Autodesk’s (formerly Alias|Wavefront’s) Maya™, Autodesk’s 3D Studio Max™
29/38
© 9/8/2016
Bin Sheng
Geometry-based graphics applications
Store mathematical descriptions, or “models,” of geometric elements (lines, polygons, polyhedrons, polygonal meshes…) and associated attributes (e.g., color, material
properties).
Geometric elements are primitive shapes, primitives for short.
Images are created via sampling of geometry for viewing, but not stored as part of model.
Users cannot usually work directly with individual pixels in geometry-based
programs; as user manipulates geometric elements, program resamples and redisplays elements
Increasingly rendering combines geometry- and sample-based graphics, both as performance hack and to increase quality of final product
CG animated characters (geometry) on painted or filmed scene images (samples)
Geometry-Based Graphics (2/2)
30/38
© 9/8/2016
Bin Sheng
What is a model?
Captures salient features (data, behavior) of object/phenomenon being modeled
data includes geometry, appearance, attributes…
note similarity to OOP ideas
Modeling allows us to cope with complexity
Our focus: modeling and viewing simple everyday objects
Consider this:
Through 3D computer graphics, we have abstract, easily changeable 3D forms, for the first time in human history
Has revolutionized working process of many fields – science, engineering, industrial design, architecture, commerce, entertainment, etc. Profound implications for visual thinking and visual literacy
“Visual truth” is gone in the Photoshop and FX-saturated world (but consider painting and photography…) – seeing no longer is believing…(or shouldn’t be!)
31/38
© 9/8/2016
Bin Sheng
Modeling
Create models
Apply materials to models
Place models around scene
Place lights in scene
Place the camera
Directional Light Ambient
Light
Point Light Spot
Light
lighting assignment by Patrick Doran
Rendering
Take “picture” with camera
Both can be done with commercial software:
Autodesk Maya
TM,3D Studio Max
TM, Blender
TM, etc.
32/38
© 9/8/2016
33/38
© 9/8/2016
Bin Sheng
Divide and Conquer
Hierarchy of geometrical components
Reduction to primitives (e.g., spheres, cubes, etc.)
Simple vs. not-so-simple elements (nail vs. screw)
Decomposition of a Geometric Model
Head Shaft
Point
composition decomposition
34/38
© 9/8/2016
Bin Sheng
Object to be modeled is (visually) analyzed, and then decomposed into collections of primitive shapes.
Tree diagram provides visual method of expressing “composed of” relationships of model
Such diagrams are part of 3D program interfaces (e.g., 3D Studio MAX, Maya)
As a data structure to be rendered, it is called a scenegraph Nail
Head (cylinder)
Body
root node
leaf nodes Shaft
(cylinder)
Point (cone)
tree diagram
35/38
© 9/8/2016
Bin Sheng
Primitives created in decomposition process must be assembled to create final object. Done with affine transformations, T, R, S (as in above example). Order matters – these are not commutative!
Composition of a Geometric Model
Primitives
in their own modeling coordinate system
Composition in world (root) coordinate system Translate
Scale and Translate
Rotate and Translate
36/38
© 9/8/2016
Bin Sheng
We manipulated primitive shapes with geometric transformations (translation, rotation, scale). These
transformations are essential for model organization, process of composing complex objects from simpler components.
Hierarchical models and geometric transformations are also essential for animation – create and edit scenegraphs
Once object’s geometry is established, must be viewedon screen: map from 3D geometry to 2D projections for viewing, and from 2D to 3D for 2D input devices (e.g., the mouse or pen/stylus, or touch)
While mapping from 3D to 2D, object (surface) material properties and lighting effects are used in rendering one’s constructions. This rendering process is also called image synthesis.
37/38
© 9/8/2016
Bin Sheng
Computer graphics involves both real-time / interactive applications and batch / offline applications
Both equally important, but different use cases
Photo-realism has really advanced
But it still takes dozens of hours on fastest computers to mimic physics of photons interacting with physical environments
Hardware evolution from vector to raster graphics
Vector graphics survives as Scalable Vector Graphics, which transforms without artifacts
Geometry-based vs. image-based graphics
Mathematical definition vs. pixel manipulation
Pixels are discrete samples of continuous functions
Causes artifacts (“jaggies”/ “aliases”) to appear (we will study fixes – “anti-aliasing”)
Geometric models typically constructed hierarchically
Scene graph data structure
38/38
In Summary
© 9/8/2016
