• Definition of inner product

Chapter 2: Inner Product Spaces

• Motivation

• Definition of inner product

• The spaces L

²

and l

²

• Schwarz and triangular inequalities

• Orthogonality

• Linear operators

• Least squares

1. Motivation

• For two vectors X=(x₁,x₂,x₃), Y=(y₁,y₂,y₃) in R³, the standard inner product of X and Y is defined as

• This definition is partly motivated by the desire to measure the length of a vector (Pythagorean theorem)

• If X is a unit vector, then is to measure the vector length of Y, i.e., the projection of Y on vector X.

3 3 2

2 1

, Y x

1

y x y x y

X = + +

X X x

x x

X ,

of

Length = ₁² + ₂² + ₃² =

Y

X ,

• Ex.點(3,6,5)到點(3,8,7)的歐氏距離為:

• Ex.向量(3,6,5)長度為:

• Ex.

如果X不是單位向量(ex:(1,0,0))﹐則<X,Y>不 再是量測Y投影到X方向的長度﹐而是有放大或縮 小的效果

: ex: X=(2,0,0) or X=(0.5,0,0)

8 4

4 0

) 5 7

( )

6 8

( )

3 3

( −

²

+ −

²

+ −

²

= + + =

70 5

6

3

²

+

²

+

²

=

2 Definition of inner product

• For any dimension n, the two vectors

) ,

, ,

( x

₁

x

₂

x

_n

X = L

_and

Y = ( y

₁

, y

₂

, L , y

_n

)

^,

the Euclidean inner product is

∑

=

= ⁿ

j

j j y x Y

X

1

,

• Complex form: Z and W are both complex vectors

∑

=

= ⁿ

j

j jw z W

Z

1

,

•複數的向量內積使用共軛複數的原因

•If X=3+2i=(3,2i), then the length of X is:

2009/04/01 5

5 4

9 4

9 )

2 , 3 ( ) 2 , 3 (

, > = ⋅ = + ² = − =

< X X i i i

Without conjugate (wrong):

With conjugate (correct):

13 4

9 4

9 )

2 , 3 ( ) 2 , 3 (

, > = ⋅ − = − ² = + =

< X X i i i

3 2i

3 The spaces L

²

and l

²

(1)

• Continuous form (連續型): L²

{

^→

_∫

^{< ∞}

}

= ^b

a f t dt

C b

a f

b a

L²([ , ]) :[ , ] ; | ( ) |²

The energy of a continuous function f in the interval is defined as:

The L² inner product in the interval [a, b] of two continuous functions is defined as:

dt t

g t f g

f

^b

L

⁼ ∫

a

^{( ) ( )}

,

2

(f and g can be complex.)

連續型內積定義

能量的定義

•能量的定義，就是將每一個element平方後相加，也 就是向量長度的平方。

•Ex. The energy of a vector (or a signal) X=(3,4,5) is defined by:

•Ex. The energy of a vector (or a signal) X=(x1,x2,x3,…,xn) is defined by:

50 5

4

3

²

+

²

+

²

=

2 2

3 2

2 2

,

X x1 x x x_n

X

>= + + + +

< L

3 The spaces L

²

and l

²

(2)

• Discrete form (離散型): l²

∑

=

=

^b

a i

i

l

x

i

y

Y X ,

2

The l² inner product in the interval [a, b] of two discrete functions is defined as:

(X and Y can be complex.)

離散型內積定義

4 Schwarz inequalities

Schwarz inequality:

||

||

||

||

| ,

| X Y ≤ X Y

X

Y X+Y

Equality holds if and only if X and Y are linear dependent.

If X and Y are linear independent, what happens?

||

||

||

||

cos

||

||

||

||

| ,

| X Y = X Y θ ≤ X Y

θ 1.

cos ,

180 or

0

If θ = θ = ° θ =

. 0 cos

, 270 or

0 9

If θ = ° θ = ° θ =

線性相依

線性無關

Triangular inequalities

Triangle inequality:

||

||

||

||

||

|| X + Y ≤ X + Y

X

Y X+Y

Equality holds if and only if X or Y is a positive multiple of the other.

0 , >

= tY t X

X Y=2X

||

||

||

||

||

|| X + Y = X + Y

5 Orthogonality 正交性 (1)

The vectors X and Y are said to be orthogonal if

0 , Y = X

The vectors X and Y are said to be orthonormal if

0 , Y = X

1

||

||

and

1

||

|| X = Y =

內積為0

長度為1 內積為0

5 Orthogonality (2)

Example:

]).

, ([

L in orthogonal

are cos

) ( and sin

) ( function The

2 −π π

=

= t g t t

t f

Proof:

0

| ) 2 4 cos(

1

) 2 2 sin(

1

cos sin

,

=

= −

=

=

ππ

− π

π

− π

π

−

∫

∫

t

dt t

tdt t

g f

內積為0，所以正交(orthogonal)

5 Orthogonality (3)

• Orthogonal projection

vectors of

collection l

orthonorma an

is } ,

, ,

{ev₁ ev₂ L ev_N

∑

=

α

= ^N

j

j j

N

e v

e e

e v

1

2

1, ,..., }, then, {

of space in the

lies If

v v

v v

v v

where

α

_j

= v v v , e

_j

ev

j

vv

α j

Origin

).

,..., ,

( is coordinate its

and }

,..., ,

{

system coordinate

new a

in spanned be

can vector

: means That

2 1

2

1 e eN N

e

v

α α

v α v

v

v

5 Orthogonality (4)

Example:

z y

x v

z z

v

y y

v

x x

v

v

v v v

v

v v v

v v

v

v v

v

7 5

3

).

1 , 0 , 0 ( where

, 7

and );

0 , 1 , 0 ( where

, 5

);

0 , 0 , 1 ( where

, 3

means it

), 7 , 5 , 3 ( is

R in A vector

³

+ +

=

⇒

=

=

=

=

=

=

=

任何向量都可以投射到新的坐標空間

Example

3

x y

4 v=(3,4) x’

y’

5 /

) 1 , 2 ( '= x

5 /

) 2 , 1 ( '= −

y _{/ 5 5/ 5 5}

2 ) 1 4 , 3 (

4721 .

4 5 2 5 / 10 5

1 / ) 2 4 , 3 (

=

⎟⎟ =

⎠

⎜⎜ ⎞

⎝

⎛−

=

=

⎟⎟ =

⎠

⎜⎜ ⎞

⎝

⎛

? )

5 ( )

5 2 ( energy

25 16

9 4

3 energy

2 2

2 2

= +

=

= +

= +

= (x,y)-plane

(x’,y’)-plane

5 5 2

新的坐標空間

投射後能量守恆

) 5 , 5 2

= ( v

矩陣表示

Example

3 4

5 5 2

⎟⎟⎠

⎜⎜ ⎞

⎝

⎟⎟⎛

⎠

⎜⎜ ⎞

⎝

= ⎛

⎟⎟⎠

⎜⎜ ⎞

⎝

= ⎛

⎟⎟⎠

⎜⎜ ⎞

⎝

⎛

4 3 1 0

0 1 4

3

2

1 v

x x

T T v v

v

⎟⎟⎠

⎜⎜ ⎞

⎝

⎟⎟⎛

⎠

⎞

⎜⎜⎝

⎛

= −

⎟⎟⎠

⎜⎜ ⎞

⎝

= ⎛

⎟⎟⎠

⎞

⎜⎜⎝

⎛

4 3 5 / 2 5 / 1

5 / 1 5

/ 2 '

' 5

5 2

2

1 v

x x

T T v v

v

) 5 , 5 2

= ( v

向量在投射後的坐標﹐可用矩陣表示法計算

新的坐標空間

v=(3,4)

本位座標空間

新的坐標空間

⎟⎟⎠

⎜⎜ ⎞

⎝

= ⎛

5 / 1

5 / ' 2

x1

⎟⎟⎠

⎜⎜ ⎞

⎝

= ⎛−

5 / 2

5 / ' 1

x2

x1

x2

新的坐標

2009/04/01 17

Example:

用途﹕在找出多數向量的主軸(主軸分析)﹐資料壓縮

axis.

new a

|| is in which ||

system, coordinate

new a

Construct .

coordinate Cartison

in vector a

be ) 7 , 5 , 3 ( Let

v z v v

′

= v

v v v

0 ,

so

), 7 , 5 , 3 83 ( 1

||

and ||

) 2 , 1 , 3 14 ( Define 1

3 1

3 1

=

=

=

−

= e

e

v e v

e v

v v

v v v

0694 .

8 ) 1 1 4 , , 9 12 (17

12 . , 17

4 9 0

7 5

3

0 2

3

. 0 ,

and 0

, :

satisfies which

) , , ( Define

2

2 3 2

1 2

= −

⇒

− =

=

⎩ ⇒

⎨⎧

= +

+

=

−

+ = = =

e

z x

z z y

y x

z y

x

e e e

e z

y x ev

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛ +

+

83 0 0 ) (

: form matrix

a in or

83 0

0 : as d represente be

can

system, coordinate

new in the

Therefore,

3 2 1

3 2

1

e , e , e

e e

e v

v v v

v v

v v

建立新的坐標系統﹐使得Z軸與給定的向量平行

找出多數向量的主軸(主軸分析)

新的坐標空間

主軸

找出主軸後﹐次要的軸就變得不重要﹐可以 忽略﹐因此可以做:

1.資料壓縮 2.降低維度

2009/04/01 19

任何一個一維空間的資料﹐可視為超高維空間的一個點(或點向量)

(

^{f f f}

)

ⁿ

f = , , , _n length = :

data D

1 _{1 2} K

( )

⎟⎟

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜⎜

⎜

⎝

⎛ + +

⎟⎟

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜⎜

⎜

⎝

⎛ +

⎟⎟

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜⎜

⎜

⎝

⎛

⇔

=

−

1 0 0 0

1 0

0 0 1 ,

, , D,

In _{1 2 1 2} M

K M

K M v

n

n f f f

f f

f f

n

n n

n n

e e f e

e f e

e f f

f

R e

e e

v v v

v K v v

v v v

v

L v v

v

, ,

,

: form new

a in d represente be

can then

} , , , { : system coordinate

another is

there If

1 2

2 1 1

1 1

2 1

+ +

+

=

∈

f e

e e g

f T

T n

T T

v vM

v v

v v ₂

1

) (

⎟ ×

⎟⎟

⎟⎟

⎠

⎞

⎜⎜

⎜⎜

⎜

⎝

⎛

=

= _{( , , , , , , )}

2

1 f e f en

e f

g v v

v K v v

v = v

此一點向量﹐可以投射到任何一個新的(維 度相同的)座標系統，這是Fourier transform的 理論基礎。

6 Linear operators (1)

Definition:

.

V for

) ( )

( )

(

satisfies W which

V : T function a

is W space

vector a

and V

space vector

a between map)

(or operator

linear A

C a,b

u,v v

bT u

aT bv

au

T + = + →∈ ∈

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

=

=

=

∑ ∑

∑

n mn

m

n

i j ij j

j j j

x x

t t

t t

x t v

T

v x v

M L

M O

M v L

v

1

1

1 11

) (

: any vector

For matrix.

a as

d represente be

can T

then l,

dimensiona finite

are W and

V If

∑

=

=

^m

i ij j

j

t w

v

T ( )

1

and v v

6 Linear operators (2)

Example . angle an

for axis

- z respect to with

) 2 , 1 ( vector

a Rotate

θ v =

x z

⎟⎟ y

⎠

⎜⎜ ⎞

⎝

⎟⎟⎛

⎠

⎜⎜ ⎞

⎝

⎛

θ θ

−

θ

= θ

2 1 cos

sin

sin ) cos

(v T

x y

6 Linear operators (3)

v=[1;2];

a=30/180*pi;

t=[cos(a) sin(a);-sin(a) cos(a)];

w=t*v;

figure(1); clf;

plot([0 v(1)],[0 v(2)],'r');

hold on;

plot([0 w(1)],[0 w(2)],'g');

axis equal;

Matlab program:

0 0.5 1 1.5 2

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

Result:

Red: Before rotation, Green: After rotation.

旋轉30度

6 Linear operators (4)

Adjoints: Definition

v ) ( ,

w ,

satisfies that

: operator linear

the is of

adjoint the

spaces, product

inner o

between tw operator

linear a

is :

If

* w T

v w

T(v)

V W

T T

W V

T

*

= →

→

∑

=

= ⁿ

j x jv j

v 1

Let v v ( )

∑

=1

= ^m

i ij i

j a w

v

T v v

Q

w w

c

w x

a v

T x v

x T

v T

m

i

i i

m i

n

j ij j i

n

j j j

n

j j j

v v

v v

v v

=

=

=

=

=

∴

∑

∑ ∑

∑

∑

=

= =

=

=

1

1 1

1

1 ) ( ) ( )

( )

(

where =

∑

ⁿ=

j ij j

i a x

c 1

∑

=

=

= ⁿ

j

j ij

i b v

w T

v w

T

1

*

*( ) , then let ( )

If v v v v

既然可以轉過去，當然也可以轉回來

6 Linear operators (5)

v v

b c

v b c w

T c w

c T

w T

n j

m i

j ij i

m i

n j

j ij i m

i i i

m

i i i

v v

v v

v v

=

=

=

=

=

∑∑

∑∑

∑

∑

= =

= = =

=

1 1

1 1

1

* 1

*

*( ) ( ) ( )

∑

₌

= ⁿ

j x jv j

v 1

v

Known v _j

m

i

ij

ib x

c =

∑

=1

∑

=

= ⁿ

j ij j

i a x

c 1

Known

∑∑

= =

= ^m

i

n j

j ij ij

j a b x

x

1 1

2009/04/01

Matrix representation:

( )

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

=

n n

x x v

v

vv v L v M¹

1, ,

( ) ^w

c c w

w v

T

m m

M v L v

v

v =

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

=

1 1, ,

) (

( )

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

=

mj j m

j

a a w

w v

T v v L v M¹

1, , )

(

( ) ( )

( )

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

=

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

⎟ =

⎟⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

=

n mn

m

n n

n n

n n

x x

a a

a a

w w

x x v

v T x

x v

v T v

T

M L

M O

M v L v L

v M v L

v M v L

v

1

1

1 11

1

1 1

1 1

, ,

) ,

, ( )

, , ( )

(

25

6 Linear operators (6)

Let and

C=AX

6 Linear operators (7)

Similarly

( )

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

=

m m

c c w

w

wv v L v M¹

1, , ( )

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

=

ni i n

i

b b v

v w

T v v L v M¹

1

*( ) , ,

(known) Let

( )

( )

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

=

=

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

⎟⎟

⎟

⎠

⎞

⎜⎜

⎜

⎝

⎛

=

n n

m nm

n

m n

x x v

v v

c c

b b

b b

v v

w T

v M v L

v

M L

M O

M v L v L

v

1 1

1

1

1 11

1

*

, , , , )

(

X=BC

X=BC=BAX BA=I

7 Least squares (1)

• Motivation:

– 1. we often use least squares to develop a

procedure for approximating signals (or functions).

– 2. we often use least squares to get model parameters in a fitting problem.

7 Least squares (2)

Best line fitting problem (overdetermined problem)

) , (x_i y_i

) ,

(x_i mx_i +b

Error at xi is ^{| y}i −^(mxi +^b)|

The best line fitting is to find the minimum total square error:

∑

=

+

−

= ^N

i

i

i mx b

y E

1

))2

(

( N>>2

7 Least squares (3)

∑

=

+

−

= ^N

i

i

i mx b

y E

1

))2

( (

⎪⎪

⎩

⎪⎪⎨

⎧

=

− +

−

→

∂ =

∂

=

− +

−

→

∂ =

∂

∑

∑

=

= N

i

i i

i i

i N

i

b mx b y

E

x b

mx m y

E

1 1

0 ) 1 ))(

( (

2 0

0 ) ))(

( (

2 0

⎪⎪

⎩

⎪⎪⎨

⎧

=

−

−

=

−

−

∑

∑

=

= N

i

i i

N

i

i i

i i

b mx y

bx mx

x y

1 1

2

0 ) (

0 ) (

⎪⎪

⎩

⎪⎪⎨

⎧

+

=

+

=

∑

∑

∑

∑ ∑ ∑

=

=

=

= = =

1 1

1

1 1 1

2

N

i N

i

i N

i i N

i

N

i

i N

i

i i

i

b mx

y

bx mx

x y

⎟⎟⎠

⎜⎜ ⎞

⎝

⎛

⎟⎟

⎟⎟

⎠

⎞

⎜⎜

⎜⎜

⎝

⎛

=

⎟⎟

⎟⎟

⎠

⎞

⎜⎜

⎜⎜

⎝

⎛

∑

∑

∑

∑

∑

=

=

=

=

=

b m N

x

x x

y x y

N

i i

N

i i N

i i N

i i N

i

i i

1

1 1

2

1 1

Linear equations.

⎟⎟

⎟⎟

⎠

⎞

⎜⎜

⎜⎜

⎝

⎛

⎟⎟

⎟⎟

⎠

⎞

⎜⎜

⎜⎜

⎝

⎛

⎟⎟ =

⎠

⎜⎜ ⎞

⎝

⎛

∑

∑

∑

∑

∑

=

=

−

=

=

=

1 1 1

1

1 1

2

N

i i N

i

i i N

i i

N

i i N

i i

y x y N

x

x x

b m

7 Least squares (4)

Matlab program:

m=2.5; b=3.7;

x=rand(1,50)*50;

y=m*x+b+randn(1,50)*5;

sx=[min(x) max(x)];

sy=m*sx+b;

figure(1); clf; plot(x,y,'.'); hold on; plot(sx,sy,'r');

A=[sum(x.^2) sum(x);

sum(x) length(x)];

B=[sum(x.*y); sum(y)];

v=inv(A)*B ; m1=v(1)

b1=v(2)

y1=m1*sx+b1;

plot(sx,y1,'g');

7 Least squares (5)

Result

Red: Ground truth, Green: Estimation.

Homework

•計算f長度,

^{f = sin t, | f |=}

∫

₋^π_π^{sin2 t dt}