• Definition of inner product

32  Download (0)

Full text

(1)

Chapter 2: Inner Product Spaces

• Motivation

• Definition of inner product

• The spaces L

2

and l

2

• Schwarz and triangular inequalities

• Orthogonality

• Linear operators

• Least squares

(2)

1. Motivation

• For two vectors X=(x1,x2,x3), Y=(y1,y2,y3) in R3, the standard inner product of X and Y is defined as

• This definition is partly motivated by the desire to measure the length of a vector (Pythagorean theorem)

• If X is a unit vector, then is to measure the vector length of Y, i.e., the projection of Y on vector X.

3 3 2

2 1

, Y x

1

y x y x y

X = + +

X X x

x x

X ,

of

Length = 12 + 22 + 32 =

Y

X ,

(3)

• Ex.點(3,6,5)到點(3,8,7)的歐氏距離為:

• Ex.向量(3,6,5)長度為:

• Ex.

如果X不是單位向量(ex:(1,0,0))﹐則<X,Y>不 再是量測Y投影到X方向的長度﹐而是有放大或縮 小的效果

: ex: X=(2,0,0) or X=(0.5,0,0)

8 4

4 0

) 5 7

( )

6 8

( )

3 3

( −

2

+ −

2

+ −

2

= + + =

70 5

6

3

2

+

2

+

2

=

(4)

2 Definition of inner product

• For any dimension n, the two vectors

) ,

, ,

( x

1

x

2

x

n

X = L

and

Y = ( y

1

, y

2

, L , y

n

)

,

the Euclidean inner product is

=

= n

j

j j y x Y

X

1

,

• Complex form: Z and W are both complex vectors

=

= n

j

j jw z W

Z

1

,

(5)

•複數的向量內積使用共軛複數的原因

•If X=3+2i=(3,2i), then the length of X is:

2009/04/01 5

5 4

9 4

9 )

2 , 3 ( ) 2 , 3 (

, > = = + 2 = =

< X X i i i

Without conjugate (wrong):

With conjugate (correct):

13 4

9 4

9 )

2 , 3 ( ) 2 , 3 (

, > = = 2 = + =

< X X i i i

3 2i

(6)

3 The spaces L

2

and l

2

(1)

• Continuous form (連續型): L2

{

<

}

= b

a f t dt

C b

a f

b a

L2([ , ]) :[ , ] ; | ( ) |2

The energy of a continuous function f in the interval is defined as:

The L2 inner product in the interval [a, b] of two continuous functions is defined as:

dt t

g t f g

f

b

L

=

a

( ) ( )

,

2

(f and g can be complex.)

連續型內積定義

(7)

能量的定義

•能量的定義,就是將每一個element平方後相加,也 就是向量長度的平方。

•Ex. The energy of a vector (or a signal) X=(3,4,5) is defined by:

•Ex. The energy of a vector (or a signal) X=(x1,x2,x3,…,xn) is defined by:

50 5

4

3

2

+

2

+

2

=

2 2

3 2

2 2

,

X x1 x x xn

X

>= + + + +

< L

(8)

3 The spaces L

2

and l

2

(2)

• Discrete form (離散型): l2

=

=

b

a i

i

l

x

i

y

Y X ,

2

The l2 inner product in the interval [a, b] of two discrete functions is defined as:

(X and Y can be complex.)

離散型內積定義

(9)

4 Schwarz inequalities

Schwarz inequality:

||

||

||

||

| ,

| X YX Y

X

Y X+Y

Equality holds if and only if X and Y are linear dependent.

If X and Y are linear independent, what happens?

||

||

||

||

cos

||

||

||

||

| ,

| X Y = X Y θ X Y

θ 1.

cos ,

180 or

0

If θ = θ = ° θ =

. 0 cos

, 270 or

0 9

If θ = ° θ = ° θ =

線性相依

線性無關

(10)

Triangular inequalities

Triangle inequality:

||

||

||

||

||

|| X + YX + Y

X

Y X+Y

Equality holds if and only if X or Y is a positive multiple of the other.

0 , >

= tY t X

X Y=2X

||

||

||

||

||

|| X + Y = X + Y

(11)

5 Orthogonality 正交性 (1)

The vectors X and Y are said to be orthogonal if

0 , Y = X

The vectors X and Y are said to be orthonormal if

0 , Y = X

1

||

||

and

1

||

|| X = Y =

內積為0

長度為1 內積為0

(12)

5 Orthogonality (2)

Example:

]).

, ([

L in orthogonal

are cos

) ( and sin

) ( function The

2 −π π

=

= t g t t

t f

Proof:

0

| ) 2 4 cos(

1

) 2 2 sin(

1

cos sin

,

=

= −

=

=

ππ

π

π

π

π

t

dt t

tdt t

g f

內積為0,所以正交(orthogonal)

(13)

5 Orthogonality (3)

• Orthogonal projection

vectors of

collection l

orthonorma an

is } ,

, ,

{ev1 ev2 L evN

=

α

= N

j

j j

N

e v

e e

e v

1

2

1, ,..., }, then, {

of space in the

lies If

v v

v v

v v

where

α

j

= v v v , e

j

ev

j

vv

α j

Origin

).

,..., ,

( is coordinate its

and }

,..., ,

{

system coordinate

new a

in spanned be

can vector

: means That

2 1

2

1 e eN N

e

v

α α

v α v

v

v

(14)

5 Orthogonality (4)

Example:

z y

x v

z z

v

y y

v

x x

v

v

v v v

v

v v v

v v

v

v v

v

7 5

3

).

1 , 0 , 0 ( where

, 7

and );

0 , 1 , 0 ( where

, 5

);

0 , 0 , 1 ( where

, 3

means it

), 7 , 5 , 3 ( is

R in A vector

3

+ +

=

=

=

=

=

=

=

=

(15)

任何向量都可以投射到新的坐標空間

Example

3

x y

4 v=(3,4) x’

y’

5 /

) 1 , 2 ( '= x

5 /

) 2 , 1 ( '= −

y / 5 5/ 5 5

2 ) 1 4 , 3 (

4721 .

4 5 2 5 / 10 5

1 / ) 2 4 , 3 (

=

⎟⎟ =

⎜⎜

⎛−

=

=

⎟⎟ =

⎜⎜

? )

5 ( )

5 2 ( energy

25 16

9 4

3 energy

2 2

2 2

= +

=

= +

= +

= (x,y)-plane

(x’,y’)-plane

5 5 2

新的坐標空間

投射後能量守恆

) 5 , 5 2

= ( v

(16)

矩陣表示

Example

3 4

5 5 2

⎟⎟

⎜⎜

⎟⎟

⎜⎜

=

⎟⎟

⎜⎜

=

⎟⎟

⎜⎜

4 3 1 0

0 1 4

3

2

1 v

x x

T T v v

v

⎟⎟

⎜⎜

⎟⎟

⎜⎜

=

⎟⎟

⎜⎜

=

⎟⎟

⎜⎜

4 3 5 / 2 5 / 1

5 / 1 5

/ 2 '

' 5

5 2

2

1 v

x x

T T v v

v

) 5 , 5 2

= ( v

向量在投射後的坐標﹐可用矩陣表示法計算

新的坐標空間

v=(3,4)

本位座標空間

新的坐標空間

⎟⎟

⎜⎜

=

5 / 1

5 / ' 2

x1

⎟⎟

⎜⎜

= ⎛−

5 / 2

5 / ' 1

x2

x1

x2

新的坐標

(17)

2009/04/01 17

Example:

用途﹕在找出多數向量的主軸(主軸分析)﹐資料壓縮

axis.

new a

|| is in which ||

system, coordinate

new a

Construct .

coordinate Cartison

in vector a

be ) 7 , 5 , 3 ( Let

v z v v

= v

v v v

0 ,

so

), 7 , 5 , 3 83 ( 1

||

and ||

) 2 , 1 , 3 14 ( Define 1

3 1

3 1

=

=

=

= e

e

v e v

e v

v v

v v v

0694 .

8 ) 1 1 4 , , 9 12 (17

12 . , 17

4 9 0

7 5

3

0 2

3

. 0 ,

and 0

, :

satisfies which

) , , ( Define

2

2 3 2

1 2

=

=

=

= +

+

=

+ = = =

e

z x

z z y

y x

z y

x

e e e

e z

y x ev

+

+

83 0 0 ) (

: form matrix

a in or

83 0

0 : as d represente be

can

system, coordinate

new in the

Therefore,

3 2 1

3 2

1

e , e , e

e e

e v

v v v

v v

v v

建立新的坐標系統﹐使得Z軸與給定的向量平行

(18)

找出多數向量的主軸(主軸分析)

新的坐標空間

主軸

找出主軸後﹐次要的軸就變得不重要﹐可以 忽略﹐因此可以做:

1.資料壓縮 2.降低維度

(19)

2009/04/01 19

任何一個一維空間的資料﹐可視為超高維空間的一個點(或點向量)

(

f f f

)

n

f = , , , n length = :

data D

1 1 2 K

( )

⎟⎟

⎜⎜

+ +

⎟⎟

⎜⎜

+

⎟⎟

⎜⎜

=

1 0 0 0

1 0

0 0 1 ,

, , D,

In 1 2 1 2 M

K M

K M v

n

n f f f

f f

f f

n

n n

n n

e e f e

e f e

e f f

f

R e

e e

v v v

v K v v

v v v

v

L v v

v

, ,

,

: form new

a in d represente be

can then

} , , , { : system coordinate

another is

there If

1 2

2 1 1

1 1

2 1

+ +

+

=

f e

e e g

f T

T n

T T

v vM

v v

v v 2

1

) (

×

=

= ( , , , , , , )

2

1 f e f en

e f

g v v

v K v v

v = v

此一點向量﹐可以投射到任何一個新的(維 度相同的)座標系統,這是Fourier transform的 理論基礎。

(20)

6 Linear operators (1)

Definition:

.

V for

) ( )

( )

(

satisfies W which

V : T function a

is W space

vector a

and V

space vector

a between map)

(or operator

linear A

C a,b

u,v v

bT u

aT bv

au

T + = +

=

=

=

∑ ∑

n mn

m

n

i j ij j

j j j

x x

t t

t t

x t v

T

v x v

M L

M O

M v L

v

1

1

1 11

) (

: any vector

For matrix.

a as

d represente be

can T

then l,

dimensiona finite

are W and

V If

=

=

m

i ij j

j

t w

v

T ( )

1

and v v

(21)

6 Linear operators (2)

Example . angle an

for axis

- z respect to with

) 2 , 1 ( vector

a Rotate

θ v =

x z

⎟⎟ y

⎜⎜

⎟⎟

⎜⎜

θ θ

θ

= θ

2 1 cos

sin

sin ) cos

(v T

x y

(22)

6 Linear operators (3)

v=[1;2];

a=30/180*pi;

t=[cos(a) sin(a);-sin(a) cos(a)];

w=t*v;

figure(1); clf;

plot([0 v(1)],[0 v(2)],'r');

hold on;

plot([0 w(1)],[0 w(2)],'g');

axis equal;

Matlab program:

0 0.5 1 1.5 2

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

Result:

Red: Before rotation, Green: After rotation.

旋轉30度

(23)

6 Linear operators (4)

Adjoints: Definition

v ) ( ,

w ,

satisfies that

: operator linear

the is of

adjoint the

spaces, product

inner o

between tw operator

linear a

is :

If

* w T

v w

T(v)

V W

T T

W V

T

*

=

=

= n

j x jv j

v 1

Let v v ( )

=1

= m

i ij i

j a w

v

T v v

Q

w w

c

w x

a v

T x v

x T

v T

m

i

i i

m i

n

j ij j i

n

j j j

n

j j j

v v

v v

v v

=

=

=

=

=

∑ ∑

=

= =

=

=

1

1 1

1

1 ) ( ) ( )

( )

(

where =

n=

j ij j

i a x

c 1

=

=

= n

j

j ij

i b v

w T

v w

T

1

*

*( ) , then let ( )

If v v v v

既然可以轉過去,當然也可以轉回來

(24)

6 Linear operators (5)

v v

b c

v b c w

T c w

c T

w T

n j

m i

j ij i

m i

n j

j ij i m

i i i

m

i i i

v v

v v

v v

=

=

=

=

=

∑∑

∑∑

= =

= = =

=

1 1

1 1

1

* 1

*

*( ) ( ) ( )

=

= n

j x jv j

v 1

v

Known v j

m

i

ij

ib x

c =

=1

=

= n

j ij j

i a x

c 1

Known

∑∑

= =

= m

i

n j

j ij ij

j a b x

x

1 1

(25)

2009/04/01

Matrix representation:

( )

=

n n

x x v

v

vv v L v M1

1, ,

( ) w

c c w

w v

T

m m

M v L v

v

v =

=

1 1, ,

) (

( )

=

mj j m

j

a a w

w v

T v v L v M1

1, , )

(

( ) ( )

( )

=

=

=

n mn

m

n n

n n

n n

x x

a a

a a

w w

x x v

v T x

x v

v T v

T

M L

M O

M v L v L

v M v L

v M v L

v

1

1

1 11

1

1 1

1 1

, ,

) ,

, ( )

, , ( )

(

25

6 Linear operators (6)

Let and

C=AX

(26)

6 Linear operators (7)

Similarly

( )

=

m m

c c w

w

wv v L v M1

1, , ( )

=

ni i n

i

b b v

v w

T v v L v M1

1

*( ) , ,

(known) Let

( )

( )

=

=

=

n n

m nm

n

m n

x x v

v v

c c

b b

b b

v v

w T

v M v L

v

M L

M O

M v L v L

v

1 1

1

1

1 11

1

*

, , , , )

(

X=BC

X=BC=BAX BA=I

(27)

7 Least squares (1)

• Motivation:

– 1. we often use least squares to develop a

procedure for approximating signals (or functions).

– 2. we often use least squares to get model parameters in a fitting problem.

(28)

7 Least squares (2)

Best line fitting problem (overdetermined problem)

) , (xi yi

) ,

(xi mxi +b

Error at xi is | yi (mxi +b)|

The best line fitting is to find the minimum total square error:

=

+

= N

i

i

i mx b

y E

1

))2

(

( N>>2

(29)

7 Least squares (3)

=

+

= N

i

i

i mx b

y E

1

))2

( (

⎪⎪

=

+

=

=

+

=

=

= N

i

i i

i i

i N

i

b mx b y

E

x b

mx m y

E

1 1

0 ) 1 ))(

( (

2 0

0 ) ))(

( (

2 0

⎪⎪

=

=

=

= N

i

i i

N

i

i i

i i

b mx y

bx mx

x y

1 1

2

0 ) (

0 ) (

⎪⎪

+

=

+

=

∑ ∑ ∑

=

=

=

= = =

1 1

1

1 1 1

2

N

i N

i

i N

i i N

i

N

i

i N

i

i i

i

b mx

y

bx mx

x y

⎟⎟

⎜⎜

=

=

=

=

=

=

b m N

x

x x

y x y

N

i i

N

i i N

i i N

i i N

i

i i

1

1 1

2

1 1

Linear equations.

⎟⎟ =

⎜⎜

=

=

=

=

=

1 1 1

1

1 1

2

N

i i N

i

i i N

i i

N

i i N

i i

y x y N

x

x x

b m

(30)

7 Least squares (4)

Matlab program:

m=2.5; b=3.7;

x=rand(1,50)*50;

y=m*x+b+randn(1,50)*5;

sx=[min(x) max(x)];

sy=m*sx+b;

figure(1); clf; plot(x,y,'.'); hold on; plot(sx,sy,'r');

A=[sum(x.^2) sum(x);

sum(x) length(x)];

B=[sum(x.*y); sum(y)];

v=inv(A)*B ; m1=v(1)

b1=v(2)

y1=m1*sx+b1;

plot(sx,y1,'g');

(31)

7 Least squares (5)

Result

Red: Ground truth, Green: Estimation.

(32)

Homework

•計算f長度,

f = sin t, | f |=

ππsin2 t dt

Figure

Updating...

References

Related subjects :