+1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

0/1 error penalizes both types

equally

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 14/25

Noise and Error Algorithmic Error Measure

Choice of Error Measure

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

0/1 error penalizes both types

equally

Noise and Error Algorithmic Error Measure

Choice of Error Measure

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

0/1 error penalizes both types

equally

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 14/25

Noise and Error Algorithmic Error Measure

Choice of Error Measure

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

0/1 error penalizes both types

equally

Noise and Error Algorithmic Error Measure

Fingerprint Verification for Supermarket

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

g +1 -1

f +1

0 10

-1

1 0

•

supermarket: fingerprint for discount

• false reject: very unhappy customer, lose future business

• false accept: give away a minor discount, intruder left fingerprint :-)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 15/25

Noise and Error Algorithmic Error Measure

Fingerprint Verification for Supermarket

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

g +1 -1

f +1

0 10

-1

1 0

•

supermarket: fingerprint for discount

• false reject: very unhappy customer, lose future business

• false accept: give away a minor discount, intruder left fingerprint :-)

Noise and Error Algorithmic Error Measure

Fingerprint Verification for Supermarket

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

g +1 -1

f +1

0 10

-1

1 0

•

supermarket: fingerprint for discount

• false reject: very unhappy customer, lose future business

• false accept: give away a minor discount, intruder left fingerprint :-)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 15/25

Noise and Error Algorithmic Error Measure

Fingerprint Verification for Supermarket

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

g +1 -1

f +1

0 10

-1

1 0

•

supermarket: fingerprint for discount

• false reject: very unhappy customer, lose future business

• false accept: give away a minor discount, intruder left fingerprint :-)

Noise and Error Algorithmic Error Measure

Fingerprint Verification for CIA

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

g

+1 -1

f +1

0 1

-1

1000 0

•

CIA: fingerprint for entrance

• false accept: very serious consequences!

• false reject: unhappy employee, but so what? :-)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 16/25

Noise and Error Algorithmic Error Measure

Fingerprint Verification for CIA

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

g

+1 -1

f +1

0 1

-1

1000 0

•

CIA: fingerprint for entrance

• false accept: very serious consequences!

• false reject: unhappy employee, but so what? :-)

Noise and Error Algorithmic Error Measure

Fingerprint Verification for CIA

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

g

+1 -1

f +1

0 1

-1

1000 0

•

CIA: fingerprint for entrance

• false accept: very serious consequences!

• false reject: unhappy employee, but so what? :-)

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 16/25

Noise and Error Algorithmic Error Measure

Fingerprint Verification for CIA

Fingerprint Verification

f

 



  +1 you

−1 intruder

two types of error:

false accept

and

false reject

g

+1 -1

f +1

no error false reject

-1

false accept no error

g

+1 -1

f +1

0 1

-1

1000 0

•

CIA: fingerprint for entrance

• false accept: very serious consequences!

• false reject: unhappy employee, but so what? :-)

Noise and Error Algorithmic Error Measure

Take-home Message for Now

err

is

application/user-dependent

Algorithmic Error Measures err c

•

true: just

err

•

plausible:

• 0/1: minimum ‘flipping noise’—NP-hard to optimize, remember? :-)

• squared: minimum Gaussian noise

•

friendly: easy to optimize forA

• closed-form solution

• convex objective function

c

err: more in next lectures

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 17/25

Noise and Error Algorithmic Error Measure

Take-home Message for Now

err

is

application/user-dependent Algorithmic Error Measures err c

•

true: just

err

•

plausible:

• 0/1: minimum ‘flipping noise’—NP-hard to optimize, remember? :-)

• squared: minimum Gaussian noise

•

friendly: easy to optimize forA

• closed-form solution

• convex objective function

c

err: more in next lectures

Noise and Error Algorithmic Error Measure

Take-home Message for Now

err

is

application/user-dependent Algorithmic Error Measures err c

•

true: just

err

•

plausible:

• 0/1: minimum ‘flipping noise’—NP-hard to optimize, remember? :-)

• squared: minimum Gaussian noise

•

friendly: easy to optimize forA

• closed-form solution

• convex objective function

c

err: more in next lectures

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 17/25

Noise and Error Algorithmic Error Measure

Take-home Message for Now

err

is

application/user-dependent Algorithmic Error Measures err c

•

true: just

err

•

plausible:

• 0/1: minimum ‘flipping noise’—NP-hard to optimize, remember? :-)

• squared: minimum Gaussian noise

•

friendly: easy to optimize forA

• closed-form solution

• convex objective function

c

err: more in next lectures

Noise and Error Algorithmic Error Measure

Take-home Message for Now

err

is

application/user-dependent Algorithmic Error Measures err c

•

true: just

err

•

plausible:

• 0/1: minimum ‘flipping noise’—NP-hard to optimize, remember? :-)

• squared: minimum Gaussian noise

•

friendly: easy to optimize forA

• closed-form solution

• convex objective function

c

err: more in next lectures

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 17/25

Noise and Error Algorithmic Error Measure

Take-home Message for Now

err

is

application/user-dependent Algorithmic Error Measures err c

•

true: just

err

•

plausible:

• 0/1: minimum ‘flipping noise’—NP-hard to optimize, remember? :-)

• squared: minimum Gaussian noise

•

friendly: easy to optimize forA

• closed-form solution

• convex objective function

c

err: more in next lectures

Noise and Error Algorithmic Error Measure

Learning Flow with Algorithmic Error Measure

unknown target distribution P(y |x) containing f (x) + noise (ideal credit approval formula)

training examples D : (x

1

, y

1

), · · · , (x

N

,y

N

) (historical records in bank)

learning algorithm

A

final hypothesis g ≈ f

(‘learned’ formula to be used)

hypothesis set H

(set of candidate formula)

unknown P on X

x

₁

, x

₂

, · · · , x

_N

x y

1

,y

2

, · · · , y

N

y

error measure err c err

err: application goal;

c

err: a key part of many

A

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 18/25

Noise and Error Algorithmic Error Measure

Fun Time

Consider err below for CIA. What is E in (g) when using this err?

g +1 -1

f +1 0 1

-1 1000 0

1 1

N

P

N n=1

Jy

n

6= g(x

n

)K

2 1

N

P

y

n

=+1

Jy

ⁿ

6= g(x

ⁿ

)K + 1000 P

y

n

=−1

Jy

ⁿ

6= g(x

ⁿ

)K

!

3 1

N

P

y

n

=+1

Jy

n

6= g(x

n

)K − 1000 P

y

n

=−1

Jy

n

6= g(x

n

)K

!

4 1

N

1000 P

y

n

=+1

Jy

n

6= g(x

n

)K + P

y

n

=−1

Jy

n

6= g(x

n

)K

!

Reference Answer: 2

When y

_n

=−1, the

false positive

made on such (x

_n

,y

_n

)is penalized

1000

times more!

Noise and Error Algorithmic Error Measure

Fun Time

Consider err below for CIA. What is E in (g) when using this err?

g +1 -1

f +1 0 1

-1 1000 0

1 1

N

P

N n=1

Jy

n

6= g(x

n

)K

2 1

N

P

y

n

=+1

Jy

ⁿ

6= g(x

ⁿ

)K + 1000 P

y

n

=−1

Jy

ⁿ

6= g(x

ⁿ

)K

!

3 1

N

P

y

n

=+1

Jy

n

6= g(x

n

)K − 1000 P

y

n

=−1

Jy

n

6= g(x

n

)K

!

4 1

N

1000 P

y

n

=+1

Jy

n

6= g(x

n

)K + P

y

n

=−1

Jy

n

6= g(x

n

)K

!

Reference Answer: 2

When y

_n

=−1, the

false positive

made on such (x

_n

,y

_n

)is penalized

1000

times more!

Hsuan-Tien Lin (NTU CSIE) Machine Learning Foundations 19/25

Noise and Error Weighted Classification

Weighted Classification

CIA Cost (Error, Loss, . . .) Matrix

h(x) +1 -1

y +1 0 1

-1 1000 0

out-of-sample

E

_out

(h) = E

(x,y )∼P



1

if y = +1

1000

if y =−1



·

Jy 6= h(x)K

in-sample

E

_in

(h) = 1 N

X

N

n=1



1

if y

_n

= +1

1000

if y

n

=−1



·

Jy n 6= h(x n ) K

weighted classification:

在文檔中 Machine Learning Foundations (ᘤ9M) (頁 66-87)