Machine Learning for Modern Artiﬁcial Intelligence

(1)

Machine Learning for Modern Artificial Intelligence

Hsuan-Tien Lin 林軒田

Dept. of Computer Science and Information Enginnering, National Taiwan University

國立臺灣大學資訊工程學系 January 25, 2022 AI & Data Science Workshop

(2)

About Me

Professor

National Taiwan University

Chief Data Science Consultant (former Chief Data Scientist)

Appier Inc.

Co-author Learning from Data

Instructor

NTU-Coursera MOOCs ML Foundations/Techniques

(3)

ML for (Modern) AI

Outline

ML for (Modern) AI

ML Research for Modern AI

ML for AI in Reality

(4)

ML for (Modern) AI

From Intelligence to Artificial Intelligence

intelligence: thinking and actingsmartly

• humanly

• rationally

artificialintelligence:computersthinking and actingsmartly

• humanly

• rationally

humanly≈smartly≈rationally

—are humans rational?

(5)

ML for (Modern) AI

Humanly versus Rationally

What if your self-driving car decides one death is better than two—and that one is you? (The Washington Post http://wpo.st/ZK-51)

You’re humming along in your self-driving car, chatting on your iPhone 37 while the machine navigates on its own. Then a swarm of people appears in the street, right in the path of the oncoming vehicle.

Car ActingHumanly tosave my (and passengers’) life, stay on track

Car ActingRationally avoid the crowd and crash the owner forminimum total loss

which issmarter?

—depending on where I am, maybe?

(6)

ML for (Modern) AI

(Traditional) Artificial Intelligence

Thinking Humanly

• cognitive modeling

—now closer to Psychology than AI

Acting Humanly

• dialog systems

• humanoid robots

• computer vision

Thinking Rationally

• formal logic—now closer to Theoreticians than AI practitioners

Acting Rationally

• recommendation systems

• cleaning robots

• character recognition

actinghumanly or rationally:

more academia/industry attention nowadays

(7)

ML for (Modern) AI

Traditional vs. Modern [My] Definition of AI

Traditional Definition

humanly ≈ intelligently ≈ rationally

My Definition

intelligently ≈ easily is your smart phone ‘smart’?

modern artificial intelligence

=applicationintelligence

(8)

ML for (Modern) AI

Examples of Application Intelligence

Siri

By Bernard Goldbach [CC BY 2.0]

Amazon Recommendations

By Kelly Sims [CC BY 2.0]

iRobot

By Yuan-Chou Lo [CC BY-NC-ND 2.0]

Vivino

From nordic.businessinsider.com

applicationintelligence is everywhere!

(9)

ML for (Modern) AI

AI Milestones

logic inference

expert system

machine learning +deep learning

begin 1st winter 2nd winter revolution

1956 1980 1993 2012 time

heat

AI history

• first AI winter: AI cannot solve ‘combinatorial explosion’ problems

• second AI winter: expert system failed to scale

reason of winters: expectation mismatch

(10)

ML for (Modern) AI

AI: Now and Next

2010–2015: AI | AI becomes promising, e.g.

• initial success of deep learningon ImageNet

• mature tools for SVM (LIBSVM) and others

2016–2020: AI + AI becomes competitive, e.g.

• super-human performance of alphaGoand others

• all big technology companies becomeAI-first

2021–: AI × AI becomes necessary

• “You’ll not be replaced by AI, butby humans who know how to use AI”

(Sun, Chief AI Scientist of Appier, 2018)

what isdiffereentnow?

(11)

ML for (Modern) AI

What is Different Now?

More Data

• cheaper storage

• Internet companies

Faster Computation

• cloud computing

• GPU computing

Better Algorithms

• decades of research

• e.g. deep learning

Healthier Mindset

• reasonable wishes

• key breakthroughs

data-enabledAI: mainstream nowadays

(12)

ML for (Modern) AI

Bigger Data Enable Easier-to-use AI

By deepanker70 on https://pixabay.com/

past

best route by shortest path

present

best route by current traffic

future

best route by predicted travel time

big datacanmake machine look smarter

(13)

ML for (Modern) AI

Machine Learning Connects Big Data and AI

From Big Data to Artificial Intelligence

big data

ML

artificial intelligence

ingredient tools/steps dish

Photos Licensed under CC BY 2.0 from Andrea Goh on Flickr

many possibilities when using the right tools

(14)

ML for (Modern) AI

ML-based AI Applications (1/4): Medicine

data

ML

^AI

By DataBase Center for Life Science;

licensed under CC BY 4.0 via Wikimedia Commons

for computer-assisted diagnosis

• data:

• patient status

• past diagnosis from doctors

• AI: dialogue system thatefficiently identifies disease of patient my student’s earlier work

as intern @ HTC DeepQ

(15)

ML for (Modern) AI

ML-based AI Applications (2/4): Communication

data

ML

^AI

By JulianVilla26;

licensed under CC BY-SA 4.0 via Wikimedia Commons

for 4G LTE communication

• data:

• channel information(the channel matrix representing mutual information)

• configuration(precoding, modulation, etc.) that reaches the highest throughput

• AI: predictbest configuration to the base stationin a new environment

my student’s earlier work as intern @ MTK

(16)

ML for (Modern) AI

ML-based AI Applications (3/4): Entertainment

data

ML

^AI

Match movie and viewer factors

predicted rating

com edy

content action

cont ent blockbu ster?

Tom Cru

isein it?

likesTomCruise? prefersblockbusters?

likesaction?

likescomedy?

movie viewer

add contributions from each factor

• data: how many users have rated some movies

• AI: predict how a user would rate an unrated movie

world-championfrom

National Taiwan Univ. in KDDCup 2011

(17)

ML for (Modern) AI

ML-based AI Applications (4/4): Security

data

ML

^AI

original picture by F.U.S.I.A. assistant and derivative work by Sylenius via Wikimedia Commons

face recognition

• data:faces and non-faces

• AI: predictwhich boxes contain faces

matureML technique, but often needtuning for differentapplication intelligenceneeds

(18)

ML for (Modern) AI

Good AI Needs Both ML and Non-ML Techniques

(Public Domain, from Wikipedia; used here for education purpose; all other rights still belong to Google DeepMind)

Non-ML Techniques

Monte C. Tree Search

≈move simulationin brain

(CC-BY-SA 3.0 by Stannered on Wikipedia)

ML Techniques

Deep Learning

≈board analysisin human brain

(CC-BY-SA 2.0 by Frej Bjon on Wikipedia)

Reinforcement Learn.

≈(self)-practicein human training

(Public Domain, from Wikipedia)

good AI: important to use theright techniques—ML& others, including human

(19)

ML for (Modern) AI

Full Picture of ML for Modern AI

big data

ML AI

human learning/

analysis

domain knowledge

(HumanI)

method

model expert system

industry: black plum is as sweet as white

(20)

ML for (Modern) AI

Example: Tropical Cyclone Intensity Estimation

meteorologists can ‘feel’ & estimate TC intensity from image

TC images

ML

_estimation^intensity

human learning/

analysis

domain knowledge

(HumanI)

CNN polar

rotation invariance

current weather

system

better than current system

(Chen et al., KDD ’18; Chen et al., Weather & Forecasting ’19)

(21)

Outline

ML for (Modern) AI

(22)

Cost-Sensitive Multiclass Classification

(23)

What is the Status of the Patient?

?

By DataBase Center for Life Science;

licensed under CC BY 4.0 via Wikimedia Commons

COVID19 cold healthy

Pictures Licensed under CC BY-SA 3.0 from 1RadicalOne on Wikimedia Commons

• aclassificationproblem

—grouping ‘patients’ into different ‘status’

are all mis-prediction costs equal?

(24)

Patient Status Prediction

error measure = society cost

XXXX

XXXXXX actual

predicted

COVID19 cold healthy

COVID19 0 1000 100000

cold 100 0 3000

healthy 100 30 0

• COVID19 mis-predicted as healthy: very high cost

• cold mis-predicted as healthy: high cost

• cold correctly predicted as cold: no cost

human doctors consider costs of decision;

how about computer-aided diagnosis?

(25)

Our Works

binary multiclass

regular well-studied well-studied

cost-sensitive known(Zadrozny et al., 2003) ongoing(our works, among others)

selected works of ours

• cost-sensitive SVM(Tu and Lin, ICML 2010)

• cost-sensitive one-versus-one(Lin, ACML 2014)

• cost-sensitive deep learning(Chung et al., IJCAI 2016)

why are peoplenot

using thosecool ML works for their AI?

(26)

Issue 1: Where Do Costs Come From?

A Real Medical Application: Classifying Bacteria

• by human doctors: different treatments⇐⇒ serious costs

• cost matrix averaged from two doctors:

Ab Ecoli HI KP LM Nm Psa Spn Sa GBS

Ab 0 1 10 7 9 9 5 8 9 1

Ecoli 3 0 10 8 10 10 5 10 10 2

HI 10 10 0 3 2 2 10 1 2 10

KP 7 7 3 0 4 4 6 3 3 8

LM 8 8 2 4 0 5 8 2 1 8

Nm 3 10 9 8 6 0 8 3 6 7

Psa 7 8 10 9 9 7 0 8 9 5

Spn 6 10 7 7 4 4 9 0 4 7

Sa 7 10 6 5 1 3 9 2 0 7

GBS 2 5 10 9 8 6 5 6 8 0

issue 2: is cost-sensitive classification really useful?

(27)

Cost-Sensitive vs. Traditional on Bacteria Data

. . . . . .

Are cost-sensitive algorithms great?

RBF kernel

0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6

OVOSVM

csOSRSVM csOVOSVM csFTSVM

algorithms

cost

.

...Cost-sensitive algorithms perform better than regular algorithm

Jan et al. (Academic Sinica) Cost-Sensitive Classiﬁcation on SERS October 31, 2011 15 / 19

(Jan et al., BIBM 2011)

cost-sensitivebetter thantraditional;

but why are peoplestill not using those cool ML works for their AI?

(28)

Issue 3: Error Rate of Cost-Sensitive Classifiers

The Problem

0.1 0.15 0.2 0.25 0.3

0 0.05 0.1 0.15 0.2

Error (%)

Cost

• cost-sensitive classifier: low cost but high error rate

• traditional classifier: low error rate but high cost

• how can we get theblueclassifiers?: low error rate and low cost

cost-and-error-sensitive:

more suitable forreal-world medical needs

(29)

Improved Classifier for Both Cost and Error

(Jan et al., KDD 2012)

Cost

iris ≈

wine ≈

glass ≈

vehicle ≈

vowel

segment

dna

satimage ≈

usps

zoo

splice ≈

ecoli ≈

soybean ≈

Error

iris

wine

glass

vehicle

vowel

segment

dna

satimage

usps

zoo

splice

ecoli

soybean

now,are people using those cool ML works for their AI?

(30)

Lessons Learned from

Research on Cost-Sensitive Multiclass Classification

? COVID19 cold healthy

See Page 16 of the Slides for Sources of the Pictures

1 more realistic (generic) in academia

6=more realistic (feasible) in application e.g. the ‘cost’ ofinputting a cost matrix?

2 cross-domain collaborationimportant

e.g. getting the ‘cost matrix’ fromdomain experts

3 not easy to winhuman trust

—humans are somewhatmulti-objective

(31)

Label Space Coding for

Multilabel Classification

(32)

What Tags?

?: {machine learning,data structure,data mining,object oriented programming,artificial intelligence,compiler, architecture,chemistry,textbook,children book,. . .etc. }

amultilabel classification problem:

tagginginput to multiple categories

(33)

Binary Relevance: Multilabel Classification via Yes/No

Binary

Classification {yes,no}

multilabel w/ L classes: LY/Nquestions machine learning(Y), data structure(N), data

mining(Y), OOP(N), AI(Y), compiler(N), architecture(N), chemistry(N), textbook(Y),

children book(N),etc.

• Binary Relevance approach:

transformation tomultiple isolated binary classification

• disadvantages:

• isolation—hidden relations not exploited (e.g. ML and DMhighly correlated, MLsubset ofAI, textbook & children bookdisjoint)

• unbalanced—fewyes, manyno

Binary Relevance: simple (& good) benchmark with known disadvantages

(34)

From Label-set to Coding View

label set apple orange strawberry binary code

{o} 0 (N) 1 (Y) 0 (N) [0, 1, 0]

{a, o} 1 (Y) 1 (Y) 0 (N) [1, 1, 0]

{a, s} 1 (Y) 0 (N) 1 (Y) [1, 0, 1]

{o} 0 (N) 1 (Y) 0 (N) [0, 1, 0]

{} 0 (N) 0 (N) 0 (N) [0, 0, 0]

subset of 2{1,2,··· ,L}⇔ length-L binary code

(35)

A NeurIPS 2009 Approach: Compressive Sensing

General Compressive Sensing

sparse (many0) binary vectorsy ∈ {0, 1}^Lcan berobustly

compressed by projecting to M L basis vectors {p₁,p₂, · · · ,p_M}

Comp. Sensing for Multilabel Classification(Hsu et al., NeurIPS 2009) 1 compress: encode original data bycompressive sensing

2 learn: getregressionfunction from compressed data

3 decode: decode regression predictions to sparse vector by compressive sensing

Compressive Sensing:

seemly strong competitor from related theoretical analysis

(36)

Our Proposed Approach:

Compressive Sensing ⇒ PCA

Principal Label Space Transformation (PLST),

i.e. PCA for Multilabel Classification (Tai and Lin, NC Journal 2012) 1 compress: encode original data byPCA

2 learn: getregressionfunction from compressed data

3 decode: decode regression predictions to label vector byreverse PCA + quantization

does PLST perform better than CS?

(37)

Hamming Loss Comparison: PLST vs. CS

0 20 40 60 80 100

0.03 0.035 0.04 0.045 0.05

Full−BR (no reduction) CS

PLST

mediamill (Linear Regression)

0 20 40 60 80 100

0.03 0.035 0.04 0.045 0.05

Full−BR (no reduction) CS

PLST

mediamill (Decision Tree)

• PLSTbetter thanCS: faster,better performance

• similar findings acrossdata sets and regression algorithms

Why? CScreates

harder-to-learnregression tasks

(38)

Our Works Continued from PLST

1 CompressionCoding(Tai & Lin, NC Journal 2012 with 342 citations)

—condense for efficiency: better (than CS) approach PLST

— key tool: PCA from Statistics/Signal Processing

2 Learnable-CompressionCoding(Chen & Lin, NeuIPS 2012 with 262 citations)

—condense learnably forbetterefficiency: better (than PLST) approach CPLST

— key tool: Ridge Regression from Statistics (+ PCA)

3 Cost-SensitiveCoding(Huang & Lin, ECML Journal Track 2017 with 48 citations)

—condense cost-sensitively towards application needs: better (than CPLST) approach CLEMS

— key tool: Multidimensional Scaling from Statistics

cannot thankstatisticans enough for those tools!

(39)

Lessons Learned from

Label Space Coding for Multilabel Classification

?: {machine learning,data structure,data mining,object oriented programming,artificial intelligence,compiler,architecture,chemistry,

textbook,children book,. . .etc. }

1 Is Statistics the same as ML? Is Statistics the same as AI?

• does it really matter?

• modern AI should embraceevery useful tool from other fields

• all fields could find theirconcrete rolesin AI

2 good toolsnot necessarily most sophisticated tools e.g. PCA possibly more useful than CS

3 more-cited paper 6= more-useful AI solution

—citation countnot the only impact measure

(40)

Active Learning by Learning

(41)

Active Learning: Learning by ‘Asking’

labeling isexpensive:

active learning ‘question asking’

—query y_nofchosenx_n

unknown target function f : X → Y

labeled training examples ( , +1), ( , +1), ( , +1)

( , -1), ( , -1), ( , -1)

learning algorithm

A

final hypothesis g≈f

+1

active: improve hypothesis with fewer labels (hopefully) by asking questionsstrategically

(42)

Pool-Based Active Learning Problem

Given

• labeled poolD_l =n

(featurex_n ,label y_n(e.g. IsApple?))oN n=1

• unlabeled pool D_u= nx˜_soS

s=1

Goal

design an algorithm that iteratively

1 strategically querysome˜x_s to get associatedy˜_s

2 move (˜x_s,y˜_s)fromD_utoD_l

3 learnclassifier g^(t)fromD_l

and improvetest accuracy of g^(t) w.r.t#queries

how toquery strategically?

(43)

How to Query Strategically?

Strategy 1

askmost confused question

Strategy 2

askmost frequent question

Strategy 3

askmost debateful question

• choosingone single strategy isnon-trivial:

0 10 20 30 40 50 60

0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8

% of unlabelled data

Accuracy

RAND UNCERTAIN PSDS QUIRE

0 10 20 30 40 50 60

0.4 0.5 0.6 0.7 0.8 0.9

Accuracy

0 10 20 30 40 50 60

0.5 0.6 0.7 0.8 0.9 1

Accuracy

application intelligence: how to choose strategy smartly?

(44)

Idea: Trial-and-Reward Like Human

when do humanstrial-and-reward?

gambling

K strategies:

A₁, A₂, · · · , A_K

tryone strategy

“goodness” of strategy asreward

K bandit machines:

B₁, B2, · · · , BK

tryone bandit machine

“luckiness” of machine asreward

intelligent choice of strategy

=⇒intelligent choice ofbandit machine

(45)

Active Learning by Learning

(Hsu and Lin, AAAI 2015)

K strategies:

A₁, A₂, · · · , A_K

try one strategy

“goodness” of strategy as reward

Given: K existing active learning strategies for t = 1, 2, . . . , T

1 let some bandit modeldecide strategy A_k to try

2 query the ˜x_ssuggested by A_k, and compute g^(t)

3 evaluategoodness of g^(t) asrewardoftrialto update model

proposed Active Learning by Learning (ALBL):

motivated but unrigorousreward design

(46)

Comparison with Single Strategies

UNCERTAINBest

510 15 20 25 30 35 40 45 50 55 60 0.55

0.6 0.65 0.7 0.75 0.8 0.85 0.9

Accuracy ALBL

vehicle

PSDSBest

510 15 20 25 30 35 40 45 50 55 60 0.5

0.55 0.6 0.65 0.7 0.75 0.8

Accuracy ALBL

sonar

QUIREBest

510 15 20 25 30 35 40 45 50 55 60 0.5

0.55 0.6 0.65 0.7 0.75

Accuracy ALBL

diabetes

• no single best strategyfor every data set

—choosing needed

• proposedALBLconsistentlymatches the best

—similar findings across other data sets

‘application intelligence’ outcome:

open-source toolreleased

(https://github.com/ntucllab/libact)

(47)

Have We Made Active Learning More Realistic? (1/2)

Yes!

open-source toollibactdeveloped (Yang, 2017) https://github.com/ntucllab/libact

• including uncertainty, QUIRE, PSDS, . . .,and ALBL

• received>700stars,>40citations, and continuousissues

“libact is a Python package designed tomake ac- tive learning easierfor real-world users”

(48)

Have We Made Active Learning More Realistic? (2/2)

No!

• single-most raisedissue: hard to install on Windows/Mac

—because several strategies requires some C packages

• performance in an industry project:

• uncertaintysamplingoften suffices

• ALBLdragged down by bad strategy

“libact is a Python package designed to make active learning easierfor real-world users”

(49)

Lessons Learned from

Research on Active Learning by Learning

by DFID - UK Department for International Development;

licensed under CC BY-SA 2.0 via Wikimedia Commons

1 scalability bottleneckof ‘application intelligence’:

choiceof methods/models/parameter/. . .

2 think outside of themathbox:

‘unrigorous’ usage may begood enough

3 important to bebraveyetpatient

• idea: 2012

• paper:(Hsu and Lin, AAAI 2015); software:(Yang et al., 2017) 4 easy-to-use in design 6=easy-to-use in reality

(50)

Tropical Cyclone Intensity Estimation

(51)

Experienced Meteorologists Can ‘Feel’ and Estimate Tropical Cyclone Intensity from Image

Can ML do the same/better?

• lack ofML-ready datasets

• lack ofmodel that properly utilizes domain knowledge issues addressed in our latest works

(Chen et al., KDD ’18; Chen et al., Weather & Forecasting ’19)

(52)

Recall: Flow behind Our Proposed Model

TC images

ML

_estimation^intensity

human learning/

analysis

domain knowledge

(HI)

CNN polar

rotation invariance

current weather

system

is proposedCNN-TCbetter than current weather system?

(53)

Results

RMS Error

ADT 11.75

AMSU 14.40

SATCON 9.66 CNN-TC 9.03

CNN-TC much betterthan current weather system (SATCON)

why are peoplenot using thiscool ML model?

(54)

Lessons Learned from

Research on Tropical Cyclone Intensity Estimation

1 again,cross-domain collaborationimportant e.g. even from ‘organizing data’ to be ML-ready

2 not easy to claimproduction ready

—can ML be used for ‘unseenly-strongTC’?

3 good AI system requiresboth human and machine learning

—still an ‘art’ to blend the two

(55)

Outline

ML for (Modern) AI

(56)

Frequently Asked Questions of ML for AI (1/4)

What is the best AI project for

(my precious big) data?

My Polite Answer

good start already , any more thoughts that you have in mind?

My Honest Answer I don’t know.

or a slightly longer answer:

if you don’t know, I don’t know.

(57)

A Similar Scenario

What is the best AI project for

(my precious big) data?

how to find a research topic for my thesis?

My Polite Answer

good start already , any more thoughts that you have in mind?

I don’t know, but perhaps you canstartby thinking aboutmotivationandfeasibility.

(58)

Finding AI Projects ≈ Finding Research Topics

• motivation: what are you interested in?

• feasibility: what can or cannot be done?

motivation

• something publishable?

oh, possiblyjust for people in academia

• something thatimproves xyz performance

• something that inspires deeper study

—helpsgeneratequestions

feasibility

• modeling

• computational

• budget

• timeline

• . . .

—helpsfilterquestions tip: important forfirst AI projectto be

of high success possibility

(59)

Frequently Asked Questions of ML for AI (2/4)

Should I use ML (or my precious Deep

Learning) for my AI project?

My Polite Answer

let’s understand more about the constraints of your project, shall we ?

if you don’t know, I don’t know.

(60)

Necessary Conditions for Using ML

machine learning: improving someAI goal

machine learning:

with experienceaccumulatedfromdata

data

ML

^{AI goal}

1 existssome “underlying pattern” to be learned

—so “AI goal” possible

2 butnoprogrammable (easy)definition

—so “ML” is needed

3 somehow there isenough dataabout the pattern

—so ML has some “inputs” to learn from

necessary, butnot sufficient, for using ML

(61)

Human Learning versus Machine Learning

big data

ML AI

human learning/

analysis

domain knowledge

(HumanI)

method

model expert system

Human Learning

• subjective

• produce domain knowledge

• fast basic solution

Machine Learning

• objective

• leverage computing power

• continuous improvement tip: use humans as much as possible first

before going to machines

(62)

Frequently Asked Questions of ML for AI (3/4)

What is the best machine learning model for (my precious big) data and AI?

My Polite Answer the best model is

data-dependent, let’schat about your data first

I don’t know aboutbest, but perhaps you can startby thinking aboutsimple models.

(63)

Sophisticated Model for AI

What is the best machine learning model for (my precious big) data and AI?

What is the most sophisticated machine learning model for (my precious big) data and AI?

• myth: my AI works best withmost sophisticatedmodel

• sophisticated model:

• time-consuming totrainandpredict

• difficult totuneormodify

• hard to “simplify” nor “analyze”

sophisticated modelshouldn’t befirst choice

(64)

Simple First

What is the first machine learning model for (my precious big) data and AI?

Taught in ML Foundations on NTU@Coursera simple model first:

• efficient totrainandpredict

• easy totuneormodify

• somewhat“analyzable”

• littlerisk

tip: KISS Principle

—Keep It Simple,^XXStupid_X_XSafe

(65)

Frequently Asked Questions of ML for AI (4/4)

How to Get my AI Project Started?

Old Me I don’t know.

New Me

I know one key factor!

let’s see what the key factor is

(66)

Todos in AI Project

machine learning (big)

data

artificial intelligence

data

• gathering

• cleaning

• storing

• · · ·

techniques

• modeling

• computation

• non-ML tech.

• · · ·

usage

• evaluation

• deployment

• scalability

• · · ·

key first step: set upevaluation criteria

(67)

Evaluation Criteria Guide AI Project Planning

(free image by Manfred Steger from Pixabay)

suggest improvement opportunities m

data hint

preparation steps

⇐=techniques assist

model/tech. choices

⇐=

usage define

acceptance goals

tip: always start with

reasonable & measurable criteria to describe prioritizedAI goal

(68)

Summary

• ML for (Modern) AI:

tools + human knowledge ⇒easy-to-use application

• ML Research for Modern AI:

need to bemore open-minded

—in methodology, in collaboration, in KPI

• ML for AI in Reality:

• motivated/feasible project withmeasurable criteria

• human and/orsimplemodel first

Thank you! Questions?