N T U M I U L A B
An Empirical Study of Content Understanding in Conversational Question Answering
Ting-Rui Chiang, Hao-Tong Ye and Yun-Nung (Vivian) Chen
https://github.com/MiuLab/CQA-Study
N T U M I U L A B
•Background
•Potential issues of the CQA task.
•Potential issues of the CQA models.
Outline
N T U M I U L A B
• Conversational question answering task
• Models we will analyze
Background
N T U M I U L A B
The Question Answering Task
Passage:
Some forms of civil disobedience, such as illegal boycotts, refusals to pay taxes, …
"although civil disobedients are constrained i n t h e i r u s e o f c o e r c i o n b y t h e i r conscientious aim to engage in moral dialogue, nevertheless they may find it necessary to employ limited coercion in order to get their issue onto the table." The Plowshares organization temporarily closed GCSB Waihopai by padlocking the gates and using sickles to deflate one of the large domes covering two satellite dishes.
Q: What goal do many of these protests have?
A: get their issue onto the table
Q: What action did Brownlee take against GCSB Waihopai that caused them to close temporarily?
A: <no answer>
Q: Brownlee argues that sometimes people behave in what way to have their issue heard?
A: limited coercion
Independent!
Independent!
N T U M I U L A B
Q: And the second ?
Conversation History
The Conversational Question Answering Task
Passage:
Secularism is the principle of the separation of government institutions and persons
mandated to represent the state from …
One manifestation of secularism is asserting the right to be free from religious rule and teachings, or, in a state declared to be
neutral on matters of belief, from the imposition by government of religion or religious practices upon its people. Another manifestation of secularism is the view that public activities and decisions, especially political ones, should be uninfluenced by religious beliefs or practices.
Q: How many manifestation of
secularism have been mentioned in this passage?
A: two
Q: Give me one of them?
A: to be free from religious rule and
teachings
N T U M I U L A B
Datasets for Conversational Question Answering
QuAC
Question Answering in Context
A Conversational Question Answering Challenge
CoQA
N T U M I U L A B
Models Studied
CoQA
QuA C
Q 1 Question 3 Passage
Attention + LSTM Intermediate Output
Attention + LSTM Intermediate Output Self-Attention + LSTM
Intermediate Output LSTM
Q 2
FlowQA
Huang et al.Question Passage
Transformer + Residual + Linear Transformer + Residual + Linear
Transformer + Residual + Linear × 6
BERT
Linear + 2 x Softmax
…
Delvin et al.
Passage Question
Multi-Level Attention
BERT
+ GloVe + Word-Level Attention LSTMs
LSTM + Self-Attention + LSTM
SDNet
Ochs et al.
N T U M I U L A B
Formulated as a Span-Selection Task
Passage:
Rawls derives two principles of justice from the original position. The first of these is the Liberty Principle, which establishes equal basic liberties for all citizens. ‘Basic’ liberty entails the freedoms of conscience, association and expression as well as democratic rights; … Thus participants would be moved to affirm a two-part second principle comprising Fair Equality of Opportunity and the famous (and controversial) difference principle. This second principle ensures that those with comparable talents and …
Q: What is the Liberty Principle?
A: establishes equal basic liberties for all citizens.
Q: What is another principle?
A: a two-part second principle comprising Fair Equality of
Opportunity and the famous (and controversial) difference principle.
Q: What is the Principles of justice?
A: The first of these is the Liberty
Principle,
N T U M I U L A B
Span-Selection Model
? Rawls derives two principles
…
The first What is
…
Question
…
Liberty Principle ,
Super Deep Model
which
2 x Softmax
Passage
✓ ✓
The first … Liberty Principle,
N T U M I U L A B
The True Story
N T U M I U L A B
Conversation History
We Have Conversation History!
Q: What is the Liberty Principle?
A: establishes equal basic liberties for all citizens.
Q: What is the Principles of justice?
A: The first of these is the Liberty Principle,
Q: What is another principle?
?
Passage:
Rawls derives two principles of justice from the original position. The first of these is the Liberty Principle, which establishes equal basic liberties for all citizens. \‘Basic\’ liberty entails the freedoms of conscience, association and expression as well as democratic rights; … Thus participants would be moved to affirm a two-part second principle comprising Fair Equality of Opportunity and the famous (and controversial) difference principle. This second principle ensures that those with comparable talents and …
N T U M I U L A B
All After DSTC7…
Transformer
Attention
Transformer Transformer
Attention
Utterance 1 Question 1 Utterance 2 Answer 1 Utterance 3 Question 2
Super Deep Model
Passage
N T U M I U L A B
Look into the Problem…
N T U M I U L A B
Super Deep Model
Feeding History with Answer Indicator
Answer Indicator
which establishes equal basic liberties for all citizens
Passage Question
. ‘ Basic
,
previous answer
N T U M I U L A B
Observation: Answer Distance
(in words)Passage :
The first of these is the Liberty Principle, which establishes equal basic liberties for all
citizens. ‘Basic’ liberty …
1 word
answer 1 answer 2
0 ~ 5
N T U M I U L A B
We Ask Question 1…
Is the position of the previous answer
a huge hint?
N T U M I U L A B Minus Text:
Passage: Rawls derives two principles of justice from the original position. The first of these is the Liberty Principle, which establishes equal basic liberties for all citizens.
‘Basic’ …
Q: What is the Liberty … ? A: establishes equal basic liberties for all citizens.
Q: What is the Principles … ? A: The first of these is the …
Train & test in 3 different settings:
Experiments to Verify it
Q: What is another principle?
Original:
Passage: Rawls derives two principles of justice from the original position. The first of these is the Liberty Principle, which establishes equal basic liberties for all citizens.
‘Basic’ …
Q: What is the Liberty … ? A: establishes equal basic liberties for all citizens.
Q: What is the Principles … ? A: The first of these is the …
mask mask
mask
mask
Minus Conversation:
Passage: Rawls derives two principles of justice from the original position. The first of these is the Liberty Principle, which establishes equal basic liberties for all citizens.
‘Basic’ …
Q: What is the Liberty … ? A: establishes equal basic liberties for all citizens.
Q: What is the Principles … ? A: The first of these is the …
<Premise>
Content understanding is necessary. → Minus Text
position only
N T U M I U L A B
Results on QuAC
52 54 56 58 60 62 64 66
FlowQA FlowQA - Text
FlowQA - Conversation
BERT BERT
- Text
BERT
- Conversation
N T U M I U L A B
Results on CoQA
52 57 62 67 72 77 82
FlowQA FlowQA - Text
FlowQA - Conversation
SDNet SDNet
- Text
SDNet - Conversation
N T U M I U L A B
• Content understanding:
• minus text ≤ minus conversation
• Performance:
• minus text >> minus conversation
• Better Score ↛ better content understanding.
Implications
We should look beyond a single metric!
N T U M I U L A B
Wait!
What do you mean by
understanding ?
NO access to content
thus NO understanding
of content
N T U M I U L A B
We Ask Question 2…
Do previous models understand
the content of conversations?
N T U M I U L A B
• Increase the distance between answers?
• The distance is increased, but the semantic is kept!
Proposed: Repeat Attack
Passage: Rawls derives two principles of justice from the original position. The first of these is the Liberty Principle, The first of these is the Liberty Principle, which establishes equal basic liberties for all citizens. ‘Basic’ …
Passage: Rawls derives two principles of justice from the original position. The first of these is the Liberty Principle, which establishes equal basic liberties for all citizens. ‘Basic’ liberty entails the freedoms of conscience, …
answer 1 answer 2
The first of these is the Liberty Principle,
Repeat
answer 1
answer 2
<Premise>
Model understand the content → ☺ even attacked
N T U M I U L A B
Results on QuAC
52 54 56 58 60 62 64 66
FlowQA BERT
N T U M I U L A B
Results on QuAC
52 54 56 58 60 62 64 66
FlowQA BERT FlowQA
- position
BERT - position
N T U M I U L A B
Results on CoQA
52 57 62 67 72 77 82
FlowQA SDNet
N T U M I U L A B
Quantitatively
N T U M I U L A B
Victim of the Attack
Passage:
In 2004, Old man returned to prominence when he landed a significant role in the Harry Potter film series, playing Harry Potter’s godfather Sirius Black. In 2004, Old man returned to prominence when he landed a significant role in the Harry Potter film series, playing Harry Potter’s godfather Sirius Black. The following year, he starred as James Gordon in Christopher Nolan’s commercially and
critically successful Batman Begins, starred as James Gordon in Christopher Nolan’s commercially and critically successful Batman Begins, ...
answer 1
Q: What was his resurgence or comeback role?
A: In 2004, Old man returned to prominence when he landed a significant role … Q: Are there any other interesting aspects about this article?
