4.2 Collocational Congruency Classification by Semantic Similarity Values
4.2.1 Applying Sematic Similarity Values and Examining Congruency . 77
The first part of the second quantitative sub-study applied the semantic similarity measures to the collocation test set so as to provide empirical evidences for the conjectured semantic similarity distribution. Subjective congruency classification of collocations was then cross examined with their computational semantic similarity.
Statistical analysis was then performed on the congruency categories for significance of difference in the numerical values of computational semantic similarity. The purpose was to evaluate the consistency of subjective congruency classification from the perspective of objective semantic similarity and to reveal potential classification conflicts.
As noted in previous chapter, the congruency classification on the collocation test set was based on an initial and subjective judgment by the researcher. The transferred word from each L1 collocate in the test set was also provided by the participants’ most common choice as an exemplar learner’s selection. The computation of semantic similarity measures between two words also involves the selection of word sense in two modes. In the single-word-sense mode, each polysemous word needs to be assigned a particular word sense for semantic similarity evaluation. In the all-word-sense mode, no word sense is assigned and all word meanings of the word pairs are considered so as to match the closest word meanings.
In other words, the semantic similarity evaluation, when operated in the all-word-sense mode, gives the highest value to represent the most similar word senses of the two polysemous words. In the L2 learning context, semantic similarity evaluation in the single-word-sense mode can be used to simulate lexical knowledge of low-level to mid-level learners, while the all-word-sense mode may assume the characteristics of more advanced learners. When selecting a particular word sense in the single-word-sense mode, L2 learners’ primary perception of word meaning would usually be a good consideration.
As introduced previously, the measures of semantic similarity, Gloss Vectors and
Adapted Lesk, provide a deterministic and algorithmic evaluation of semantic
similarity between any pair of English words. However, the two measures do not produce a similar range of values. To provide a more convenient and complementary similarity observation of the two measures, the values of the Adapted Lesk measure were converted by logarithm, as Log (Lesk+1). The addition of one to the Lesk values before logarithmic conversion was to avoid mathematic peculiarity because Lesk value started from zero. For binary classification of similarity level, the thresholds were judiciously ascertained at 0.6 for the Gloss Vectors measure and 99 for theAdapted Lesk measure (i.e., 2 for Log (Lesk+1)). In other words, semantic similarity
of a pair of L1/L2 collocates was classified as high if the evaluated value of GlossVectors measure was higher than 0.6 and/or if the evaluated value of Log (Lesk+1)
was higher than 2. In a few cases when the evaluative grades were not consistent between the two measures, the grade (high similarity or low) given by the GlossVectors measure was adopted.
Table 4.2.1 reports the similarity values of both the subjectively congruent and subjectively incongruent verb-noun L2/L1 collocate pairs in the collocation test set based on the two semantic measures, Gloss Vectors and Adpated Lesk, in both
contexts of single word sense and all word senses.
Table 4.2.1 Semantic Similarity Values of Verb-Noun L2/L1 Collocates in Subjective Congruency Classification
Single Word Sense All Word Sense Vector Lesk Vector Lesk
overcome (challenges) conquer 0.750 1.491 0.750 1.491
Incongruent
In Table 4.2.1, it was noted that all subjectively congruent L1/L2 verb collocates were indeed of high semantic similarity. In addition, eight out of the ten verb collocate pairs received the highest similarity value in the Gloss Vectors measure. The similarity evaluations were also not affected by difference in learners’ lexical knowledge as the similarity values were the same in the two modes of
single-word-sense and all-word-sense. The experiments showed that subjective and objective evaluations for congruency were consistent on semantically similar pairs of verb collocate. However, most transferred words from participants’ common choice were incorrect use. This indicated congruent verb-noun collocations may not be assumed to be easy and straightforward for L2 learners. More experimental data will be further examined in the third quantitative sub-study.
For subjectively incongruence, some variations were observed. In the single-word-sense mode, e.g., learners’ lexical knowledge on the collocates was assumed to be limited to primary word meanings, most subjectively incongruent verb collocate pairs were indeed of low semantic similarity. The collocate pairs of “solve and break”, “make and get”, were the only two exceptions and surprisingly showed high similarity. In the all-word-sense mode, when assumption of learners’ lexical knowledge on the collocates was extended to comprehensive word meanings, however, most subjectively incongruent verb collocate pairs showed high semantic similarity. Only three collocate pairs, e.g., “make and say”, “leave and make”,
“restore and recover”, remained of low semantic similarity. This empirical results showed that congruency might depend on learners’ lexical knowledge on the candidate collocates. Again, congruency effects on learners’ collocation production and performance will be discussed later.
The semantic similarity distributions of verb noun collocations in single word sense and in all word senses were shown in Figure 4.2.4 and Figure 4.2.5, respectively.
Each dot represented a collocate pair whose coordinate in the two dimensional space was the two similarity values from the Gloss Vectors measure and the Adapted Lesk measure. The horizontal and vertical dotted lines were borderlines that distinguish between regions of low and high similarity. Collocate pairs that were subjectively classified as congruent were shown in filled circle dots, while blank rectangular dots
indicated collocate pairs that were subjectively considered as incongruent.
Both distributions in Figure 4.2.4 and Figure 4.2.5 formed diagonal regions from lower left to upper right, as previously expected in Figure 4.2.2. The two groups of dots also showed some intercrossing in regional boundary, more so in Figure 4.2.5 than in Figure 4.2.4. As conjectured previously in Figure 4.2.3, this empirical result
0 0.2 0.4 0.6 0.8 1 1.2
0 0.5 1 1.5 2 2.5 3 3.5 4
Vector
Log (Lesk + 1) Subjectively Congruent: ● Subjectively Incongruent: □
Figure 4.2.4 Semantic Similarity Distribution of VN Collocations in Single Word Sense
0 0.2 0.4 0.6 0.8 1 1.2
0 0.5 1 1.5 2 2.5 3 3.5 4
Vector
Log (Lesk + 1) Subjectively Congruent: ● Subjectively Incongruent: □
Figure 4.2.5 Semantic Similarity Distribution of VN Collocations in All Word Senses
attested to the inconsistency between subjective and objective congruency, especially in L2 learning context when learners’ lexical knowledge might vary.
Table 4.2.2 Semantic Similarity Values
of Adjective-Noun L2/L1 Collocates in Subjective Congruency Classification
Subjective
Single Word Sense All Word Sense Vector Lesk Vector Lesk
strong (coffee) concentrated 0.252 1.556 0.628 2.009 light (sleep) shallow 0.101 1.653 0.721 1.954
small (hours) early 0.307 1.862 0.307 2.413
For the adjective-noun type, Table 4.2.2 summarizes the similarity values of both the subjectively congruent and incongruent L2/L1 collocate pairs in both contexts of single word sense and all word senses. It was observed that three of the ten subjectively congruent collocate pairs, e.g., “common and general”, “rough and
approximate”, “raw and original” were surprisingly of low similarity in single word
sense. When more comprehensive word meanings were assumed, one collocate pair, e.g., “raw and original”, still remained of low similarity. This example highlighted the potentially unreliable evaluation of human judgment. Both “raw” and “original”had exactly the same Chinese translation as “原始 (yuan-shih)” but their word meanings in English were actually quite different.
For the part of subjective incongruence, all adjective L2/L1 collocate pairs were indeed of low similarity in single word sense. However, in the context of all word senses, half of the ten collocate pairs, e.g., “heavy and severe”, “regular and frequent”,
“heavy and big”, “strong and concentrated”, “light and shallow”, were evaluated as of high similarity. Again, this result showed that learners’ lexical knowledge on the candidate collocates actually changed congruency classification. It was also conjectured that L2 learners’ difficulties in collcoational use, as observed in the mostly erroneous transferred words, might be attributed to both congruent and incongruent relations.
Figure 4.2.6 and Figure 4.2.7 show the semantic similarity distributions of adjective noun collocations in single word sense and in all word senses, respectively.
Again, diagonal regions from lower left to upper right appeared in both distributions, as previously expected in Figure 4.2.2. The regional dot interacrossing was also more evident. In the context of single word sense, three of the subjectively congruent dots were even at the lower boundary of the subjectively incongruent region. In the context of all word senses, the region of subjectively congruent dots almost shifted away from the bottom left subspace, which was the designated location of the objectively congruent category. The result also indicated the inconsistency between subjective and objective congruency and the potential defect of subjective congruency in simulating learners’ L2 learning context.
Figure 4.2.6 Semantic Similarity Distribution of AdjN Collocations in Single Word Sense
Figure 4.2.7 Semantic Similarity Distribution of AdjN Collocations in All Word Senses
The semantic similarity analysis on both verb-noun and adjective-noun collocations revealed a similar problematic pattern of inconsistent classification between human (subjective) and computational (objective) evaluations. This inconsistency of congruency evaluation was further aggravated by the different
0 0.2 0.4 0.6 0.8 1 1.2
0 0.5 1 1.5 2 2.5 3 3.5
Vector
Log (Lesk + 1) Subjectively Congruent: ● Subjectively Incongruent: □
0 0.2 0.4 0.6 0.8 1 1.2
0 0.5 1 1.5 2 2.5 3 3.5
Vector
Log (Lesk + 1) Subjectively Congruent: ● Subjectively Incongruent: □
conditions of learners’ word sense level. For verb-noun collocations, the worst inconsistency occurred in the subjectively incongruent category with the assumption of learners’ all word senses, where seven out of ten collocations that were humanly judged as of low similarity, were instead, computationally considered as of high similarity. For adjective noun collocations, both subjectively congruent and incongruent categories exhibited inconsistency albeit in different contexts of learners’
word senses. More inconsistency, e.g., three out of ten collocations were evaluated differently between human and computational views, occurred in the subjectively congruent category assuming learners’ single word sense and in the subjectively incongruent category assuming learners’ all word senses.
Table 4.2.3 Descriptive Summary of Computational Semantic Similarity in Verb Noun Collocations
Word Sense
Subjective Congruency N
Gloss Vectors Log (Lesk+1) Mean Std.
Dev. Min. Max. Mean Std.
Dev. Min. Max.
Single Congruent 10 0.945 0.118 0.695 1.0 2.707 0.544 1.491 3.364 Incongruent 10 0.355 0.305 0.066 1.0 1.674 0.649 0.903 2.967 All Congruent 10 0.945 0.118 0.695 1.0 2.707 0.544 1.491 3.364 Incongruent 10 0.793 0.094 0.197 1.0 2.342 0.389 1.591 2.967
For further verification, a statistical analysis was also performed on the semantic similarity differences between congruency categories. Table 4.2.3 reported the descriptive summary of the computational semantic similarity in verb-noun collocations. Mean values of the Vector measures of subjectively congruent collocations, in both contexts of learners’ single word sense and all word sense, was very close to 1.0, indicating that semantics of the L2 collocates and the transferred words from the L1 counterpart were almost identical. For subjectively incongruent verb noun collocations, mean values of the Vector measures in the context of learners’
single word sense indicated low similarity. However, in the context of learners’ all word senses, mean values of the Vector measures of subjectively incongruent verb-noun collocations more than doubled and indicated high similarity. A similar pattern of varying similarity evaluation between subjectively congruent and incongruent verb-noun collocations under different contexts of learners’ proficiency levels was also observed on the Lesk measures.
Table 4.2.4 One-Way ANOVA on Computational Semantic Similarity between Subjective Congruency Categories in Verb Noun Collocations
Word Sense Subjective Congruency Sum of
Squares df Mean
Square F Sig.
Single
Vector Between Groups 1.735 1 1.735 32.448 .000 Within Groups .962 18 .053
Total 2.698 19
Lesk Between Groups 5.334 1 5.334 14.877 .001 Within Groups 6.454 18 .359
Total 11.787 19
All
Vector Between Groups .114 1 .114 2.228 .153 Within Groups .924 18 .051
Total 1.038 19
Lesk Between Groups .666 1 .666 2.984 .101 Within Groups 4.019 18 .223
Total 4.685 19
Table 4.2.4 reports the statistical comparison between subjective congruency categories by two computational measures of semantic dimilarity under learners’
different proficiency contexts. It has shown that semantic similarity differences between subjective congruency categories were statistically significant by both measures in the context of learners’ single word sense, e.g., F (1, 18) = 32.448, p = 0.000 < 0.05, and F(1, 18) = 14.877, p = 0.001 < 0.05. However, in the context of learners’ all word senses, there was no statistically significant difference in the
semantic similarity by both measures between congruency categories, e.g., F(1, 18) = 2.228, p = 0.153 > 0.05, and F(1, 18) = 2.984, p = 0.101 > 0.05.
Table 4.2.5 Descriptive Summary of the Computational Semantic Similarity in Adjective Noun Collocations
Word Sense
Subjective Congruency N
Vector Lesk
Mean Std.
Dev. Min. Max. Mean Std.
Dev. Min. Max.
Single Congruent 10 0.660 0.434 0.043 1.0 2.010 0.861 0.778 3.212 Incongruent 10 0.191 0.093 0.043 0.319 1.562 0.339 1.041 2.201 All Congruent 10 0.835 0.279 0.121 1.0 2.345 0.504 1.415 3.212 Incongruent 10 0.499 0.345 0.043 1.0 1.929 0.391 1.322 2.413
For adjective noun collocations, Table 4.2.5 reports the descriptive summary of the computational semantic similarity by both measures. Mean values of the Vector measures on subjectively congruent category in learners’ single word sense was just slightly over the threshold that might be considered as indicating high similarity. Even in learners’ all word senses, mean values of the Vector measure on subjective congruent category increased to indicate high similarity, but not to the level of identical semantics. For subjectively incongruent adjective noun collocations, mean values of the Vector measure in the context of learners’ single word sense was very low and indicated little similarity. In the context of learners’ all word senses, mean values of the Vector measure of subjectively incongruent adjective noun collocations increased but remained low similarity. The Lesk measures showed a similar pattern of varying similarity evaluation between subjectively congruent and incongruent adjective noun collocations under different contexts of learners’ proficiency levels.
Table 4.2.6 reports the statistical comparison between subjective congruency categories by two computational measures under learners’ different proficiency contexts. It has shown that semantic similarity differences between subjective
congruency categories were statistically significant by the Vector measure in both contexts of learners’ proficiency levels, e.g., F(1, 18) = 11.199, p = 0.004 < 0.05, and
F(1, 18) = 5.758, p = 0.027 < 0.05. However, for the Lesk measure, there was no
statistically significant difference in the semantic similarity between subjective congruency categories in both contexts of learners’ proficiency levels, e.g., F(1, 18) = 2.344, p = 0.143 > 0.05, and F(1, 18) = 4.247, p = 0.054 > 0.05.Table 4.2.6 One-Way ANOVA on Computational Semantic Similarity between Subjective Congruency Categories in Adjective Noun Collocations Word Sense Subjective Congruency Sum of
Squares df Mean
Square F Sig.
Single
Vector Between Groups 1.103 1 1.103 11.199 .004 Within Groups 1.773 18 .098
Total 2.876 19
Lesk Between Groups 1.003 1 1.003 2.344 .143 Within Groups 7.703 18 .428
Total 8.706 19
All
Vector Between Groups .567 1 .567 5.758 .027 Within Groups 1.771 18 .098
Total 2.338 19
Lesk Between Groups .864 1 .864 4.247 .054 Within Groups 3.662 18 .203
Total 4.526 19
The inconsistency between human (subjective) and computational (objective) congruency classification was manifested in both verb noun collocations and adjective noun collocations. Both the item-level and the category-level examination showed that computational and human congruency evaluations might not share the same view.
In addition, human congruency evaluation might not account for learners’ varying proficiency levels. This analysis revealed that congruency could become ambiguous and disconcerted in the contexts of human evaluation and learners’ various
proficiency levels. Further studies on better congruency classification and its effects on L2 learners’ collocation performance were required.
In conclusion, the first part of the second quantitative sub-study empirically verified the applicability of computational semantic measures in classification of L2 collocation congruency. It has shown that objective evaluation of congruency required an input of transferred words from L1 collocate and then operated purely on the L2.
This might avoid the fallacy of subjective and cross-linguistic evaluation of congruency. In addition, this learner-centered congruency evaluation more closely simulated the context of L2 learners’ lexical decision process.
4.2.2 Objective Congruency Classification
The use of computational measures of semantic similarity provided an objective way of classifying collocation congruency. This helped clarify the notion of congruency and avoided inconsistent classification by subjective human evaluation.
Another advantage was the refined capability to consider learners’ proficiency levels in congruency classification. In this part of empirical sub-study, objective congruency classification based on semantic similarity was performed under learners’ different proficiency levels. This new objective congruency classification was, then, examined by a statistical analysis for significance of difference.
Table 4.2.7 and Table 4.2.8 report the objective congruency classification of verb noun collocations based on semantic similarity in the contexts of learners’ single word sense and all word senses, respectively. Semantic similarity values of the Gloss
Vectors measure higher than 0.6 were considered as indicating high similarity. L2
collocates with high similarity to the transferred words from L1 counterparts were classified as objectively congruent and those with low similarity were classified as objectively incongruent.Table 4.2.7 Objective Congruency Classification on Verb Noun Collocations in the Context of Learners’ Single Word Sense
Objective Congruence
acquire (knowledge) get 1.0 surf (Internet) browse 0.066
seek (information) search 1.0 make (apology) say 0.421
make (effort) do 1.0 study (English) read 0.201
see (play) watch 1.0 carry (lantern) hold 0.141
increase (ability) increase 1.0 ease (worry) relieve 0.249 maintain (relationship) keep 1.0 conduct (heat) transmit 0.083 preserve (culture) conserve 1.0 make (impression) leave 0.465 make (trouble) make 1.0 restore (vitality) recover 0.197
take (action) do 0.695
overcome (challenge) conquer 0.750
solve (crime) break 0.731
make (conclusion) get 1.0
In Table 4.2.7 where learners’ single word sense was assumed, the objectively congruent category had two more verb noun collocations, re-allocated from the subjectively incongruent category. In particular, “solve crime” and “make conclusion”
were now classified as congruent based on their high similarity with transferred words from L1 counterparts. In Table 4.2.8 where learners’ all word senses was assumed, five more verb noun collocations were re-classified to the objectively congruent category and only three verb noun collocations remained in the objectively incongruent category. In particular, “surf Internet”, “study English”, “carry lanterns”,
“ease worry”, and “conduct heat” showed identical semantics to their transferred words from L1 counterparts and were contrary to human evaluation. This showed that objective classification with additional factors of leaners’ proficiency levels could be
considerably different to a particular instance of subjective classification.
Table 4.2.8 Objective Congruency Classification on Verb Noun Collocations in the Context of Learners’ All Word Senses
Objective Congruence
seek (information) search 1.0 make (impression) leave 0.465
make (effort) do 1.0 restore (vitality) recover 0.197
see (play) watch 1.0
increase (ability) increase 1.0 maintain (relationship) keep 1.0 preserve (culture) conserve 1.0
make (trouble) make 1.0
take (action) do 0.695
overcome (challenge) conquer 0.750 surf (Internet) browse 1.0
conduct (heat) transmit 1.0
A statistical analysis was further performed to examine the consistency of objective congruency classification of verb noun collocations based on similarity measures. Table 4.2.9 reports the descriptive summary of the Vectors measure for objective congruency classification in two different contexts of learners’ word sense levels. As compared to Table 4.2.3, the means difference between congruent and incongruent categories in both learners’ word sense contexts became wider, which
indicated better congruency distinction in the semantic sense. A one-way ANOVA on the Vectors measure for objective congruency classification of verb noun collocations, as reported in Table 4.2.10, also supported the evaluation of better distinction of classification. The significance of difference of the Vectors measure between objective congruency categories in both learners’ contexts were verified, e.g., F(1, 18) = 132.841, p = 0.000 < 0.05, and F(1, 18) = 52.944, p = 0.000 < 0.05. Compared to that of subjectively congruency categories, e.g., F(1, 18) = 32.448, p = 0.000 < 0.05, and
F(1, 18) = 2.228, p = 0.153 > 0.05, reported in Table 4.2.4, the objective congruency
classification showed a higher level of semantic differentiation and a better consistency in different contexts of learners’ word sense levels.Table 4.2.9 Descriptive Summary of the Vector Measure for Objective Congruency Classification of Verb Noun Collocations
Word Sense Objective
Congruency N Mean SD Min. Max.
Single Congruent 12 0.931 0.125 0.695 1.0
Incongruent 8 0.228 0.147 0.066 0.465
All Congruent 17 0.952 0.108 0.695 1.0
Incongruent 3 0.400 0.194 0.197 0.465
Table 4.2.10 One-Way ANOVA on the Vectors Measure for Objective Congruency Classification of Verb Noun Collocations
Word Sense Objective Congruency
Sum of
Squares df Mean
Square F Sig.
Single
Between Groups 2.376 1 2.376 132.841 .000
Within Groups .322 18 .018
Total 2.698 19
All
Between Groups .775 1 .775 52.944 .000
Within Groups .263 18 .015
Total 1.038 19
For adjective noun collocations, the same objective congruency classification criteria and procedure were conducted. Table 4.2.11 and Table 4.2.12 report the results of objective congruency classification in both learners’ word sense contexts. In the context of learners’ single word sense, the objective congruency classification re-allocated three adjective noun collocations from subjective congruent category to objective incongruent category. In particular, “common cold”, “rough estimate”, and
“raw data”, previously considered as congruent by the researcher, were objectively classified as incongruent based on their low similarity.
Table 4.2.11 Objective Congruency Classification on Adjective Noun Collocations in the Context of Learners’ Single Word Sense
Objective Congruence
high (tech) high 1.0 rough (estimate) approximate 0.067
inside (information) inner 1.0 raw (data) original 0.047
mental (age) mental 1.0 heavy (drinker) severe 0.089
severe (pain) serious 1.0 sound (diet) perfect 0.177
mutual (respect) reciprocal 1.0 narrow (victory) near 0.212 rare (wildlife) scarce 0.700 regular
(customer) frequent 0.319
runny (nose) watery 0.043
heavy (rain) big 0.239
foul (play) illegal 0.165 strong (coffee) concentrated 0.252 light (sleep) shallow 0.101
small (hours) early 0.307
In the context of learners’ all word senses, five adjective noun collocations, e.g.,
“heavy drinker”, “regular customer”, “heavy rain”, “strong coffee”, and “light sleep”,
previously considered as incongruent by the researcher, were now objectively classified as congruent based on their high similarity. One adjective noun collocation, e.g., “raw data”, subjectively considered as congruent, was objectively classified as incongruent based on its consistently low similarity. This result also has shown the considerable congruency variation between subjective and objective classification.
Table 4.2.12 Objective Congruency Classification on Adjective Noun Collocations in the Context of Learners’ All Word Senses
Objective Congruence
inside (information) inner 1.0 narrow victory near 0.212
mental (age) mental 1.0 runny nose watery 0.043
severe (pain) serious 1.0 foul play illegal 0.165
mutual (respect) reciprocal 1.0 small hours early 0.307
rare (wildlife) scarce 0.700
common (cold) general 0.787
rough (estimate) approximate 1.0
heavy (drinker) severe 0.627
regular (customer) frequent 1.0
heavy (rain) big 1.0
strong (coffee) concentrated 0.628 light (sleep) shallow 0.721
A statistical analysis on objective congruency classification for adjective noun collocations was also performed. Table 4.2.13 reports the descriptive summary of the
A statistical analysis on objective congruency classification for adjective noun collocations was also performed. Table 4.2.13 reports the descriptive summary of the