C. In the union, we count the element the sum of the number of times it appears in B and in C.
Example 3.2 : The bag-similarity of bags {a, a, a, b} and {a, a, b, b, c} is 1/3.
The intersection counts a twice and b once, so its size is 3. The size of the union of two bags is always the sum of the sizes of the two bags, or 9 in this case. 2
3.1.4 Exercises for Section 3.1
Exercise 3.1.1 : Compute the Jaccard similarities of each pair of the following three sets: {1, 2, 3, 4}, {2, 3, 5, 7}, and {2, 4, 6}.
Exercise 3.1.2 : Compute the Jaccard bag similarity of each pair of the fol-lowing three bags: {1, 1, 1, 2}, {1, 1, 2, 2, 3}, and {1, 2, 3, 4}.
!! Exercise 3.1.3 : Suppose we have a universal set U of n elements, and we choose two subsets S and T at random, each with m of the n elements. What is the expected value of the Jaccard similarity of S and T ?
3.2 Shingling of Documents
The most effective way to represent documents as sets, for the purpose of iden-tifying lexically similar documents is to construct from the document the set of short strings that appear within it. If we do so, then documents that share pieces as short as sentences or even phrases will have many common elements in their sets, even if those sentences appear in different orders in the two docu-ments. In this section, we introduce the simplest and most common approach, shingling, as well as an interesting variation.
3.2.1 k -Shingles
A document is a string of characters. Define a k-shingle for a document to be any substring of length k found within the document. Then, we may associate with each document the set of k-shingles that appear one or more times within that document.
Example 3.3 : Suppose our document D is the string abcdabd, and we pick k = 2. Then the set of 2-shingles for D is{ab, bc, cd, da, bd}.
Note that the substring ab appears twice within D, but appears only once as a shingle. A variation of shingling produces a bag, rather than a set, so each shingle would appear in the result as many times as it appears in the document.
However, we shall not use bags of shingles here. 2
There are several options regarding how white space (blank, tab, newline, etc.) is treated. It probably makes sense to replace any sequence of one or more
white-space characters by a single blank. That way, we distinguish shingles that cover two or more words from those that do not.
Example 3.4 : If we use k = 9, but eliminate whitespace altogether, then we would see some lexical similarity in the sentences “The plane was ready for touch down”. and “The quarterback scored a touchdown”. However, if we retain the blanks, then the first has shingles touch dow and ouch down, while the second has touchdown. If we eliminated the blanks, then both would have touchdown. 2
3.2.2 Choosing the Shingle Size
We can pick k to be any constant we like. However, if we pick k too small, then we would expect most sequences of k characters to appear in most documents.
If so, then we could have documents whose shingle-sets had high Jaccard simi-larity, yet the documents had none of the same sentences or even phrases. As an extreme example, if we use k = 1, most Web pages will have most of the common characters and few other characters, so almost all Web pages will have high similarity.
How large k should be depends on how long typical documents are and how large the set of typical characters is. The important thing to remember is:
• k should be picked large enough that the probability of any given shingle appearing in any given document is low.
Thus, if our corpus of documents is emails, picking k = 5 should be fine.
To see why, suppose that only letters and a general white-space character ap-pear in emails (although in practice, most of the printable ASCII characters can be expected to appear occasionally). If so, then there would be 275 = 14,348,907 possible shingles. Since the typical email is much smaller than 14 million characters long, we would expect k = 5 to work well, and indeed it does.
However, the calculation is a bit more subtle. Surely, more than 27 charac-ters appear in emails, However, all characcharac-ters do not appear with equal proba-bility. Common letters and blanks dominate, while ”z” and other letters that have high point-value in Scrabble are rare. Thus, even short emails will have many 5-shingles consisting of common letters, and the chances of unrelated emails sharing these common shingles is greater than would be implied by the calculation in the paragraph above. A good rule of thumb is to imagine that there are only 20 characters and estimate the number of k-shingles as 20k. For large documents, such as research articles, choice k = 9 is considered safe.
3.2.3 Hashing Shingles
Instead of using substrings directly as shingles, we can pick a hash function that maps strings of length k to some number of buckets and treat the resulting bucket number as the shingle. The set representing a document is then the
3.2. SHINGLING OF DOCUMENTS 61 set of integers that are bucket numbers of one or more k-shingles that appear in the document. For instance, we could construct the set of 9-shingles for a document and then map each of those 9-shingles to a bucket number in the range 0 to 232− 1. Thus, each shingle is represented by four bytes instead of nine. Not only has the data been compacted, but we can now manipulate (hashed) shingles by single-word machine operations.
Notice that we can differentiate documents better if we use 9-shingles and hash them down to four bytes than to use 4-shingles, even though the space used to represent a shingle is the same. The reason was touched upon in Section 3.2.2.
If we use 4-shingles, most sequences of four bytes are unlikely or impossible to find in typical documents. Thus, the effective number of different shingles is much less than 232− 1. If, as in Section 3.2.2, we assume only 20 characters are frequent in English text, then the number of different 4-shingles that are likely to occur is only (20)4 = 160,000. However, if we use 9-shingles, there are many more than 232likely shingles. When we hash them down to four bytes, we can expect almost any sequence of four bytes to be possible, as was discussed in Section 1.3.2.
3.2.4 Shingles Built from Words
An alternative form of shingle has proved effective for the problem of identifying similar news articles, mentioned in Section 3.1.2. The exploitable distinction for this problem is that the news articles are written in a rather different style than are other elements that typically appear on the page with the article. News articles, and most prose, have a lot of stop words (see Section 1.3.1), the most common words such as “and,” “you,” “to,” and so on. In many applications, we want to ignore stop words, since they don’t tell us anything useful about the article, such as its topic.
However, for the problem of finding similar news articles, it was found that defining a shingle to be a stop word followed by the next two words, regardless of whether or not they were stop words, formed a useful set of shingles. The advantage of this approach is that the news article would then contribute more shingles to the set representing the Web page than would the surrounding ele-ments. Recall that the goal of the exercise is to find pages that had the same articles, regardless of the surrounding elements. By biasing the set of shingles in favor of the article, pages with the same article and different surrounding material have higher Jaccard similarity than pages with the same surrounding material but with a different article.
Example 3.5 : An ad might have the simple text “Buy Sudzo.” However, a news article with the same idea might read something like “A spokesperson for the Sudzo Corporation revealed today that studies have shown it is good for people to buy Sudzo products.” Here, we have italicized all the likely stop words, although there is no set number of the most frequent words that should be considered stop words. The first three shingles made from a stop word and the next two following are:
A spokesperson for for the Sudzo
the Sudzo Corporation
There are nine shingles from the sentence, but none from the “ad.” 2
3.2.5 Exercises for Section 3.2
Exercise 3.2.1 : What are the first ten 3-shingles in the first sentence of Sec-tion 3.2?
Exercise 3.2.2 : If we use the stop-word-based shingles of Section 3.2.4, and we take the stop words to be all the words of three or fewer letters, then what are the shingles in the first sentence of Section 3.2?
Exercise 3.2.3 : What is the largest number of k-shingles a document of n bytes can have? You may assume that the size of the alphabet is large enough that the number of possible strings of length k is at least as n.