Machine Learning (NTU, Fall 2008) instructor: Hsuan-Tien Lin
PROOF OF (THE INDUCTION STEP OF) THE SAUER’S LEMMA
Hsuan-Tien Lin
Setup
1. A set of N -bit vectors of +/− is called a N -long set. For instance, the following matrix represents a 4-long set, where each row is an element of the set:
+ + + +
+ + - +
+ - - +
- + + -
2. An N -long set is said to be complete if it includes all possible 2N distinct vectors.
Otherwise it is said to be incomplete. For instance, the following matrix represents an incomplete set:
+ +
+ -
- -
3. An N -long set is said to be M -incomplete if projecting the vectors to any M dimensions results in an incomplete set. For instance, by projecting the 4-long set above to any of the two columns (dimensions), we see that the set is 2-incomplete:
1 2 1 3 1 4 2 3 2 4 3 4
+ + + + + + + + + + + +
+ - + - - - + - - + - +
- + - + - - + - + -
4. B(N, M ) is defined to be the maximum number of unique elements in an N -long and M -incomplete set. For instance, the following matrix represents a set that achieves B(4, 2):
+ + + +
+ + - +
+ - - +
- + + -
+ + + -
The Main Lemma
Lemma 1 (The induction step)
B(N, M ) ≤ B(N − 1, M ) + B(N − 1, M − 1).
1 of 2
Machine Learning (NTU, Fall 2008) instructor: Hsuan-Tien Lin
Proof. Consider the set S that achieves B(N, M ). We first project the vectors in S into the first N − 1 dimensions to get V = {vi}, where vi’s are unique. For instance, consider the set that achieves B(4, 2) above, after projecting we get:
v1 + + + v2 + + - v3 + - - v4 - + + We can then separate V to three disjoint subsets:
• A1: there is only (vi, +) in S, but no (vi, −). For instance, {v2, v3}.
• A2: there is only (vi, −) in S, but no (vi, +). For instance, {v4}.
• A3: both (vi, +) and (vi, −) are in S. For instance, {v1}.
By reorganizing the rows, we get
S =
A1 + A2 − A3 + A3 −
; V =
A1 A2 A3
.
Now let a = |A1| + |A2|, and b = |A3|. We see that1
|V | = a + b ; B(N, M ) = |S| = a + 2b. (1) 1. We first look at V :
V =
A1 A2
A3
.
If V is not M -incomplete, obviously S is also not M -incomplete—a contradition!.
Thus, V must be M -incomplete, and
|V | ≤ B(N − 1, M ). (2)
2. We now look at the subset (submatrix) S3 = A3 +
A3 −
If A3 is not (M − 1)-incomplete, then S3 (and hence S) is not M -incomplete—a contradition! Thus, A3 must be (M −1)-incomplete, and
b = |A3| ≤ B(N − 1, M − 1). (3)
By combining (1), (2), and (3), we get the desired result.
Lemma 2 (Sauer’s Lemma)
B(N, M ) ≤
M −1
X
m=0
C(N, m) ≤ NM −1+ 1.
Proof. The first inequality can be proved using mathematical induction (with Lemma 1).
The second inequality can be proved using mathematical induction, too.
1Here | · | means the size of the set, or (equivalently) the number of rows in the representing matrix.
2 of 2
