CHAPTER 3 THE PROPOSED METHOD
3.1 Preprocessing
3.1.2 Noise removal
Noises of an image could cause disturbing values in the print. To eliminate noises occurring from imprint and scanning process, images are divided into blocks, each of size 16*16. For each block B, we calculate its mean and variance by
∑∑
= = is set to white color value (that is, the gray level of each pixel in the block is 255).Otherwise, keep the original contents.
After block‐based noise removing, we scan all pixels in the shoeprint image.
Pixel values greater than 200 are all set to 255 (pure white).
3.1.4 Edge Detection
Fundamentally, an edge is a “local” concept where as a region boundary [10].
Edges of an image characterize boundaries and are therefore a crucial issue in image processing. Edge detecting an image significantly reduces the amount of data and filters out useless information while preserving the important structural properties in an image.
We use Sobel operator to extract the edges for the shoeprints. The approximation of the magnitude of the gradient is carried out by using absolute values:
2
1 d
d y
x G G G
G
f = + + +
∇ (4)
where , , , and are the magnitude of the gradients of horizontal, vertical, and two diagonal directions.
Gx Gy Gd1 Gd2
The extracted edge image is used for further processing. The mask shown in Fig.
5 represents the contents of the operator.
‐1 ‐2 ‐1
0 0 0
1 2 1
‐1 0 1
‐2 0 2
‐1 0 1
0 1 2
‐1 0 1
‐2 ‐1 0
‐2 ‐1 0
‐1 0 1
0 1 2
Fig. 5. Four Sobel operators Gx, Gy, Gd1, and Gd2 for detecting vertical, horizontal, and two diagonal edges.
3.1.5 Bi‐level Thresholding
After performing the stages described above, the obtained image is a 256‐graylevel image. For simplicity of implementation, a thresholding method is designed to extract high spectral pixels from the shoeprint image.
Let
T be an image of size of 256 level )
(i
H be the histogram of the image, i=0,...,255
∑
== 255 ( )
j i
j H i
CE be the cumulative energy of H(i) from j to 255 The threshold value was determined by l
{
j CE CE r}
l =max | j / 0 > , where r is a predefined ration
Pixels with intensity greater than are considered as background points and are all set to 255, others are all set to 0. The entire image is of only two values then, pure black and pure white.
l
3.1.6 Principal Component Transform
The orientations of the shoeprint images are of multiple variations. These rotations bring about a problem for feature extraction due to inconsistency of shoeprint images. To solve this problem, we make use of principal components for variable orientations. Aligning the image with its principal eigenvectors provides a reliable mean for removing the effects of rotation.
Let
L( ji, ) be the intensity value of a pixel ( ji, ) of the input image T
{
( , )|( , )∈ (, )=255}
= i j i j T and L i j
X be a set of points with white
color X
K = be the number of points in the set X
1. Approximate the mean vector by
2. Compute the covariance matrix for the input image as follows:
∑
= 3. The matrix A is created by having all rows formed from the eigenvectors ofordering eigenvectors according to decreasing eigenvalues.
Cx
Then, the matrix A is the transformation matrix to project the points in X into points denoted by Y using the expression
yk = A
(
xk −mx)
(7) Shoeprint images transformed with transformation matrix A might differ in orientations in 180 degrees. Properties of normal human thenar reveal that the front‐half part of shoe has bigger size in width comparing to the rear one. Therefore, after applying the transform, print was further examined to see if it matches the rule.Otherwise, rotate the image by 180 degrees.
Fig. 6(a) is the original shoeprint image. Fig. 6(b) shows the shoeprint image with its corresponding eigenvectors drawn with dashed lines. Fig. 6(c) is the one transformed with respect to the eigenvectors of the largest eigenvalue.
3.2 Feature Extraction
The most obvious pattern in shoeprints is directionality. Fig. 7(a) shows a shoeprint pattern with simple vertical lines. Fig. 7(b) shows a shoeprint pattern with circular shapes. The shoeprint image shown in Fig. 7(c) presents a pattern of various geometries. From simple line direction to complex geometric analogy, these drafts all represent a unique model of its own pattern. Therefore, a feature extraction method based on directionality is designed. To estimate the energy of each direction in the whole image, three different approaches are provided. The first one uses co‐occurrence matrices to calculate the relation between pixels. The second uses a region based direction mask to compute the local ridge orientation for the entire image. The third takes global directionality, enhanced Fourier transform extends from normal Fourier transform is employed. Taking only global characteristics into consideration, however, might lose some local information. For the sake of this flaw, enhanced Fourier transform is also applied to local areas of the shoeprint to extract local information.
Fig. 7. An example of shoeprint patterns (a) Vertical lines. (b) Circular shapes.
(a) (b) (c)
3.2.1 Co‐occurrence Matrices
Co‐occurrence matrix, which was first introduced by Haralick in 1979 [11], is defined over an image to be the distribution value of co‐occurring values at a given offset. Here we will introduce the definition of co‐occurrence matrices and its characteristics for an image.
Let
T be an image of 2 color level (pure black and pure white) )
, (p q
L be the intensity value of pixel (p,q)in image T )
, ( x y
v= Δ Δ be an offset vector
The co‐occurrence matrix Cv(i,j) is the 2×2 matrix defined as follows:
∑∑
= = ⎩⎨⎧ = +Δ +Δ == n
p m
q
v otherwise
j respectively intensity i and j with displacement offset v.
It can be seen that co‐occurrence matrix collects the second‐order statistics of image T with the characteristics that the main diagonal of the matrix with offset is as closer to the histogram of the image as is related to the corresponding direction [12]. Based on this observation, we propose a measure as follows:
H be the histogram of the image, i=0,255
The discrepancy of co‐occurrence matrix C, D(v) is defined by
(
(0) (0,0) (255) (255,255))
value to the direction , where denotes the normalized value in of. Then
v D' v( ) [0,1]
) (v
D CM(v), v∈{(i,j)|i=−4,−3,3,4 and j =−4,−3,3,4} is selected as a feature vector for pattern matching. Note that the normalization for is defined by
) (v D
min )
where and denote the minimum value and the maximum value of all with
Fig. 8(b) shows the value of the Fig. 8(a). Each square with coordinate relative to the image center represents the offset . Pure black means the strongest response, while pure white means the slightest response. Solidus regions are unconsidered offset values. Note that offsets with
) shoeprint image. (b) An gray‐level representation of CM .
(a) (b)
Co‐occurrence matrix gives the behavior between pairs of pixels according to the offsets. However, due to the noises in shoeprints, the relevance measure of co‐occurrence matrix might be incorrect. To obtain relevance of directionality from another view, directional masks are proposed. 12 masks, each of size 7*7, refer to predefined directions 0∘, 15∘, 30∘, 45∘, 60∘, 75∘, 90∘, 105∘, 120∘, 135∘, 150∘, 165∘, as shown in Fig. 9 are used.
The steps for ridge direction retrieval are
2. Sliding the kth mask through the image, compute the convolution for each 3. Then the energy for each direction is
12
is selected as a feature vector for similarity measurement, where denotes the normalized value in of .
12
Table 1 shows an energy result using directional mask corresponding to the input image in Fig. 8(a). From the table, we can see that the strongest response lies in the 00 degree from the horizontal lines of the input image.
Fig. 9. Twelve directional masks of 0∘, 15∘, 30∘, 45∘, 60∘, 75∘, 90∘, 105∘, 120∘, 135∘, 150∘, 165∘.
The Energy Result of Directional Matrix Corresponding to the Image in Fig. 8 (a)
TABLE 1
Angle (degree)
0∘ 15∘ 30∘ 45∘ 60∘ 75∘
Energy 1 0.373 0.078 0.314 0.039 0
Angle (degree)
90∘ 105∘ 120∘ 135∘ 150∘ 165∘
Energy 0.255 0.137 0 0.039 0 0
3.2.3 Global Fourier Transform
Fourier Transform is an operator that maps a function to another function in terms of sin and cosine basis functions. More specifically, for an image, the Fourier Transform represents the image as a summation of various sine‐like or cosine‐like images. Due to the property of the Fourier Transform, it is widely employed for processing image to analyze the frequencies contained in an image. In this section, we first make use of the Fourier Transform to evaluate the strength of directionality over the image. Then, the masking operation is performed to remove the high‐frequency and low‐frequency noises. After applying Fourier transform on the target image, we perform the Fourier transform on the Fourier spectrum again to get the enhanced Fourier Spectrum. The enhanced Fourier spectrum is more prominent than the original spectrum. It reinforces the peaks with the same periods and contributes to the same frequency while for those which are not periodic ones, they do not contribute to the same frequency [13].
This phenomenon relatively enhances those peaks and eliminates the non‐periodic noises in an image. Fig. 10(a) shows the Fourier spectrum of the target image in Fig. 8(a). Fig. 10(b) is the Fourier spectrum image obtained by applying Fourier transform on Fig. 10(a).
Low frequencies correspond to the slowly varying components of an image. For prints, these low‐frequency components are of enormous spectrum value and might result in noisy information. To remove these needless components, a mask for removing low‐frequency elements is applied. By masking, only frequencies that have Euclidean distance from the zero spatial‐frequency point greater than the predefined threshold are preserved as candidate features.
High‐frequency components which occur owing to outsole scuffs and nicks of the shoe may result in outstanding values over the underlying shoeprint patter [7].
Fig. 11 represents these high frequency noises. Similarly, a mask for removing the high‐frequency components is applied. Only frequencies that have Euclidean distance less than a predetermined threshold from the zero spatial‐frequency point are preserved as candidate features. Fig. 10(c) shows the spectrum with both high‐frequency and low‐frequency masks applied on Fig. 10(b).
Fig. 10. An example of the enhanced Fourier spectrum and the masking operation. (a) The Fourier spectrum of Fig. 8(a). (b) The enhanced Fourier spectrum of (a). (c) The spectrum with low‐frequency and high‐frequency masks apply on (b)
(a) (b) (c)
Following the procedure, the enhanced Fourier spectrum image that goes through low‐frequency and high‐frequency masks is denoted by F’ of size . Base on F’, we define a feature vector
n
Fig. 11. An example of the high‐frequency components. (a) A print with full integrity. (b) The same print with (a) with scuffs and nicks. (c) The enhanced Fourier spectrum of (a). (d) The enhanced Fourier spectrum of (b).
All three methods mentioned above take the full prints into consideration.
Nevertheless, features extracted merely from the full image might lose details in local areas. To compensate this shortage, the local Fourier Transform based on the frequency domain is used.
The processed shoeprint image is divided into 15 equal regions as shown in Fig.
12(a). Following that, Fourier transform is performed on each region twice to get the enhanced Fourier spectrum of the area.
Low‐frequency mask and High‐frequency mask are also applied on each region k to get the masked Fourier spectrum . The remaining values, which are denoted by
high-frequency nor in low-frequency areas}, are selected as a feature vector.
Fig. 12(b) shows a result after applying local Fourier transform on Fig. 12(a).
Feature vectors extracted from the operations described above, including CM, DM, GF, and LF, are taken for pattern matching at this stage.
In order to obtain the most likely shoeprint image that compares to the reference image, we use sum‐of‐absolute‐difference (SAD) as the similarity measure.
The SAD is the most primitive technique for matching. Let images T and T" be the database image and reference image respectively, the similarity between the selected feature vector of the image pair is calculated by
where denotes the element of a feature vector , and .
F∈ K denotes for the number of elements in the feature vector. F is a feature vector belongs to T . is a feature vector belongs to
' F '
T .
from the database is corresponding to the examined shoeprint. Prints of databases are then sorted according to the similarity measure with the most similar image shown at the front of the results. Scores are given to the sorted categories of database from 1 to 315.
Combination of multiple feature vectors is conducted by summing up the given scores of each category from every individual feature vector. The lower the sum of the score means the more similar the category is comparing to the reference image.
Therefore, categories are sorted according to the total scores by increasing order.
3.4 K‐means Algorithm for Database Creation
K‐means algorithm is a common algorithm for clustering. More specifically, it is an algorithm to group the objects based on attributes into K number of groups.
Hence, for each of the print sets of 10 shoeprint images, we use the K‐means algorithm to select 3 representative images as the database category.
Feature vectors CM , DM , GF , and LF extracted using the methods described above are taken as attributes f ={CM,DM,GF,LF}, and used in the K‐means algorithm to form three groups.
The algorithm performs the following steps until convergence:
1. Determine the centroid coordinate of each group.
In the first iteration, shoeprints are assigned randomly into 3 groups.
2. Determine the distance of each shoeprint to the centroid of each group.
The distance is calculated by Euclidean distance
∑
=−
= K
i
i
i c
f dist
1
)2
(
Where denotes the attribute of the shoeprint image, and denotes the attribute of the centroid.
fi ith ci
ith
3. Group the shoeprints based on the minimum distance.
Each shoeprint is assigned to the group with the minimum distance. If no new assignment is performed, then the grouping procedure ends. Otherwise, repeat step 1 to step 3.
After applying the K‐means algorithm, we will have three groups of images for each individual shoeprint image set. From each group, the image with the closest disatance to the centroid of the group is taken as the representative image. The selected three images are then taken as database images. Seven of others are taken as query images.
Chapter 4
Experimental Results
Experiments are conducted to evaluate the performance of the proposed method. 1050 shoeprint images collected from 105 distinct shoes a re used to test our algorithm. 315 out of 1050 prints are taken as the database images according to the K‐means process described above. The remaining 735 shoeprint images become the query images. Fig. 13 shows 105 distinct shoeprints. Every print in the database is examined in turn for comparison with the input shoeprint. The similarity measures calculated based on each feature vector are then used to sort the shoeprint images in the database from the most similar print to the least similar one.
The method is designed to find similar shoeprints and sort the corresponding categories of database in response to a reference image. With higher performance, the result is expected to present fewer nonmatching shoeprint categories before a matching category. In view of this, “Average Match Score (AMS)“ is used to evaluate the performance of the results.
The “Average Match Score” measures the average percentage of the database categories before a correct match is delivered. In our experiments, each shoe pattern contains 3 corresponding shoeprint images in the database which are gathered from an identical right shoe. Hence, the performance is determined by counting the number of nonmatching categories until hitting the correct one. Then, the process continues to find the second and the third category that correctly matches to the reference shoeprint image with their searching cost.
Table 2 displays an example of query. With respect to the query shoeprint, each row shows the top 5 query results according to different feature vectors. In the first row, taking the feature vector of co‐occurrence matrix, the correct matching is
delivered in the 3rd, the 4th, and the 5th position. While taking all features into consideration, the correct matching is delivered in the 1st, the 2nd, and the 4th position.
‐
Query Results
1st 2nd 3rd 4th 5th
Co‐occurrence
Direction
Global Fourier
Local Fourier
Query shoeprint
All
TABLE 2
An Example of Query Results
Each feature vector is conducted independently first, and then combined together for further assessment. The best performance is the one with the combination of all proposed features vectors. Table 3 shows the results of AMS of the method in [7], which utilizes Fourier transform for matching. From the first column of
images should be examined in average to get one correct match. While for getting all three matching shoeprints, 13.88% of shoeprint database images should be examined in average. Results of the proposed method for different feature vectors are shown in Table 4. From the table we can see that 1.29%, 4.71%, and 11.59% of shoeprint database images should be examined in average respectively before the 1st, the 2nd, and the 3rd correct matching with the feature vector of co‐occurrence matrix.
While taking all features into consideration, 0.38%, 1.19%, and 2.75% of the database images are examined in average before retrieving the 1st, the 2nd, and the 3rd correct shoeprint. The proposed method is much more accurate than the method provided in [7].
First Data Second Data Third Data Average
AMS (%) 3.51 11.75 26.39 13.88
TABLE 3
Average Match Scores (%) for the Chazal and Flynn’s Method on the Proposed Shoeprint Images with Image Resolution of 512x512
Features First Data Second Data Third Data Average
Co‐occurrence 1.29 4.71 11.59 5.86
Direction 1.61 5.68 13.71 7
Global Fourier 0.64 3.09 7.39 3.71
Local Fourier 0.92 3.03 7.67 3.87
All Features 0.38 1.19 2.75 1.44
TABLE 4
Average Match Scores (%) for the Proposed Method on 735 Shoeprint Images from 105 Individual Shoes, Database of 315 Shoeprint Images with Different Features
Chapter 5 Conclusion
The study proposed a novel method for automatically recognizing the shoeprint image using the properties of directionality. Firstly, a series of preprocessing processes are applied to eliminate distortions in the print including rotations, translations, and noises. Four feature extraction methods based on the directionality are then performed on the preprocessed image. In the end, a similarity measure using SAD is calculated in response to the reference image.
Based on the algorithm, a system can be built to help forensic scientists seeking for the model of a shoe from a shoeprint image. Experiments designed for assessment of performance showed the accuracy and efficiency of the proposed method. It can accelerate human observer identifying the shoeprint pattern with respect to the reference image. However, shoeprints used in the study were obtained under human controlled circumstances, while those acquired at crime scene were of lower quality and of desperate distortions. These shoeprints were mostly partial prints and tough for matching. Improvements may be achieved by employing new de‐noise methods in preprocessing and delicately designed the database images from acquired shoeprints.
References
[1] A. Girod, “Computer Classification of the Shoeprint of Burglar Soles,” Forensic Science Int’l, vol. 82, pp. 59‐65, 1996.
[2] A. Girod, “Shoeprints ‐ Coherent Exploitation and Management,” European Meeting for Shoeprint/Toolmark Examiners, The Netherlands, 1997.
[3] W.J. Bodziak, “Footwear Impression Evidence Detection,” Recovery and Examination, 2nd ed. CRC Press, 2000.
[4] N. Sawyer, “’SHOE‐FIT’ A Computerized Shoe Print Database,” Proc. European Convention on Security and Detection, pp. 86‐89, May 1995.
[5] W. Ashley, “What Shoe Was That? The Use of Computerized Image Database to Assist in Identification,” Forensic Science Int’l, vol. 82, pp. 67‐79, 1996.
[6] A. Bouridance, A. Alexander, M. Nibouche, and D. Crookes, “Application of Fractals to the Detection and Classification of Shoeprints,” Proc. 2000 Int’l Conf. Image Processing, vol. 1, pp. 474‐477, 2000.
[7] P. de Chazal, J. Flynn, and R. B. Reilly, “Automated Processing of Shoeprint Images Based on the Fourier Tranform for Use in Forensic Science,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 27, no. 3, pp. 341‐350, March 2005.
[8] K. Jack, “Video Demystified,” 5th ed. Elsevier, 2007
[9] R. Fisher, S. Perkins, A. Walker, and E. Wolfart. “Image Processing Learning Resources,” http://homepages.inf.ed.ac.uk/rbf/HIPR2/gsmooth.htm.
[10] R. C. Gonzalez, R. E. Woods, “Digital Image Processing,” 2nd ed. Addison Wesley, 2002.
[11] R. M. Haralick, “Statistical and Structural Approach to Texture,” Proc. IEEE, vol.
67, no. 5, pp. 786‐804, 1979.
[12] A. K. Julesz, “Dialogues on Perception,” MIT Press, Cambridge MA, 1995.
[13] K. L. Lee, L. H. Chen, “A New Method for Coarse Classification of Textures and Class Weight Estimation for Texture Retrieval,” Pattern Recognition and Image Analysis, vol. 12, no. 4, pp. 400‐410, 2002.