Binarization and recognition of pattern blocks

After the captured modulated pattern image IP′′ are segmented into unit blocks, we try to recover the pattern blocks in pattern image IP by grouping every four mutually-connected unit blocks as a pattern block since the size of a pattern block is 2×2. The number of pattern blocks in the horizontal or vertical direction in IP'' is so just NT = NS/2. Subsequently, the moment-preserving thresholding technique [73] is

applied to each pattern block Ti''' to binarize it automatically. And the four unit blocks in each resulting pattern block Ti''' are denoted as Fi1', Fi2', Fi3', and Fi4',respectively.

(a) (b)

(c)

(d)

Figure 6.8. Block number identification. (a) Captured modulated pattern image IP′′.

(b) Gradient values of (a). (c) Average gradient values of pixels on candidate spitting lines for different NS. (d) Image division result according to determined number of unit blocks, NS = 64.

Next, how to classify each Ti''' as one of the eight possible code patterns, which we denote as BPk with k = 1 ~ 8, as shown in Figure 6.4 is the issue now. This is an eight-class pattern classification problem. To solve it, we use a minimum absolute distance classifier. Specifically, each possible code pattern BPk has four unit blocks, say denoted as Fk1'' through Fk4'', and the color of each Fkj'' is either black or white.

Hence, we may utilize the feature of blackness to describe Fkj''; that is, if the color of the unit block Fkj'' is black, then we take the blackness feature bfkj of Fkj'' to be “1”;

else, to be “0.” Next, we compute the real blackness feature bfij of each unit block Fij' in Ti''' by: code pattern with the minimum absolute distance is BP4 with ADi4 = 0.1425. So, the corresponding four bits bi1' through bi4' of the pattern block Ti''' are “0111.” Finally, we extract the original three message bits bi1, bi2, and bi3 as “011” based on bi1' through bi4' by the rule bij = bij' for j =1, 2, and 3 according to (28).

An example of results yielded by the above message extraction process is shown in Figure 6.9, where a captured modulated pattern image IP'' is shown in Figure 6.9(a), which, after being binarized, results in Figure 6.9(b); the result of code-pattern classification is shown in Figure 6.9(c); and the final extracted bit stream are shown in Figure 6.9(d). These results show that the proposed code-pattern classification scheme corresponding to the minimum absolute distance criterion works correctly for the purpose of embedded message extraction. More experimental results will be presented later to prove this statement.

Table 6.1. An example of code pattern recognition.

binary code pattern

Corresponding

4 bits Absolute distance ADik

0001 ADi1 = (0.1+0.87+0.78+0.12)/4 = 0.4675 0010 ADi2 = (0.1+0.87+0.22+0.88)/4 = 0.5175 0100 ADi3 = (0.1+0.13+0.78+0.88)/4 = 0.4725 0111 ADi4 = (0.1+0.13+0.22+0.12)/4 = 0.1425 1000 ADi5 = (0.9+0.87+0.78+0.88)/4 = 0.8575 1011 ADi6 = (0.9+0.87+0.22+0.12)/4 = 0.5275 1101 ADi7 = (0.9+0.13+0.78+0.12)/4 = 0.4825 1110 ADi8 = (0.9+0.13+0.22+0.88)/4 = 0.5325

6.4.4 Message extraction algorithm Algorithm 6.2. Message extraction.

Input: a captured version Id of a message-rich code image.

Output: the message M embedded originally in Id. Steps:

Stage 1 － localizing the message-rich code image.

Step 1. Find the largest non-white quadrangle Q in Id by the Hough transform and polygonal approximation.

Stage 2 － correcting geometric distortion.

Step 2. Perform an inverse perspective transform on Q to correct the perspective distortion and take the Y-component of Q as the captured modulated pattern image IP′′.

Stage 3 － identifying pattern blocks in the code image.

Step 3. Compute the gradient value gxy of each pixel Rxy in IP′′ according to (24).

(a) (b)

110101010……

(c) (d)

Figure 6.9. Binarization and code-pattern recognition. (a) Captured modulated pattern image. (b) Binarization result of (a). (c) Result of code-pattern recognition of (b). (d) Extracted message.

Step 4. For each possible value nj of NS, compute the average gradient values AGⁿj of the pixels on each candidate spitting line according to (26).

Step 5. Select the value nj yielding the largest AGⁿj, denoted as AGo, and those nj’s yielding AGⁿj close to AGo; pick the largest nj from all the selected nj’s for use as the desired number NS of blocks of IP′′ in the horizontal or vertical direction; and divide IP'' accordingly into unit blocks.

Stage 4 － binarizing the pattern blocks to extract the message.

Step 6. Group every four mutually-connected unit blocks and denote them as Fi1' through Fi4' to form a pattern block Ti′′' in IP′′.

Step 7. Extract three bits from each pattern block Ti''' by the following steps.

(a) For each unit block Fij' in Ti''', compute its blackness feature bfij according to (30).

(b) Computing the absolute distance ADik of Ti''' to each of the eight possible code patterns BPk shown in Figure 6.4 according to (31).

(c) Select the code pattern BPm with the minimum absolute distance and take the corresponding four bits of BPm as the recognized four bits bi1 through bi4' of Ti'''.

(d) Extract the original three message bits bi1, bi2, bi3 as bi1', bi2', bi3', respectively, according to (28).

Step 8. Concatenate the extracted bits into a bit stream B and transform reversely B to get the embedded message M.

6.5 Experimental Results

The proposed method was implemented on a 3.0GHz PC with a Core i7 CPU and 8G RAM using the language Microsoft C#.NET, and generated message-rich code images were captured with an iPhone 4S and analyzed to extract the embedded messages in a series of experiments. Corresponding statistics were plotted as well to show the accuracy of the extracted messages using different parameters including: (1) the contrast threshold  for the minimum difference between the representative values r1′ and r2′; and (2) the number of unit blocks NS used in the horizontal or vertical direction of the created pattern image. Figures 6.10(a), 6.10(c), and 6.10(e) show three test target images used in the experiments. The corresponding message-rich code images generated with parameters NS = 128 and  = 40 are shown in Figures 6.10(b), 6.10(d), and 6.10(f), respectively. These images were all printed to be of the same size of 127127 mm.

One of the parameters that influence the accuracy of the extracted message is the contrast threshold value  for the minimum difference between the two representative values r1′ and r2′. If  is too small, r1′ and r2′ will be too close so that the extracted message might be wrong. Figure 6.12(a) illustrates the accuracy rates of message extraction with  = 0, 20, 40, and 60, which shows that the larger the value of , the higher the accuracy rate of the extracted message; when  > 40, an accuracy of 99.8%

is reached; and when  > 60, an accuracy of 100% is reached. Figure 6.12(b) shows that the larger the value of , the larger the RMSE of the resulting message-rich code image with respect to the target image. So there is a tradeoff between achieving higher message extraction accuracy and obtaining better visual quality in the generated code

image. Figure 6.11 shows some code images created with different contrast threshold values of , where the target image is Figure 6.10(e) and NS = 64. As can be seen, Figure 6.11(a) has the best visual appearance when compared with the others, but has the lowest message extraction accuracy for only 85.60%, because the two representative values are too close (so that the colors of most regions in Figure 6.11(a) look like the same).

(a) (b)

(c) (d)

(e) (f)

Figure 6.10. Created message-rich code images. (a), (c), and (e) Target images. (b), (d) and (f) Resulting message-rich code images with NS = 128 and  = 40.

(a) (b)

(c) (d)

Figure 6.11. Created message-rich code images with different contrast threshold values of , where NS = 64. (a) Resulting message-rich code image with RMSE = 66.35 and accuracy rate = 85.60%, where  = 0. (b) Resulting code image with RMSE = 66.57 and accuracy rate = 98.97%, where  = 20. (c) Resulting code images with RMSE = 68.47 and accuracy rate = 100%, where  = 40. (b) Resulting images with RMSE = 72.27 and accuracy rate = 100%, where  = 60.

Another parameter that influences the message extraction accuracy is the number NS of unit blocks in the horizontal or vertical direction in the created pattern or code image. The larger the value of NS, the larger the message embedding capacity of the created code image, yet the smaller the size of the unit block and so the lower the message extraction accuracy. This can be seen from Figure 6.12(c), where when NS = 16, the accuracy of 100% is reached; when NS = 64, the accuracy of 99.76% is reached; and when NS = 128, the lower accuracy of 96.31% is yielded. Figure 6.13 shows some message-rich code images generated with different values of NS with Figure 6.10(a) as the target image and the contrast threshold  = 40. As can be seen, when NS is larger, the visual appearance of the created image is better with a smaller RMSE, but the message extraction accuracy is lower. Specifically, the accuracy of Figure 6.13 (d) is 99.11%, instead of 100% which is reached by the other three cases.

(a) (b)

(c) (d)

Figure 6.12. Plots of trends of results using various parameters. (a) Accuracy rates of extracted messages with different contrast threshold values , with #unit blocks NS = 32. (b) RMSE values of created message-rich code images with respect to target images for different contrast threshold values of , with #unit blocks NS = 32. (c) Accuracy rates of extracted messages with different #unit blocks NS with contrast threshold  = 40. (d) RMSE values of created message-rich code images with respect to target images with different #unit blocks NS and contrast threshold  = 40.

Table 6.2 shows a comparison of the results of the proposed method via message-rich code images in this chapter and those of the method via message-rich character image in Chapter 5 with the target images as shown in Figure 6.10 against different numbers NS of unit blocks in the horizontal or vertical direction in the created pattern or message images. As can be seen from the table, the proposed method yields higher message extraction accuracy than the method via message-rich character image in Chapter 5, e.g., when NS = 32, the message extraction accuracy yielded by the proposed method reaches 99.80% while that yielded by the method via message-rich character image in Chapter 5 is only 88.28%. Moreover, when NS = 64, the message extraction accuracy of 99.76% yielded by the proposed method is much higher than that yielded by the method via message-rich character image in Chapter 5, which is only 34.70%.

(a) (b)

(c) (d)

Figure 6.13. Created message-rich code images with different #unit blocks NS, where contrast threshold value  = 40. (a) Resulting message-rich code image with RMSE = 47.66 and accuracy rate = 100%, where NS = 16. (b) Resulting message-rich code image with RMSE = 44.63 and accuracy rate

= 100%, where NS = 32. (c) Resulting message-rich code image with RMSE = 42.05 and accuracy rate = 100.00%, where NS = 64. (d) Resulting message-rich code image with RMSE = 39.43 and accuracy rate = 99.11%, where NS = 128.

Also, Figure 6.14 shows the resulting binarized captured message-rich images of the proposed method via message-rich code image and the method via message-rich character image in Chapter 5, where the value of NS is 32 in Figures 6.14(a) and 6.14(b) and 64 in Figures 6.14(c) and 6.14(d). As can be seen from Figure 6.14(a), with NS = 32 the characters in the binarized captured message-rich character image created by the method in Chapter 5 are still clear enough so that the message extraction accuracy yielded with Figure 6.14(a) as the input is still high, reaching 98.61%. However, with NS = 64, as seen from Figure 6.14(c), the characters in the binarized captured message-rich character image created by the method in Chapter 5

become undistinguishable so that the message extraction accuracy yielded with Figure 6.14(c) as the input becomes worse, only 41.25%. Furthermore, as seen from Figures 6.14(b) and 6.14(d), the pattern blocks in the binarized captured message-rich code images created by the proposed method are both clear enough so that the message extraction accuracy rates yielded by them are both still high, reaching 99.80% and 99.76%, respectively.

In addition, we compare the times consumed by the code-pattern recognition steps in the proposed method and the method in Chapter 5. As can be seen from Table 6.2, the recognition time used by the proposed method in this chapter is much less than that used by the method in Chapter 5. This is owing to the time-consuming OCR operation conducted by the method in Chapter 5 on every character image, which computes the similarity of the character image with each possible character image in the database and selects the most similar one as the recognition result. In contrast, the proposed method only needs to recognize each pattern block as coming from one of eight possible code-pattern classes by computing the absolute distances of the pattern block to the eight classes and selecting the one with the minimum absolute difference.

Table 6.2. Comparison of results of proposed method in this chapter and method in

As a summary, the proposed method has the following merits with respect to the method in Chapter 5: (1) the yielded message-rich code image has a better visual appearance since a larger number NS of unit blocks can be utilized in the proposed method; (2) the message extraction accuracy is higher since much less details are contained in a unit block of the proposed method; (3) the message extraction speed is higher since classification of only eight classes need be conducted to extract the corresponding four bits of each binarized pattern block.

(a) (b)

(c) (d)

Figure 6.14. Binarized captured message-rich images created by method in Chapter 5 and proposed method in this chapter and respective message extraction accuracy rates, where the target image of these resulting images is Figure 6.10(c). (a) Binarized image by method in Chapter 5 with NS = 32 and accuracy rate = 98.61%. (b) Binarized image by proposed method in this chapter with NS = 32 and accuracy rate = 99.80%. (c) Binarized image by method in Chapter 5 with NS = 64 and accuracy rate = 41.25%. (d) Binarized image by proposed method in this chapter with NS = 64 and accuracy rate = 99.76%.

Furthermore, as shown in Figure 6.4, only eight binary code patterns are utilized by the proposed bit expansion scheme. However, the possible number of all code patterns is 16 since the possible number of combinations of the four expanded bits is 2⁴ = 16. Also, only two of the possible 16 code patterns will cause the undistinguishable problem as illustrated in Figure 6.3. Hence, 14 code patterns can be utilized in the bit expansion scheme. For this, as shown in Figure 6.15, we conducted an experiment to test another bit expansion scheme, in which the first three bits expansion scheme and the new bit expansion scheme just described, respectively. And Figures 6.16(c) and 6.16(d) show the created message-rich code images corresponding to Figures 6.16(a) and 6.16(b), respectively, where Figure 6.16(d) shows that the created message-rich code image by using the new bit expansion

Figure 6.15. Performing another bit expansion scheme on every three message bits to yield 14 binary code patterns represented by pattern blocks.

(a) (b)

(c) (d)

Figure 6.16. Results yielded by using two different bit expansion schemes with NS = 64 and  = 20. (a) Pattern image yielded by the original bit expansion scheme. (b) Pattern image yielded by the new bit expansion scheme. (d) Message-rich code image yielded by the original bit expansion scheme with RMSE = 55.97. (d) Message-rich code image yielded by the new bit expansion scheme with RMSE = 55.44.

6.6 Summary

A new data hiding technique via message-rich code image for applications of automatic identification and data capture has been proposed, which is created from a target image for use as a carrier of a given message. The artistic favor of the target image is kept in the created image, achieving pervasive communication. Skillful techniques of code pattern design, unit block segmentation, pattern block classification, etc. have been proposed for message data embedding and extraction.

Comparing with other automatic identification and data capture techniques like the use of barcodes and data hiding, automatic identification and data capture using the

proposed message-rich code image has several merits: (1) the image has the visual appearance of any pre-selected target image (this is not the case for the case of using barcodes [17]-[19]); (2) the proposed method can endure more distortions in acquired versions of the code image like perspective transformation, noise, screen blurring, etc.

(this is not the case for data hiding [4]-[19]); (3) the message can be extracted from an image captured by a mobile device (this is not the case for data hiding [4]-[19]). Also, the proposed method in this chapter has following additional merits when compared with the method in Chapter 5: (1) the yielded message-rich code image has a better visual appearance; (2) the message data extraction accuracy is higher; (3) the data extraction speed is higher. Experimental results show the feasibility of the proposed method. Further studies may be directed to applying error-correction techniques to the result of code-pattern classification in order to increase the resulting message extraction rate, such as using Reed-Solomon codes [74], and designing more types of message-rich images or extending the idea to videos for different applications.

Chapter 7

Conclusions and Suggestions for Future Studies

In this dissertation study, the new concept of message-rich multimedia is proposed, which enriches human-environment interaction and advances pervasive communication. Various data hiding methods have been designed for creating message-rich multimedia for different pervasive communication applications as described specifically in the following.

(1) For images with large data volumes, a large-volume data hiding technique to hide these secret images with large data volumes into any target images of the same size has been proposed.

(2) For encrypted images, also proposed is a data hiding technique based on the techniques of double image encryption and spatial correlation comparison to improve the performance of two previous methods when dealing with flat cover images.

(3) A text data hiding technique via creations of fake collaboratively-written documents on collaborative writing platforms has been proposed. With this camouflage, an attacker will take the stego-document as a normal collaborative writing work and will not be expected to realize the existence of the hidden message.

(4) Two data hiding techniques via message-rich character images and message-rich code images for automatic identification and data capture applications have been proposed to realize the innovative idea of pervasive communication on hard copies of images on papers or displays of monitors or TVs. The created image is visually similar to the target image with the function similar to those of barcodes or QR codes, achieving the effect of pervasive communication.

In the following, conclusions of each method and suggestions for future researches are given.

7.1 Conclusions

(1) A new data hiding method has been proposed, which not only can create meaningful mosaic images but also can transform a secret image into a mosaic

one with the same data size for use as a camouflage of the secret image. By the use of proper pixel color transformations as well as a skillful scheme for handling overflows and underflows in the converted values of the pixels’ colors, secret-fragment-visible mosaic images with very high visual similarities to arbitrarily-selected target images can be created with no need of a target image database. Also, the original secret images can be recovered nearly losslessly from the created mosaic images. Future works may be directed to applying the proposed method to images of color models other than the RGB one.

(2) A new data hiding method based on double image encryptions and refined spatial correlation comparison on encrypted images has been proposed, which solves a problem encountered in the two existing methods [55]-[56] when dealing with flat cover images. This problem comes from the way of flipping the three LSBs of each pixel in part of each block in an encrypted image to embed a message bit.

The proposed method improves this by encrypting the four LSBs of each pixel of every block instead of flipping three of them to embed a bit. Also, a refined side-match scheme utilizing the spatial correlations of both recovered and unrecovered blocks has been proposed to decrease the bit-extraction error rate, in contrast with Hong et al. [56] which utilizes only those of recovered blocks.

Future works may be directed to applying the proposed method for various information hiding purposes.

(3) A new data hiding method via collaboratively-written articles with simulated revision history records on collaborative writing platforms has been proposed.

An input secret message is embedded in the revision history of the resulting stego-document through a simulated collaborative writing process with multiple virtual authors. With this camouflage, people will take the stego-document as a normal collaborative writing work and will not be expected to realize the existence of the hidden message. To generate simulated revisions more realistically, a collaborative writing database was mined from Wikipedia, and the Huffman coding technique was used to encode the mined word sequences in the

An input secret message is embedded in the revision history of the resulting stego-document through a simulated collaborative writing process with multiple virtual authors. With this camouflage, people will take the stego-document as a normal collaborative writing work and will not be expected to realize the existence of the hidden message. To generate simulated revisions more realistically, a collaborative writing database was mined from Wikipedia, and the Huffman coding technique was used to encode the mined word sequences in the