Few Deficiencies of Robots.txt and Robots Meta tags in Respect of

3.4.1 Some uncertainties with respect to new authorization function

This new function of the Robots.txt and Robots Meta tags has transferred them from ethnic advice to a set of powerful tools; the webmaster can rely on these tools to obtain a more secure guarantee. On the other hand, even this new function conferred by case law is so imprecise that there are a few uncertainties that need to be clarified.

3.4.1.1 About “[No]index” tag

The first point that should be noticed here is that the “[No]index” tag may give rise to some misunderstanding: as we have seen in section 2. Software robots are used in many different areas; some may only use robots to maintain links instead of making the index of the collected data. Therefore, a “[No]index” tag may cause some doubts as to what the webmaster’s real wish is. Does he want to exclude all robots or just exclude the robots used by search engines? Since the tag may lead to legal

consequences, we believe that it is safer to explicitly explain the wish expressed within the “Noindex” is only for excluding search engines’ robots or, more broadly speaking, excluding all robots with further indexing possibility. In other words, the absence of a “Noindex” tag can not definitely result in a conclusion that the webmaster grants a license to “all” robots because the “Noindex” tag does not explicitly represent the copyright owner’s wish in this situation.

3.4.1.2 About “[No]follow” tag

When the page containing a “[No]follow” tag and the pages followed by links are owned by the same person, any robot that disobeys this tag and copies the next page may infringement the copyrights of the page owner. However, sometimes, these two pages are not owned by the same person; on such an occasion, any robot that ignores the “Nofollow” tag and follows the link to access to other pages may not violate the copyright law, especially when the owner of next page dose not explicitly exclude software robots by employing any tag: because tags employed by any other but the copyright owner of the page are meaningless.

3.4.2 No appropriate tags to cover all copyright rights possibly infringed by software robots

In the above section, we have demonstrated that the complete access process of a robot can be divided into three steps: the accessing, the processing and the distributing step. In each step, the robots or the robot users could infringe the webmaster’s copyright without proper authorization. As we have seen above, the Robots.txt and Robots Meta tags are the best potential tools to be used for such a purpose as they are simple and widespread. In terms of the scope of authorization, it is the rights holders who have the right to decide the scope. But as we have shown in section 4.3, in case the page creators or the right holders do not have their own servers, the Robots.txt can not represent the real wish of the rights holders since the right holders have no right of access to the root directory. From this perspective, the Robots Meta tags, to speak more specifically, the “[No]index” tag, is the only tag which can be adopted to

represent the scope of the authorization, as another tag of the Robots Meta tags, the

“[No]follow” tag, is useless in terms of authorization as we mentioned in section 3.3.1.2.

Nevertheless, unfortunately, in terms of the original meaning of “[No]index” tag, it can only be used to exclude software robots with further indexing possibility, rather than excluding all types of robots (Koster, 1997) and, furthermore, it can not cover all three steps the software robots involving and all copyright rights possibly infringed in these three steps.

The rights referred in all three steps are different, as in the following table:

Table VIII. Possible Copyright Infringement caused by robots and the tags Step Possible Copyright Infringement

1 Accessing None

Infringement of the reproduction right, since the crawler user always need to store the data

2 Processing

Infringement of the adaptation right since the crawler user may modify the work

3 Distributing Infringement of the distribution right since the distribution may be unauthorized

It should be noticed that, in respect of the “accessing” step, that reproduction of software robots does not infringe the reproduction right of the right owners. The first and most obvious reason of this conclusion is: the contents of the website, at least in most circumstances, are authorized all viewers, including the software robots, on the Internet to access; the accessing here inevitably reproducing the contents of the website into the memory or the disk of the viewers’ computers and, as a result, the reproduction is lawful. Even though in some limited circumstances, the right owners try to exclude some viewers and software robots, the limitations and exceptions appearing in copyright laws still form possible executions of viewers and software robots (Sterling, 2003b). The last reason is that, even the software robots are excluded

by the “[No]index” tag, a robot would have to access at least part of a page before reading the instruction not to access it, and many robots probably download entire pages before processing any instructions contained in them. Accordingly, it is unreasonable to allege that the software robots infringe the owner’s copyright in this step. To sum up, with regard to the copyright authorization of software robots, the reproduction, adaptation rights in the “processing” step and the distribution rights in the “distributing” step are the rights we should concern. ,

However, the “[No]index” tag, which is only mapped to the second “processing”

step, at most can be used to express the wish of the authors in respect of reproduction and nothing of adaptation. In addition, as for distribution of the work in the third

“distributing” step, the wish expressed in this tag is ineffectual in resolving the potential infringements resulted from ambiguous authorization scope.

3.5 Adding Tags to Fully Express Copyright Authorization