WH

This paper is from the SANS Institute Reading Room site. Reposting is not permitted without express written permission. Robots.txt Although this GIAC gold paper is not about search engine optimization, or SEO, this paper will explore a key element of SEO, the robots.txt file. This file is often neglected or misunderstood by HTML designers and web server administrators. The robots.txt file will impact your page rank rating with search engine providers. Configuration errors can result in web site revenue losses, not the kind of problem you want resting on your shoulders. A mis-configured robots.txt file can also lead to information disclosure, a foo... Copyright SANS Institut