txt file is then parsed and can instruct the robot concerning which web pages will not be to get crawled. Being a online search engine crawler may perhaps continue to keep a cached copy of this file, it may on occasion crawl internet pages a webmaster doesn't desire to crawl. Webpages commonly p