Robots.txt is often a file that exists on the root Listing of each Site and may be used to instruct search engines like google and yahoo on which directories/data files of the website they can crawl and involve within their index. In line with Rubright, a far better solution is https://news.rafeeg.ae