what is robots.txt file in a blog? - TechinBit- Best Techy Content available on this Site

Latest

propellars ads

20191218

what is robots.txt file in a blog?

What is robots.txt ?

What is robots.txt ?


The robots.txt file provides a scanner that displays your website that may   or may not be requested on your site. This is used in an arbitrary manner. it is not a mechanism for retaining a webpage from Google. You must use noindex directives or password protect your page.

The robots.txt file is mainly used to indicate which parts of your website should be scanned by spiders or web crawlers. It can indicate different rules for different spiders.

Googlebot is an example of a spider. It is used by Google to crawl the Internet and record website information, so it knows how to rank various websites high in search results.


Using robots.txt on your website is the web standard. Spiders look for a robots.txt file in the host directory (or main folder) of your site. This text file is always called “robots.txt”.

Most major spiders follow the directions given in the robots.txt files, but vile spiders may not do this. The contents of the robot.txt files are publicly available. You can try to block unwanted spiders by editing the .htaccess file associated with your site.

It’s important that marketers check their robots.txt file to make sure search engines are encouraged to crawl important pages. If you ask search engines not to crawl your website, it will not appear in the search results.
You can also use the robots.txt file to show spiders where to find the site map of your site, which can make your content more accessible for detection.

You can also specify a scan delay or the number of seconds that robots must wait before collecting additional information. Some websites may need to use this option if bots absorb traffic and slow down the loading of your site for human visitors.

How to add robots.txt file in blogger

Robots.text code is here

# Blogger Sitemap
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://www.techinbit.com/atom.xml?redirect=false&start-index=1&max-results=500

Replace above my blog address  with  your blog address  and paste it into robot.txt file.