Robots-txt and sitemap command

Robots.txt is a command file, we create it for instruct to google search engine and other engine for crawl our website as per instructions. It isn't mean that our website posts or URL added in google console, but it is just a instruction for google BOT and other engine BOTS.

Sitemap.xml is a bunch of all URL of our website, sitemap allow search engine to find all your website pages, its a simple process for our website for indexing or detail about all web pages.

You need to copy below same text and past it into NOTEPAD and save it as a "Robots.txt" if your have word press blog, you can upload it in your WordPress blog, or you can paste it in your blog setting CRAWL options. command are as under.

 

User-agent: *

Allow: /

 

User-agent: Mediapartners-Google*

Allow: /

 

User-agent: Googlebot-Image

Allow: /

 

User-agent: Adsbot-Google

Allow: /

 

User-agent: Googlebot-Mobile

Allow: /

 

Sitemap: https://www.shahanionlinerozgar.com/atom.xml?redirect=false&start-index=1&max-results=500

 

#Note: Red Color (my website URL) you must change, add your website.

=======================================

or you can use the blow code also (different in only in sitemap)

User-agent: *

Allow: /

 

User-agent: Mediapartners-Google*

Allow: /

 

User-agent: Googlebot-Image

Allow: /

 

User-agent: Adsbot-Google

Allow: /

 

User-agent: Googlebot-Mobile

Allow: /

 

Sitemap: https://www.shahanionlinerozgar.com/sitemap.xml

==============================

If you have Disallow: /a, it blocks all URLs whose paths begin with /a. That could be /a.html, /a/b/c/hello, or /about.

Don't use "Disallow: / " option, it means it block all URL, or if you add any text or word that mean will be it block that URL or path begin with that word.

BACKLINKS TOOLS

This tool will give you information about your website backlinks, just put your website name and enter, its total free.