How to Ensure Your Site is Being Crawled by Google and Bing

How to Ensure Your Site is Being Crawled by Google and Bing

I've worked on websites for months and realised they weren't even being crawled by Google. Even with highly SEO-optimised CMS like Ghost, you need to submit a sitemap to make sure you're on their radar.

(I know, it's crazy that they don't just find your webpage magically by default... but how would they? Nothing links to you... yet!)

Step 1: Get your sitemap URL

Ghost automatically generates an XML file, at /sitemap.xml. For example the one here is http://hooshmand.net/sitemap.xml.

If you're using WordPress, you need to ensure a sitemap is being generated by a plugin. Fortunately a number of free plugins do this for free. Yoast does it, as does Jetpack. Neither of these are necessary, but both make it way easier to manage your website.

Step 2: Submit it everywhere

Once you have your sitemap XML URL, you need to submit it to various search engines.

The main one is Google. To submit your XML sitemap to Google Search Console.

Bing is also important (we get search traffic from Bing on all our sites). To submit to Bing, go to Bing Webmaster Tools.

Step Infinity: Keep your site authenticated

For both of these, you need to authenticate your sites. While there are a number of options on how to do it, I hate playing around with header tags. Sometimes I forget why they're there, or I change some plugins and the tags disappear. So download the XML file, put it in your root folder and forget about it.