Detect Googlebot device in sitecore for dynamic pages
Hello friends, You must heard about googlebot !!! Yes it’s Google’s web crawling bot (sometimes also called a "spider"). Is the process by which Googlebot discovers new and updated pages to be added to the Google index. It is very important for SEO prospective..
Sometime in the programming world we just focus on the performance of the webpage and try to write the best code and just skip the SEO Interest. Yes google crawling is very important factor to increase the rank of the page.
Sometime in dynamic content or dynamic page where the DOM is not fixed and content itself ( html) coming through ajax calls may create the problem to googlebot.
In the recent project we are using AJAX based HTML pages where content(DOM) updates through ajax calls. Then our SEO team came up with the problem that the google crawler (googlebot) is not crawling the page properly, and the rank of the page is going down.
So we researched a lot and came with the below solution.
1.) Create 2 version of the same page one for the Google and one for the end user.
- The First versions of the page is for google is the static page means normal postback not used more complex ajax calls. The whole purpose is to give the proper HTML(Dom) so that google can crawled it properly.
- The second version of the page is fully AJAX based to the rest of the world.
2.) Create the different device for googlebot to detect the googlebot agent and assign the different presentation to this device as below:
- Go to the device >
- Create one more device named Googlebot and assign the value as Googlebot in Browser agent field.
- Then go to the page item > Presentation and assign the different presentation (without ajax call layout) to googlebot device.
So when Google crawl the page, sitecore detect the agent “googlebot” and rendered the different presentation for google as we already defined in the presentation section.
So finally we served to the google with the desired HTML- DOM format that google supposed to crawled , so the page can be crawled properly.
Now my page ranking is up, and my SEO team is happy :)
We can also verify the “googlebot” agent presentation on the page through the browser, after manually setup the user-agent of the browser,
- Install google chrome extenstion “User-Agent Switcher for Chrome” https://chrome.google.com/webstore/detail/user-agent-switcher-for-c/djflhoibgkdhkhhcedjiklpkjnoahfmg
- Go to the icon and select agent as googlebot as below:
I hope it will help you.
Feel free to post your comment/query or any suggestion.
You can also connect me on twitter https://twitter.com/sitecore_ashish