How to Track Search Bots/Spiders with Google Analytics.
I went accross a french blog post some weeks ago on SEO analytics with Google Analytics, which is in my opinion one the biggest bomb I read regarding a Google Analytcs technique in the last 6 months. I decided to translate that post on my blog to tell you about it, and share this secret with the english speaking community.
As most of us know, this is impossible to track search engine bots with GA as the tool get data through Javascript tags and the only way to track bots is with Log files. Those 2 french guys found a solution, which will give you the opportunity to see on Google Analytics, which Spiders visited your site, and which pages they crawled!
You first need to create a new profile within your Google Analytics account. The profile has to be for a new domain (see below). You can give to the profile the name you want although ROBOTS.yoursite.com is in my opinion a good way to keep things clear within your GA account.
You then need to download this folder, which contains 3 PHP file, and change 3 small things within the config.php file which are:
- Adding the Google Analytics ID of the profile you have just created.
- Adding you domain name
- Adding the Hash of your domain, which you can find looking at the value of your _utma cookie on your website (see pic below).
The next step is to include the analytics file (downloaded earlier, and with the changes made) in the source code of your website (the header is the most appropriate location in my opinion).
<?php include_once( dirname(__FILE__).’/analytics.php’); ?>
That’s it! That is the only thing to do. I manage to implement this in 30 minutes, without having strong technical experience. The Outcome is very interesting, as you can then see which bots visited your website.
When you click on the bot name, you will be able to see which pages have been crawled by this specific spider.
And finally, if you click in one of the page, you will be able to see which machine visited it (IP).
I’m not the real author of this post, so please visit the real guys who made it, and give them some nice deserved comments. Web analytics blog AND SEO blog.
23 Responses to “How to Track Search Bots/Spiders with Google Analytics.”
Hi there.
First of all thanks for the tutorial. I really hope it works
I have done everything from this tutorial, but I was wondering if I have to put the GA-urchin-code, for the new account, in my index-file as well?
I have my original urchin-code in my index-file already off course, but what about the code from the new account? Can I have both of them on my site at the same time or what do I do?
I was also wondering if the include of the analytics.php have to be on every single page that i would like to track, or is it good enougt just to put it in the index-file in my root-folder?
Best regards
Jürg
Hi Jurg,
Yes, you must include the analytics.php file on every page of your website. Just add it to the header of your page.
The new GA-Urchin code created must NOT be on your website. In fact, you don’t need to paste the GA-Urchin code from the new profile anywhere. All you need is the analytics ID (UA-XXXXXX)of that new profile which you need then to add to the config.php file.
I hope this help. Sorry for the late reply.
Philippe
So, i add the include on every page that I want tracked correct?
I also take the analytics ID (UA-XXXXXX-X) and place that in the config.php file and place the first set of numbers before the (.) in the cookie and also place that in the config.php?
As of right now, GA is saying tracking not installed. any ideas on why not?
I have the original analytics in the source code for the domain, and the include file in the head with the updated UA for the new domain. Does that make sense?
It sometimes takes a whole day before the tracking is detected. Please check again today, and let me know if it is now working.
Where shall I upload the file
@Carlos Guevara
Hi Carlos,
You can upload the file in fact anywhere you want, as far as you call that file from every page of your website, indicating the location where you the file.
excellent tutorial. very easy to implement. thanks for this great feature (in english).
@russell
You’re welcome Russell, it was a pleasure to share.
Hello,
you mention “The profile has to be for a new domain”. Does that mean it doesn’t work for existing domains already added?
Thanks.
@Reggie
No, that is not what I meant. Basically, you have your existing account which might have runned for years maybe. The data showing is only your visitors as originally GA does not track bots. The post explained you how to see exclusively the bots traffic within one extra profile. Instead of having 1 profile mixing real people with bots, you keep the existing one the way it is, and you create an extra profile which will show exclusively bots traffic. That profile for the bots does not have to be associated with any website in fact. It is advised to create it like it is a different domain..
Thanks for the reply Philippe.
For other people, to avoid errors, you might need to change the include to:
Waiting if it will work (24 hours delay), should this script insert another tracking code into page immediatelly after application?
Does any one successfully implemented it other than the author??????????
Thank you so much Philippe, I will try this out and get back to you if need any help. Hope you will
Thanks
Sankar
Hey Philippe,
It would be great if you could try this and prepare a video, So that we all can follow you. Anyways I will experiment it. If i succeed then for sure i will write a post and prepare a video too.
Wish me all the best
Thanks
Sankar
Hi Philippe,
I did as you mentioned in the blog. Made changes in the config file and uploaded 3 files to root folder. Kept the following code in the header section. Could you please let me know is that right? or else Do i need to change any thing in the following code.
waiting for your reply…
Thanks
Sankar
Forgot to fill name field in the above comment. It’s me who made the above comment. Still in Google analytics showing not tracking yellow symbol.
Let me have a look sankar, and I get back to you.
The problem might come from the “include” php script which is not calling the right file name. I’ll implement this to a new site I have and get back to you for more details, as I have not implemented this in the last 6 months. You’re right, I should do a video for this stuff!
Everything goes ok untill i add the :
to my header (or footer). My WP teme get’s screwed up and gives me an error. Anyone has the same ?
Dave
@dave
I had the same issue the first time
Very quickly, if you put the whole folder “analytics” which contains the 3 files in the folder which contains the file footer.php, your include once script must be a little different: it should be:
and not
Try this and let me know.
I did it just last week-end for a friend’s blog and it’s working fine.
The reason most of time it does not work is because of the include_once script which does not call the file from the right place.
Damn, anytime we want to write the include in comments it just does not work! Here again:
It should be for wordpress:
dirname(__FILE__).’analytics/analytics.php’
AND NOT
dirname(__FILE__).’/analytics.php’
@Philippe thxs, will give that a try
Dave
Thanks Phillipe for this excellent post!!
i want to implement this on my sites , but i have to to ask you this:
on all of my pages there is the new GA tracking code installed.
you said eariler that it must not be installed. so how can i implement this?
also, do i need to change something on the GA code? repalce the site id? won’t my visitor tracking be stopped this way?
Ok basically, you will need one Google Analytics profile for your normal traffic, which you will add on every page of your site. You must create a new Google Analytics profile and follow this tutorial in order to get the bots traffic. This 2 entirely different things!
Leave a Reply