Stats, always stats, this is good, but hey, how can we track spiders without just pointing stats ?!
My Spiders Tracker is simple: each time a spider hit a page, the engine store that info in the dB… one hit, one entry. Ok, this add one query per page hit, on the other side, spiders have no access to most of the features of the vBulletin engine itself and they have 30% less queries for each page loaded than the regular visitors, so the server is not requested that much.
Why i do that tracker ?!… comparing to Google Analytics, my tracker will trace all activities from the spiders, making all their moves useful to see what pages are indexed, what pages are avoided, and where there is possible threats (because spiders usually visit any link they can, so a no-follow could be useful sometimes!)
This can also be a good engine for SEO… i suppose Lizard King can give more info on that..
…
So, what the system can track ?!
- each pages hit by a spider, with date and time
- each search queries coming from google, yahoo etc, and their page hit
What can we do with these:
- Verify the periods in days/weeks where spiders are visiting, making it easier to see where they are consuming server-side resources and what they search
- Verify how your sitemap is exploited by spiders (if you have one)
- Verify what pages are most visited (you can even put a page tracker displaying info like « this page was indexed X times by spiders »)
Views: 14
Spiders may hit the sites pretty hard and actually it is not important which links they target because if you can correctly optimize your pages , you can pretty easily control bots to index the correct links on your site. Just an example of optimization can be found at this link .
A busy site can be hammered by yahoo each other day and additional query can be harmfull on some sites. The query should be carefully adopted and in my point of view instead of tracking which bot followed which link focusing on incoming search queries is more important. This can easily give the administrator to easily focus which keywords are effective , which are not. What can be improved , what should be dropped etc…
vBSEO Sitemap Generator has a system which allows you to track spider activity on your board.You can easily check which bots visited which threads easily. However vBSEO doesnt have a referral tracking and Google Analytics is not that good on referral tracking either. The system should track referring serch engines , referred keyword , refering links and actually display them with detailed info.
You hit the exact point i was focussing on, as usual Liz!
but one important thing is… people are not all here when the word « optimization » comes from you… as you always say, vBulletin lack a load of SEO things, that means most of the style designers are not aware of all the lacks and they don’t know a single bit of what to do to make a perfect style for vBulletin…