A bundle of features for the Spiders access. Hits tracking, complete listing for info, Specific usergroup for Spiders, etc

It’s globally a merge with some small addons i wrote in the past, and as i did not want to release a ton of minimal tools that just fit together, i make a real bundle, 4 or 5 tools together, with activation and permissions settings when needed.

Settings:

Spiders List: a little spiders tracker for your forum. It’s not tracking each page the engine is viewing, because this is pointless. Instead, It is listing the name of the spiders that visit your sites, the last date of a visit, the number of unique visits and the number of pages viewed. That information is not very important for the indexation of your site, but it helps to see why your site may be occupied or not. You can then take action if a crawler is visiting and still giving no result on search engines.

You can see it in action here: vbEnhancer.com – Crawlers List

Specific Usergroup for Spiders: i released this addon on vb.org long time ago, and it was copied in source, but this version is updated and have more flexibility. You simply have to choose the proper usergroup in the settings so when a spider/crawler visit your site, it is considered having some permissions… it’s useful if you do not want to fill your robots.txt file with strange access blocks. This let you give access to crawlers for profiles but not visitors messages, etc…

Also remember to follow the TOS of the search engines you are registered to. Google until lately was blocking sites that were ghosting their content.

Display Spiders in WOL: and in any page showing « Currently Active Users » (showthread, forumdisplay, etc) … that way, you see where these beasts are visiting.. 🙂

As you can see in this listing, the markup for the usergroup applied to the crawlers give some style to the web crawlers, easier to trace that way.

… some other tools are to be decided to join in the bundle, i’ll see later!

CRON JOB:
to make it easier on the server, there is a cronjob storing the hourly stats about the crawlers… once the cronjob is done once (it’s the cron named Hourly #1), the stats appear in the right place…:

…update: may 1st, 10:50, a small change, the Crawlers listing will now update the spiders list in cache if the file changed, so you can update it when needed.

…update: may 26th, a change related to a request by Calystos here, as we can apply a usergroup to the crawlers, we will now be able to add some markup to that usergroup and it will show in the WOL and online.php …

and in the Who’s Online page (demo vbEnhancer.com):

i made it so the « Spider » in front of each spider is deactivated in the online.php page, because it’s pointless if you ask me… but you can deactivate the plugin of the hook « online_bit_complete » if you prefer.

note: 17/06/09: update to 1.1.1, now will update the proper count and names of web crawlers in Active Users of Showthread and Forumdisplay pages… thanks to all who reported it, mainly [user]xOBKx[/user]… 🙂

note: 19/06/09: no version change, but added the spiders count in the WOL page itself… from [user]xOBKx[/user]’s suggestion.

note: 09/07/09: no version change yet, Dream updated his spiders_vbulletin.xml, so i provide it in this first post, if you want to upload it to your /includes/xml/ directory… it will update the list instantly when needed.

note: 09/07/09 by night: bundle updated with the latest spiders list from Dream, and updated some bug fixes suggested by [user]xOBKx[/user], like the extra comma when there was nobody online, and the uncached template.

Views: 109