It's three weeks after we launched theyworkforyou.com, and I finally have time to write an a proper entry about it.
As well as a providing a decently searchable, accessible, commentable, trackbackable, RSS-able, and shareable version of the parliamentary record, the site is intended to perform another function; to inject as much of hansard into google as quickly as possible. It is designed to be easily navigable by the Googlebot, who should find all the pleasure and nourishment it needs within the site, and be able to reach all content.
So how well are we doing?
to track it, I've extracted all the lines from the logs with 'Googlebot' as a referrer:
Googlebot's activity on theyworkforyou.com
This gives raw visit data for the bot, but the bot could visit the same content chunk several times, so I wrote a little script that strips out duplicate hits for the same page.
deduped page requests from the Googlebot
In summary, google has indexed (almost all) MP pages, all glossary items, and is making steady progress through the Written Answers and debates.
These stats are updated daily.