h2g2 The Hitchhiker's Guide to the Galaxy: Earth Edition

Hi Jim ,

There have been a couple of threads lately where people have been discussing why Edited Entries now feature so poorly in websearches especially google.

This is the latest conversation F55683?thread=582344 and I was wondering if you or one of the h2g2 Editors could give an official explanation there.

I'll ask the Eds directly (Feedback-Technical?), although I'll leave it a day or so as the above thread is less than 24 hour old, and in case you're the right person to have asked first.

many thanks,
kea.

Hi Jim,

If it's ok with kea, I hope I can jump in here in this conversation, rather than start a new one on the same subject. Jimster suggested that I should contact you directly.

Would you have any thoughts or comments on our suggestion to leave the H2G2 Edited Guide open to Google robots so that we can get get a wider audience for our entries? What effect would a change to the META tag on the browse pages have? Would it be possible to open the Guide to crawlers at certain times of lowered activity? Could robots be restricted just to those from the principal search engines?

We all feel strongly about the quality of the entries we create here, and I think that is good thing; but when all our more recent entries are absent on Google, it is a demotivator as it now appears that we are writing our entries for a very small audience.

Thanks,

Woodpigeon

I'll look into relaxing the rules and see what impact that has on traffic.

At present, it appears that all Edited Entries are marked as "index, nofollow", which means that if the spider happens to come across them it will index them. But there is no way that the spider can find them, because Categorisation pages are marked "noindex, nofollow".

My proposal is that Categorisation pages should be marked "noindex, follow", which would result in the approximately 6,000 pages of the Edited Guide being available in Google.

And the index pages as well, just to be complete.

I see that the change has been made. Thanks Jim.

This change has now been made. We've opened up a few more pages for indexing and made the category and index pages allow link following, so Google et al should be able to find all the content.

We'll see if it has any effect on performance over the next couple of weeks.

Oh my golly, that was a weird simulpost.. !!

Thanks!

It's worked!

See F55683?thread=582344&latest=1 and F54784?thread=600648&latest=1

Thanks, Jim. I hope the extra load on the servers is in fact minimal and not the cause of the infamous SSO problems yesterday.

I'm pretty sure yesterday's (and today's) problems were unrelated to any server load. Yesterday was a routing problem on the BBC's content network and it affected practically every dynamic application on bbc.co.uk