h2g2 The Hitchhiker's Guide to the Galaxy: Earth Edition

Hi Jim,

I'm not an expert on this kind of stuff, but I've been meaning to ask you something for ages.

I've noticed that all h2g2 pages have the metatag 'content="nofollow"'. My understanding of this is that incoming links on pages boost their rankings in searches, but search engine crawlers will not follow links in Entries because of this tag. In that respect, we would appear to doing ourselves a disservice; if every h2g2 Edited Entry links to, say, ten other Entries, this tag deprives h2g2 of 70,000 links which would improve its visibility to search engines considerably.

Now, I can see why the tag would be there for Unedited Entries; I could sneak a link to my business in an Entry I'd written about any old thing. No one would check the link, and it would be unlikely that anyone would ever find it to yikes it. Entries that go through Peer Review, however, are subedited and approved by the in-house team; to my mind they have a long process of quality control and any advertising or spurious links are routinely stripped out. So it seems, to a half-educated user such as myself, a bit odd that we're not allowing robots to follow links which are officially sanctioned.

So I thought I'd pop by and ask if there is a good reason I'm missing for the metatag to be there and, if not, if there is a way for it to be removed on Edited Entries? I don't know if there would be an easy way to do take it out anyway - to my mind, I don't suppose it would be a good use of time to strip out the tag manually, but then again I don't know if there is some magic button that could be pressed to remove the tag from all of them.

Sorry for the long-winded question, but I had to ask to satisfy my curiosity if nothing else

Traveller in Time just butting in
"How about the links to conversations on the forum of the Edited Entry ?
How about the links to the Personal Spaces of Researcher and Sub-Editors ? "

Did you see this, Jim?

I don't quite understand your point, TiT. We already link to those things, it's just that crawlers don't see them. There's no reason why the pages you mention shouldn't still have the tag in place to prevent users from boosting their own sites' rankings, which is (presumably) the only reason to have the tag there in the first place. If links are effectively sanctioned by the Eds for humans, why not for crawlers?

Traveller in Time reading the categorisation system
"It says


All pages published to the search engines should to be checked by the sitehosts.

Those pages however should not be used as an index, links from those pages end up on your Personal Space. Saying to a Robot: 'NoFollow' could lead to the interpretation not to follow links from that page.

By the way Jim, should the frontpage not be a nofollow ? "

Traveller in Time looking up
"Skankyrich,

>If links are effectively sanctioned by the Eds for humans, why not for crawlers?

You are referring to the Edited state of internal links.

External links in Entries already have to be maintained, most sites do not remain the same for ever . "

That's right, TiT. Basically I'm asking if it's possible to take the tag out of Edited Entries and leave it in all the others. I know links from within are less valuable than those from outside, but this must be balanced out by the fact that the links are from bbc.co.uk. So if we could harness those tens of thousand of internal links between Edited Entries, we could see some of our more popular Entries moving towards the top of searches. That can only be good for the site.

The nofollow tag was added several years ago, because the site was being overwhelmed by search engines. This was primarily because we'd just unhidden the huge pre-BBC backlog of conversations, and the search engines were trying to crawl all of those pages, and effectively causing a Denial of Service.

The nofollow means that crawlers don't try to crawl the conversation forums, and stick to guide entries. Guide Entries get linked from the index pages and the category pages.

We've been talking about relaxing the rules on robots recently, so perhaps it's worth trying this experiment to see a) if it makes a difference to search rankings, and b) if it causes excessive server load.

Thanks for the explanation of how it all works, Jim. I only have a rudimentary knowledge of these things, and something like server load isn't something that would ever occur to me. I do appreciate you taking the time to reply

If you do experiment, let me know and I'll help you measure it if I can. I could pick a dozen or so well-linked pages at random and give you their Google ranks before and after, if it helps. I've always linked heavily while subbing Entries thinking it would help the PageRank of each, but at the moment it doesn't - if it turns out that it does have an impact, we'll at least be able to let the Subs and Curators know that it's suddenly become a more important part of our job.

Thanks again for your time, especially at such an early hour, and if there's anything I can do give me a shout.