A Conversation for The Nearly but Not Quite 'Official' Peer Review Discussion Forum

Quality of Peer Review - Discuss

Post 1

Icy North

As some of you may have seen, Felonius Monk has been questioning the quality of Peer Review in general over on Alex Ashman's PR thread for the update of the Writing Guidelines. I've started this conversation here as it's a better forum in which to hold this discussion.

I'll try to paraphrase FM's concerns (please correct me if I don't get it across correctly - his original posts are at F48874?thread=5475211). Some of these points may overlap:


1. Peer Review today only has a handful of entries compared to h2g2 in its heyday.

2. Many of the current entries aren't up to scratch on quality, and aren't in the spirit of h2g2.

3. It's not enough just to recruit new writers, as they are able to get away with writing low-quality entries.

4. h2g2 needs to discover its distinctive voice.

5. Long-term researchers need to show some leadership on this issue.

6. There isn't enough 'review' going on in Peer Review - too much chat, etc going on in there.

7. There's not enough 'field research' going on - people are only writing about things they currently know about.

8. Whimsicality is now seen as an end in itself, rather than a starting-point for more incisive, factually-based writing.


So, who'd like to start!


Quality of Peer Review - Discuss

Post 2

BMT

Point 1 - Already noted via the Researchers Group http://www.bbc.co.uk/dna/h2g2/brunel/F8622433?thread=5407218&latest=1 Point 2 - There are only a couple that I can see that shouldn't be in PR. Taking out the Updates, what's left is a good mix for all sections in terms of 'Life, The Universe and Everything'. Point 3 - I've only seen a very few new writers that try to 'get away with low quality', in fact tried to get away is the appropriate term as they've failed miserably. The last debacle with Henryprefect is an example. Point 4 - Not sure what this means to be honest, h2g2 is distinctive by its very nature in comparison to other sites like W*K* or social networking sites. We don't want to be going down that road. Point 5 - agreed. Point 6 - Some threads do tend to go off topic and thats for the scouts to deal with. Doesn't seem to be enough 'policing' of threads for want of a better term. Point 7 - My latest article, by coincidence, was as a result of a trip out and seeing the place I then wrote about. So this argument doesn't run. A couple of the Heavy Horse articles were based on previous visits and personal knowledge as well. I think so long as research is done thoroughly there isn't always a need to be 'out and visiting' in order to write authoritively[sp] on a subject. Point 8 - I just plain disagree with this statement. I think whimsical and or quirky fits well with the ethos of the guide. I think when so called 'old hands' have the sort of rant that's occurred to prompt this type of discussion thread they should really try to do what writers attempt to do. At least have a balanced arguement, as in, don't just be negative, try putting forward some positives or positive ideas. Being nasty and personal just doesn't wash. Its a lesson hard learned on occasion. We're all human, we all have our moments of madness so to speak, but at the end of the day we all want whats best for HooToo and especially the EG. ST MKII


Quality of Peer Review - Discuss

Post 3

Elentari

7. There's not enough 'field research' going on - people are only writing about things they currently know about.

This assumes that most entries are about places (whether towns, buildings, or whatever) which is not the case. I'd like to see more entries of that nature but there are many entries which don't require research in the field.


Quality of Peer Review - Discuss

Post 4

Mu Beta

smiley - popcorn "Peer Review today only has a handful of entries compared to h2g2 in its heyday."

Well, when was its heyday, then? The first year or so was spent scratching around, writing some quite pointless and silly entries. I suppose that the alleged 'glory years' were around 2000-2, because people were already complaining that the quality of the site was deteriorating when I turned up (maybe not a coincidence).

We have various strategies designed to engage interest in the Edited Guide - I have to say that my own personal initiative of EoTM is presently receiving a disappointing level of response. It has to be also considered that the guide is filling up - we'll never be of W*k*p*d** proportions, but the average casual visitor may well be disappointed to find that he was no longer able to write a definitive Guide Entry on - say - keeping slugs out of your garden, or snooker.

The site, like everything else, evolves. We have no control over who signs up and who leaves, and anyone who thinks otherwise is deluding themselves.

smiley - popcorn "Many of the current entries aren't up to scratch on quality, and aren't in the spirit of h2g2."

I was never informed what the spirit of h2g2 was, although I personally appreciate a good scotch. This is idealism, pure and simple. We have good entries and poor entries; good authors and bad authors. This is no reason to discourage those less able - if I ran my classroom in that fashion, I'd be sacked.

I have noticed in the past FM's attempts to formulate a one-man Peer Review (oxymoron?) by decrying an Entry or genre of Entries simply because he decided he didn't like them. My personal approach is just to steer well clear of Entries that don't meet my tastes.

smiley - popcorn"It's not enough just to recruit new writers, as they are able to get away with writing low-quality entries."

If any new writers read sentiments such as this, it's not surprising that we don't recruit enough. Yes, there are new writers with low-quality Entries, but Peer Review is also a constructive process, and as long as there are volunteers willing to help and said newbies are willing to take on board constructive criticism, there is no problem. On the flipside, I know at least one 'old hand' that turns out a stinker of an Entry on a regular basis, but is virtually immune to construction.

smiley - popcorn"h2g2 needs to discover its distinctive voice."

The distinctive Entries are still there, aren't they? If anyone wants to see the best of h2g2, then the old hands know exactly where to direct them.

smiley - popcorn "Long-term researchers need to show some leadership on this issue."

My personal slant is that the Eds need to show some leadership on the issue. While I appreciate that 7 Editors was gross over-staffing, there is much less community spirit since the cutbacks. Long-term researchers are generally willing to do what they can, but they are only volunteers. It is not coincidental that a sizeable number of sub-U200000 members were students when they signed up, but now have proper jobs and less time. Likewise, many good Researchers signed off when they left for University.

smiley - popcorn "There isn't enough 'review' going on in Peer Review - too much chat, etc going on in there."

An overly focussed review is likely to alienate many people from PR - and we've all seen that happen in the past. A bit of chat is no bad thing where an Entry is humorous or a clear shoe-in for the guide (and, yes, I know people's individual standards will differ, but there is some common sense left). I think that standards in PR are often too exacting; newbies are scared off by being dumped into the EGWW, where too little goes on. Surely we are overdue to combine the two, along with a more stringent Scout team with less (or no) deadline pressure. I quit my Scout job because I didn't feel there was a realistic number of quality Entries to pick by the deadline.

smiley - popcorn "There's not enough 'field research' going on - people are only writing about things they currently know about."

Quite right too. Writing for the sake of writing is arduous enough. If we don't encourage people to write about their knowledge and interests, we need to start paying them.

smiley - popcorn "Whimsicality is now seen as an end in itself, rather than a starting-point for more incisive, factually-based writing."

See points 2 and 4. The Guide is whimsical, to a point. The original HHGTTG was described as whimsical, among a whole host of other adjectives, by DNA. Yes, there is need for a more serious style of writing here if we wish the BBC to keep funding us. But a throwaway entry in suitable style is just as good and bolsters readership.

B


Quality of Peer Review - Discuss

Post 5

Icy North

Well here's my smiley - 2cents

FM isn't alone in these views, and I sympathise with much of this. It must be difficult for the long-term researchers who have contributed significantly to the project, but who have seen their healthy pipeline of entries dwindle. They must struggle to understand why it's tailed off so much.

I only started contributing in 2005, but as a subeditor I've read a lot of the older entries when I've been assigning links. I don't believe that the overall quality was any better then, but I agree with FM that the style has shifted.

I think some of it lies in Editorial cutbacks. When I started we had Paully and Jimster spending a lot of time in Peer Review. They were always around to steer entries in the right direction - with authority. Scouts on their own don't really have the same impact - Peer Review can feel more like a collaborative writing forum at times.

I'd like to invite researchers old and new to suggest entries which they feel are most in the spirit of h2g2, so we can understand how to recapture some of that, both in our own writing and in helping others to do so.

smiley - cheers Icy


Quality of Peer Review - Discuss

Post 6

Secretly Not Here Any More

smiley - book


Quality of Peer Review - Discuss

Post 7

aka Bel - A87832164

Good points made by all, I don't really have much to add.

Just to repeat something I said elsewhere on this matter: I agree with >>if any new writers read sentiments such as this, it's not surprising that we don't recruit enough.<<

It is off-putting, to only see such negative comments, the balance (as pointed out here) is very important. Criticism is important, but constructive criticism will gain so much more than just bashing something or somebody.


Quality of Peer Review - Discuss

Post 8

U168592

I guess I qualify as one of the old-timers now, although I don't enforce my views anywhere, never have.

But as far as I see it there are concerns that h2g2 is on its way out.

Maybe it is.

Things run their course, c'est la vie.

I don't want to comment really, as people will take offence. But I agree that some good old fashioned field research rather than rehashing of information already available elsewhere would be nice to see in PR more.


Quality of Peer Review - Discuss

Post 9

Mina

PR is definitely the quietest I've ever seen it since I really started taking notice back when the Beeb took over.

A lot of the trouble I've found (and I have seen other Researchers say this) that the better you get at writing, the less people bother to review your work. It's one of the reasons I've just given up. If I am now so good (or possibly, so boring) that people can't be bothered to review it, why bother?

So, encouraging 'low quality' new writers seems to be the best way to go to get things moving. I'd rather see 100 new writers contribute and improve an entry each than applaud yet one more person for writing 100 entries on their own.


Quality of Peer Review - Discuss

Post 10

J

Very quickly...
In the other thread in PR, FM named an entry currently in PR which typified his frustrations with the quality of the guide. He was then criticized for doing so. Years ago, when I was complaining about the quality of the Guide, I refused to publicly offer any examples because I didn't want to offend anyone or personalize the issue. I was then criticized for that. I was told (in so many words) that my lack of specific examples was an indication that I was some kind of modern Chicken Little running around, declaring the sky to be falling without having looked up.

If a person feels that an entry is not up to the standards of the Guide, he should be allowed to say so (wherever he wants) without being told he's out of line. It's the only way to keep bad entries out of the Guide. When MB says, "I know at least one 'old hand' that turns out a stinker of an Entry on a regular basis, but is virtually immune to construction." it makes me think that he should say so in the PR thread. (and I fully expect to see him doing so in my next one)

That's one kind of leadership we need.


Quality of Peer Review - Discuss

Post 11

Secretly Not Here Any More

Right, I apologise in advance to every single person I'm about to offend with this. I love each and every one of you, like your work and feel guilty for what's going to be said, but this seems like as good a time as any to be blunt.

My answers to the OP:
1. Peer Review today only has a handful of entries compared to h2g2 in its heyday.
- Agreed. I'm not sure when the heyday was, but it's definitely quieter than when I joined back in '03. And worryingly, it's dominated by the same faces.

2. Many of the current entries aren't up to scratch on quality, and aren't in the spirit of h2g2.
- Agreed again. For every entry that we do well (slow-cooked beef A35979259) we've got a handful that (despite the very best of intentions) are boring. The everlasting constellations series is a prime example. GB's a great writer, and one that I admire, but are we really going to have an entry on every star in the sky when we've got nothing on "what to do in Berlin for 8 hours with only 15 euros in your pocket"? It stinks of trying to get a "full set" on a subject, just to say "we're as good as wiki!" h2g2 should have gaps! If I want a life and times of Josef Stalin, I'll read Montifiore, not this place! I'm guilty of this - for every quirky "how to go to a footie match" or "what to do if you're a student in Manc who hates RnB", I've got an interminable entry on history. We need to be the Guide, not the encyclopaedia.

3. It's not enough just to recruit new writers, as they are able to get away with writing low-quality entries.
Wrong. Wrong, wrong, wrong. New writers bring a new outlook, new ideas and fresh eyes. Too many newbies are redirected to the EGWW. Why the hell do we do that, and then not follow it up? Who here, honestly, contributes in the EGWW? When we get rough diamonds, we bury them instead of polishing them. Look at the "Raven Kids" - young, enthusiastic, computer literate. Why are we having the same old debate here instead of seeing what they can offer?

4. h2g2 needs to discover its distinctive voice.
Damn right. See my answer to 2.

5. Long-term researchers need to show some leadership on this issue.
I'm not sure. We've had people like Skankyrich rally the troops, and all that happens is we all try for a short time, then either burn out (like me) or start doing endless short entries for the sake of completion (Galaxy Babe). If the old heads we had were the answer, we'd be fine. Not all of us have 200 entries in us. 10 is pushing it for most of us. We're failing the new blood because we're not showing leadership, but that's because we're confident that hard graft from the current bunch will see us through.

6. There isn't enough 'review' going on in Peer Review - too much chat, etc going on in there.
Chat's needed. There's nothing more demoralising than having one reply saying "change this" and then nothing until it's been noticed that "this" has been changed. A bit of chat lets researchers know that their contribution is being noticed.

7. There's not enough 'field research' going on - people are only writing about things they currently know about.
True - but that's always been true. The whole point of h2g2 is that we write about what we know - it's just we're writing about the wrong things. I know how to eat well on £10 a week from a Tesco Metro. I also know what Josef Mengele did at Auschwitz. Guess which one I made into an entry? The wrong one.

8. Whimsicality is now seen as an end in itself, rather than a starting-point for more incisive, factually-based writing.
Seen no evidence of that. We need more whimsy in PR. For my money's worth, the best writer we currently have on-site is SWL. He's got a knack for writing engaging, interesting and unique entries that could still theoretically come in useful (should one fall out of a plane, that is.) We need more of that, and less "Here's my dissertation on brain chemistry".

(Apologies again to everyone, especially Galaxy Babe. Really, I am terribly sorry for picking on your entries and I sincerely hope you don't take it to heart.)


Quality of Peer Review - Discuss

Post 12

Galaxy Babe - eclectic editor


Quality of Peer Review - Discuss

Post 13

Secretly Not Here Any More

I feel really guilty now! I'm sorry GB!


Quality of Peer Review - Discuss

Post 14

Malabarista - now with added pony

Found this by accident.

Went back and read the "original" discussion in the PR thread for Alex' entry.

Was surprised to find my superball entry (which I'm rather proud of, I must admit) in there as a negative example.

So, yes, that's off-putting, especially considering the amount of research, etc that went into it. And no comment from FM about how to make it better, or that it wasn't guide material at all in the PR thread itself... If you don't like it, do something about it. And explain why.

Honestly don't think I'll be submitting any more for a while, though I have a few nearly-complete ones lined up. smiley - zen





Sorry. Back to the topic of this thread.

Negativity won't work. And if I find an entry on a subject I really don't like or know nothing helpful about, I either ignore it, or, as with the Wolfenstein 3D one, just ignore the *content* and try to help clarify facts, find typos, etc.

Only positives aren't the way to go, either. People like to hear that their entry is appreciated, but it doesn't help you make it any better. So, don't spare the constructive criticism. smiley - ok



Oh, and I don't see the point of solo entry badges - that discourages collaboration, and if we want a general guide rather than just a guide to very, very specific things, collaboration (on hootoo's worldwide scale! smiley - earthsmiley - wow) is the way to go, I think.


Quality of Peer Review - Discuss

Post 15

Fizzymouse- no place like home



I have the entry in PR that FM and others so despise but not one of them - NOT ONE - has had anything to say in the PR thread attached to that entry.smiley - doh


It strikes me that they should either put up or shut up - if they don't like it go to the entry and say that - if they know how to make it better - go and say that, if they think it shouldn't be in the guide at all - hell say that ... just don't carry on like a load of old women bitchin' about people in a totally unrelated PR thread - or in here for that matter. smiley - 2cents


smiley - mouse


Quality of Peer Review - Discuss

Post 16

Malabarista - now with added pony

That's why I don't want to "bitch" in here, either.

The comment that I'd rather have constructive criticism than someone silently resenting my submissions, because otherwise, I can't make them better, was *not* meant as an attack on FM. smiley - peacedove

It's just what I hope can be a general observation. If you can't say anything nice, at least say something helpful. If you're ignoring an entry because it's not your kind of subject, fine. If you're ignoring it in hopes that it will go away, not very useful smiley - winkeye

Also - the voice of the guide, or the style of the guide can't be decided upon by one person alone - nobody can appoint themselves editor-in-chief and expect that everyone will follow.

I, for one, find restaurant reviews for places I'll never go extremely boring - especially as I have no money to dine out anyway. That doesn't mean I want them out of the Guide, because I know they're helpful to people.

But it's a big place. We have three very different Front Page entries every day. So why not let it be a bit diverse? That doesn't mean having entries that don't follow the Guidelines in spirit as well as in the letter. Just that if there's a wide range, there's something for everyone.


Quality of Peer Review - Discuss

Post 17

Skankyrich [?]

If the Guide was composed of just one type of Entry, I wouldn't bother with it. I rather like the fact that we have both A260272, a serious guide to Plymouth, and A1146098, a rather lighter Entry about one of its pub teams, sitting alongside each other. I don't favour one style over the other, and I think that getting the Entry as good as it can be is far more important.

A while ago, I started a project which has rather flagged due to a lack of walking time, in which I planned to cover the whole of the South West Coast Path - walking routes, towns and all. The idea was that you could walk the whole of the route without a guidebook and taking all the information you needed from h2g2, and I thought this was quite in the spirit of the Guide. So I don't see why GB should have to take criticism over the Constellations project; it's a big, ambitious undertaking on a subject she clearly enjoys writing about, and if it adds 50 or so intelligent, well-written Entries to the Guide I think that's brilliant. It's worth noting that the Constellations project contains almost as many Entries as there are in the queue - without them, we'd be in a much more precarious position.

It is very difficult to 'show leadership', and the Researchers' Group was a case in point. The problem is that it's very difficult for most of us to sustain a level of writing and contributing beyond what we'd normally do. I certainly found it harder to feel inspired to write more often - I actually ended up with writers' block from trying too hard - and I had trouble keeping up with my reviewing duties. I'm sure most members felt the same, and some disappeared off the radar almost immediately. Six months on, I'm not sure the initiative actually had any lasting effect. The likes of Fizzy, GB and ST kept writing as they were anyway, and most of the rest of us failed to keep up our quotas.

There are a few comments here along the lines of, for example:

'I'd rather see 100 new writers contribute and improve an entry each than applaud yet one more person for writing 100 entries on their own.'

'I know how to eat well on £10 a week from a Tesco Metro. I also know what Josef Mengele did at Auschwitz. Guess which one I made into an entry? The wrong one.'

But they aren't mutually exclusive, are they? Why not do both?


Fizzy, reading back through the thread I think the only person who criticised your article was FM himself.


Quality of Peer Review - Discuss

Post 18

J

"Oh, and I don't see the point of solo entry badges - that discourages collaboration, and if we want a general guide rather than just a guide to very, very specific things, collaboration (on hootoo's worldwide scale! smiley - earthsmiley - wow) is the way to go, I think."

Collaboration is great and you're probably right that solo entry badges discourage it, but I don't agree that collaboration is generally the way to go. There are times when collaboration works well - especially for entries about places, experiences and advice (among other things). But some of the strongest entries I've read have been those written in a single voice, or with an overall point. That sorts of entries are harder to produce in a multi-author piece.

I think I understand your overall point, Mala smiley - smiley But I'm not comfortable in getting used to thinking of collaborative entries as inherently stronger works.


Quality of Peer Review - Discuss

Post 19

aka Bel - A87832164

Hmmm, interesting that apparently it's now the 'old hands' who are to blame with remarks like:

>>'I'd rather see 100 new writers contribute and improve an entry each than applaud yet one more person for writing 100 entries on their own.'<<

and

>>or start doing endless short entries for the sake of completion.<<

I have to admit that this is indeed the impression you could get when the project was started. My own reaction was: omg, do I really want to know about every single constellation? and my answer was 'no', and I didn't read a single one. And guess who lost out here? Yes, I did. I got a constellation entry to sub-edit, then a couple more. They are brilliant. They are very well written and thoroughly researched (from what I can tell, I can't check the facts, but there are others who do). They are easily accessible for the idiot like me, a pleasure to read, and a pleasure to sub-edit. Oh, and don't forget, this is a project of the h2g2 astronomers, so there are other researchers who work on this. Go and read their entries, then compare with GB's entries. I'm sure you'll soon find whose entries you like best - I know which entries I like best. And I'm with Skankyrich here: we should be grateful that she writes and submits them, although they get hardly any reviewers in PR, and despite the fact that she knows exactly that most people think like Psycorp (as I said, I was guilty of that myself for a while). That's stamina. Go on, GB! and everybody else (Mala, Fizzy), you're doing well.


Quality of Peer Review - Discuss

Post 20

Elentari

Agreed.

I know this has been said before, but I think it's worth reiterating: I'm sure most of us would like to see more of the slightly off-the-wall, unusual entries (SWL's How to Survive Falling Out of A Plane, as mentioned earlier, is a great example) but not everyone can write those. I know I struggle because it's just not my style. I don't think that means I should stop writing altogether. I'm well aware that most of my entries probably don't interest the majority of readers, but that doesn't mean they're not worth writing.


Key: Complain about this post