A Conversation for Ask h2g2
Peer Review and Quality Control
J Posted Apr 26, 2003
Oh and to correct one of my last posts, I have written a new Calvin and Hobbes article and I put it in PR.
Shameless advertisement? Yes, but that's how we make sure the process is working. Lots of feedback
Peer Review and Quality Control
There is only one thing worse than being Gosho, and that is not being Gosho Posted Apr 26, 2003
Problems such as this often arise because a Researcher has been fired up to write an entry about a subject which they have more enthusiasm for than knowledge of. Or they may have good knowledge of part of a subject but not the whole picture.
In an ideal world Peer Review would fix that because another Researcher or Researchers would come along and say "You've missed such-and-such", but there are always entries which get little attention. Some of them get picked, most don't, the majority should never have been in PR in the first place.
Although working in PR is a lot of fun, at times it can be quite a slog. There are only so many 'fluffy' ways you can ask a Researcher to read Writing-Guidelines and remove their entry from PR. Only so many 'fluffy' ways you can tell a Researcher that although they've written an interesting entry there's little likelihood of it being recommended. Only so many ways you can try to revive an old, neglected thread before asking Anna to move it back to entry or consign it to the Flea Market. Perhaps 'burn-out' is the phrase I'm looking for. I know it keeps me from involving myself in many threads as often as I ought to.
There was a discussion somewhere a while ago (I can't recall whether it was on h2g2 or the Scouts group) about changes which could be made to the submission process to reduce the number of unsuitable entries submitted to PR, or to make sure that they're at least submitted to the *correct* PR forum. I don't know what's become of any of the suggestions that were made. I'm sure that PR would run more smoothly if some of them were implemented.
We should also ask ourselves if the focus of h2g2 has shifted from being an 'earthbound version of the Hitchhikers Guide to the Galaxy', to just another web community (albeit one of the best ). What proportion of current Researchers are actively involved in the EG (in whatever capacity) compared to say, pre-Rupert? What can we do to encourage more Researchers to involve themselves in the editing process by subscribing to and commenting in PR? The more people there are reading submitted entries, the greater the knowledge base, and the less likelihood there is of entries being edited which 'have many omissions and contain much that is apocryphal, or at least wildly innacurate.'
Peer Review and Quality Control
anhaga Posted Apr 26, 2003
re: your last paragraph, Gosho.
I've wondered the same thing. It seems to me there are two communities here with a certain degree of overlap. There are a number here who are really interested in making the guide and there are a number who don't really seem to even be aware of the guide. I have no idea what the proportions are. I don't know it this division, if it is a division, is a problem or not. It seems that the first group mingles freely with the second group, which is nice. I hope the second group spends time reading entries. I don't know.
Peer Review and Quality Control
J Posted Apr 26, 2003
I disagree with your first sentence. "Problems such as this often arise because a Researcher has been fired up to write an entry about a subject which they have more enthusiasm for than knowledge of" The Writing-Guidelines themself say that you should write entries you're interested in. I started out with most of my currect EG articles with nothing but a computer and ambition. That's all you really need.
In greeting newbies (I'm not an ACE, though I occasionally check who's online and drop by friendly looking people) I always try to encourage them to write entries and go to PR, but it's not an easy process for newbies. And, (Much like the tobbacco industry) we have to hook them while they are young (To the site)
I would have quit this site long ago if it hadn't been for PR though. You can't expect someone who can't write to write an entry. It's as much of an issue of no new authors as new authors who can't write though. As someone said earlier (I think) the unsuitable entries are clogging up PR, slowing down the scouts and ultimately reducing the quality of entries.
I like the philosophy as putting in as much as you take out of the system. Two of the most Edited writers, Gnomon and DD (For whom I have an enormous respect for) don't have any EG process badges (Though I'm sure writing keeps them quite occupied). I'm trying to not be a hypocrite
I really need to go to sleep now anyway. If it explains anything, I'm too sleepy to make a coherent train of thought.
Peer Review and Quality Control
SEF Posted Apr 26, 2003
I was thinking of putting some of my points as rhetorical questions but I went to bed instead. It seemed like a good idea at the time. Anyhow I do now have one possibly constructive suggestion:
Edited Guide articles end up in categories. Perhaps this could be extended back into peer review to make it easier for people with specific expertise to find them. Some sub-divisions might even be linked to clubs, eg The philosophers home-page for philosophy, the maths lab, the gamers (wherever they are) etc etc. I think it's not unreasonable to expect scouts to have some idea of where something they've picked would fit into the guide and it could always be moved later (as might well be the case with author self-submissions). Anyhow some sort of linkage would provide instant advertising to those people more likely to make worthwhile comments and, even without that, the subdivisions would make it easier for people patrolling the peer review list.
Peer Review and Quality Control
Whoami - iD dislikes punctuation Posted Apr 26, 2003
I've now read the entire backlog and there are a few comments I'd like to comment on:
Jodan: "Perhaps if you think a recommended entry is incomplete, or is structurally flawed, you could say something to the Italics. But they don't like to add things, just correct them."
Contact the Sub-Editor while they've still got the Entry. Once it's past Peer Review, the Sub has control of it for around a month and they can add (or in extreme cases remove) material as they see necessary. Subs are generally proud of their work and want it to be as good as possible.
Jodan: "Too much work? Clearly for volunteers."
Admittedly, having to squeeze subbing around RL isn't ideal, but it's the best way we have.
Z: "You can't expect the scout to know that there are glaring errors in a piece if it is on a area he/she/it knows nothing about."
I like to check up on key facts in an Entry I'm of a mind to pick, just as I might run entire sentences through Google to check it isn't copied. That's worked well for me so far.
Z: You can't expect [Subs] to be an expert on the topic, however they should check the PR thread.
Yes - that and again, read around the Entry. This would be easier with a little technological streamlining allowing the Scout to pass their finds onto the Sub more easily.
Gnomon: "I don't like the idea of having to provide links to prove that your stuff is factually accurate."
References to back up points are OK, but I don't think the burden should be on the Author to prove to the Italics that their Entry is factually adept. They should be satisfied for themselves that their Entry is good, and that should suffice.
SEF: "I disagree with so much of the current system as it appears to be implemented (though not the ideal of it) that it is hard to post on this topic."
SEF, I'm interested to know what you think. You can always email me at [email protected] and I'll treat what you say about the Process in the strictest confidence, if you think that's necessary. However, allowing me to pass on your thoughts in a calm form to the Italics might be more productive.
Jimi X: "Do Scouts sometimes pick entries that I wouldn't pick? ... Yes."
That's a good thing in some cases - but I agree some borderline stuff gets picked.
Jimi X: "No system is going to be perfect."
You said it before I had the chance!
Jimi X: "I've read some interesting stuff that I might not have otherwise because somebody else felt the topic was EG-worthy. That's part of the site's appeal to me."
That's why I said that it was good in some cases - I love that too. That's one of my reasons for being a Volunteer.
Gosho: "Problems such as this often arise because a Researcher has been fired up to write an entry about a subject which they have more enthusiasm for than knowledge of."
Gosho, that's rather negative! We're all called 'Researchers' - so if the Entry is well *researched* by the Author, everything will be fine.
SEF: "Edited Guide articles end up in categories. Perhaps this could be extended back into peer review to make it easier for people with specific expertise to find them."
For what my opinion is worth, I think this would be dangerous. PR is big, but splitting it up could make certain topics implode due to a lack of discussion.
I'm going to have another think about a few things here and there and see how it goes.
Whoami?
Peer Review and Quality Control
There is only one thing worse than being Gosho, and that is not being Gosho Posted Apr 26, 2003
"Gosho, that's rather negative! We're all called 'Researchers' - so if the Entry is well *researched* by the Author, everything will be fine."
It's not in the least bit negative This thread started out with Hoo pointing out an edited entry which has some omissions which render it to be not the entry it should be. The entry in question clearly *hadn't* been well researched, and everything wasn't fine. He finished his opening post with a paragraph asking if PR is failing Researchers who submit entries, and failing the EG. He asked the question "Peer Review needs better quality control. Anyone got a better idea?".
I've often wanted to write an entry on a subject I have scant knowledge of but am very interested in, like Yin and Yang for instance. That one came up during the PR thread discussing Jodan's Feng Shui entry. I had lots of enthusiasm to research and write an entry, but I decided against it because I really don't have enough direct contact with the subject, and could quite possibly leave out something (or somethings) very basic to the subject. At some point in the future, another Researcher may come along and say the same kind of thing that Hoo did to begin this conversation - that PR has failed this entry because it's full of inaccuracies and omissions. I think it would be very arrogant of me to write an entry about something which I know little of other than what I've gleaned from a bunch of websites.
"...nothing but a computer and ambition. That's all you really need."
I disagree strongly Jodan. You need experience of, and immersion in a subject to write an authoratative entry about it. Without those two things you'll have no idea of what else you *could* have included until someone like Hoo comes along and says,"Hey, you've left out xxxxxx! How could you possibly have had this entry edited without talking about xxxxxx?"
The people who regularly post to PR do their best, and will point out inaccuracies in submitted entries when they are aware of them. Look at your 'Grand Canyon' entry for example. It was pointed out by me and another Researcher that the section on geology was factually and grammatically wrong. In that case PR did its job well. Should we have ignored that and let it pass for the sake of getting the entry into the EG just because of the author's ambition?
You need more than just ambition or enthusiasm to write a good entry. If that was the case I'd have submitted ten times the number of entries to PR as I have done so far.
Peer Review and Quality Control
J Posted Apr 26, 2003
>>Contact the Sub-Editor while they've still got the Entry. Once it's past Peer Review, the Sub has control of it for around a month and they can add (or in extreme cases remove) material as they see necessary. Subs are generally proud of their work and want it to be as good as possible.
Entries aren't widely publicized until after they get away from the Sub. The place where an entry has the most chance to be spotted as having a problem is on the front page, after the sub has let it go. That's the problem. If we can get more people in PR, that's the best solution, if we can get people lurking on the <./>comingup</.> page, that's another solution, though less plausible.
Peer Review and Quality Control
J Posted Apr 26, 2003
Oops, simulpost
I have to say Gosho, I disagree with a few parts. Your examples are good because they pertain to me, and I'm prone to error. Unfortunately the Grand Canyon mention is irrelevant as, in that case PR did do it's job. An over-enthusiastic researcher such as myself will not be stopped from writing things he knows little about (Though I can argue being an American gives me first hand authority on the Grand Canyon compared to people from Canada, the UK or australia) and PR has to help those people. In the end, my entry was downsized to Visiting the Grand Canyon. PR did well, and I was satisfied.
But if you need prior knowledge of the subject, how can you expect obscure entries to be done? Anhaga made a good point, that his current entry is so obscure that neither him, nor peer review complete knowledge of the subject. That's one of the reason you write an entry. To learn interesting things! If you HAD to have prior knowledge of a subject no obscure entries would be done. And that's what gives h2g2 some of it's charm. If there were only obvious entries, this site would be much less fun
As for Feng Shui and Yin Yang, I encourage you to write an entry on Yin Yang. Maybe it won't be perfect for PR, but I'm sure Zarquon's Singing Fish will come along and share some interesting facts
Peer Review and Quality Control
anhaga Posted Apr 26, 2003
Just a small correction, Jodan:
I always have absolutely complete knowledge of the subjects about which I write entries. (actually, in the case of my obscure entry, I have read all published materials on the subject except for one that's just come out and should arrive in the mail quite soon, and the subject has been rolling around in my head for two decades.)
I agree that lurking on coming up is a great thing to do. That's where I noticed the entry that has been troubling me and seems to have entred limbo.
Peer Review and Quality Control
J Posted Apr 26, 2003
I apologize Anhaga. I go to ComingUp quite a lot. Consequently, I don't read many entries when they go to the front page... because I've already read them
Peer Review and Quality Control
Mikey the Humming Mouse - A3938628 Learn More About the Edited Guide! Posted Apr 26, 2003
Some points about researching entries, and sources, etc.
People have commented that authors need to do a better job of researching their entries, and some have suggested that they perhaps should provide links to verify factual accuracy.
For me, that wouldn't cut it. There's far too much info out there on the web that's factually inaccurate as it is. There's far too much out there in *books* that's factually inaccurate as it is. In the information age, it's becoming increasingly difficult to be a discerning customer of information, to tell the good from the bad -- which to me, was what this started about with HVL's post.
For example, take Mina's recent entry on Newgate prison. I can clearly tell that she thoroughly researched the entry. That didn't prevent factual inaccuracies from ending up in the edited entry. There are both history books and historical websites that equate typhus with typhoid, so the fact that she did the same in her entry is not because she didn't do the research, it's because of what kind of sources she used. And because she was writing a historical entry, she used historical sources -- well, enough, really. Ideally, this kind of error should have been caught in PR -- but I never saw the entry in PR, didn't see it until it hit the front page (again, as an editor's choice entry), and pointed out the error.
Is it possible that the incubation period for PR is too short? One week simply isn't long enough to ensure that errors are caught? I can think of more than a few entries that have slipped through because they have been so *well-written*, that those in PR have assumed that they were completely accurate, and they were picked as soon as they hit 7 days -- not enough time for enough people to see the entries and spot the errors lying in wait. Frankly, I think upping the incubation period to 2 weeks might solve a lot of these issues, although I can see it peeving a lot of people off.
On the other hand, someone earlier in this thread complained about not having seen my entry on FoxTrot before it was picked -- while I will fully and freely admit that out of every EG entry I've written, this is the worst -- it is ALSO the one that spent the longest time in PR -- several months, I believe. So I don't think people can use the excuse that they didn't have the time to see it for that one.
Peer Review and Quality Control
J Posted Apr 26, 2003
I mentioned that on FoxTrot. It's a fine entry, but I had a few more points I would have brought up. I'm not complaining, I don't even particularly like the strip, it just seemed a bit... breif.
Peer Review and Quality Control
Teasswill Posted Apr 26, 2003
This thread seems to have sparked off some similar sentiments to those in the H2G2 Legends & Heavyweights thread.
A few thoughts: I don't feel that omissions are as bad as inaccuracies. Even with the wealth of expertise available amongst researchers, I doubt that many of them are the world authority on their subject. Thus I don't think there is a way to ensure total accuracy. Perhaps we should have a clear caveat to that effect.
I just wonder sometimes, are we taking it all too seriously? Thinking of the entry on the Earth in the original guide, one could hardly call that full & accurate.
Peer Review and Quality Control
anhaga Posted Apr 26, 2003
"I just wonder sometimes, are we taking it all too seriously? "
I agree Teasswill. I think that may have been one of my points in a previous post (but I'm not sure; I confuse myself) But, gee, I had the impression that a few researchers were the world authority on every subject.
Peer Review and Quality Control
Hoovooloo Posted Apr 26, 2003
"are we taking it all too seriously?"
Put pithily, the question is - do you care?
If not, and if the general consensus is that we don't and shouldn't, then this conversation is redundant. We just play at compiling the guide, and don't worry about making it accurate.
It's just that I *did* care when I was writing entries. And even now, disillusioned though I've become with the process to the point where I'm not putting stuff in PR any more, I *still* care, for reasons I can't adequately articulate. Which is why I started this thread.
I just think if you're going to have a site like this at all, it should be the best it can be, because if you're not going to make it that, why bother with Peer Review, the Editing Process and all that at all?
Annoyingly, I don't really have a good answer this time. Longer PR incubation might help. However, there is sometimes an apparent unseemly haste to get entries out of PR. The PR thread for the Console Wars entry is a good example of this. I don't know why this happens - perhaps a Scout can suggest? Is PR that short of stuff? If so, perhaps five entries per day on the Front Page is too much.
Like I say, it comes down to how much you care about quality. If quality of submissions is high, there's no problem. If it suffers, it's difficult to know what to do if the production line is DEMANDING five entries a day without fail.
Just another worth, I still haven't any answers...
H.
Peer Review and Quality Control
anhaga Posted Apr 26, 2003
Hoovooloo:
although I may seem to be falling off both sides of the fence when I say this, I do care very much as well about the accuracy of the entries I write. I also care very much about the accuracy of the information I try to provide in Peer Review. If I'm not sure I either post as a question or don't post. I think the fact is that no matter how much we try, there will be errors. Yes, we should all try as hard as we can but we should probably resign ourselves to the fact that we're always going to be saying " how did that get through?!"
I don't know if we can make a better system than the one in place. Hopefully we can all use the system more effectively.
In the world of scholarly publication, where Peer Review is done by professionals who are world experts in the field, errors creep in; we should probably feel pretty proud of the really fine entries this band of amatures has produced.
Key: Complain about this post
Peer Review and Quality Control
- 21: J (Apr 26, 2003)
- 22: J (Apr 26, 2003)
- 23: There is only one thing worse than being Gosho, and that is not being Gosho (Apr 26, 2003)
- 24: anhaga (Apr 26, 2003)
- 25: J (Apr 26, 2003)
- 26: SEF (Apr 26, 2003)
- 27: Whoami - iD dislikes punctuation (Apr 26, 2003)
- 28: There is only one thing worse than being Gosho, and that is not being Gosho (Apr 26, 2003)
- 29: J (Apr 26, 2003)
- 30: J (Apr 26, 2003)
- 31: anhaga (Apr 26, 2003)
- 32: J (Apr 26, 2003)
- 33: Mikey the Humming Mouse - A3938628 Learn More About the Edited Guide! (Apr 26, 2003)
- 34: J (Apr 26, 2003)
- 35: Teasswill (Apr 26, 2003)
- 36: anhaga (Apr 26, 2003)
- 37: Teasswill (Apr 26, 2003)
- 38: Hoovooloo (Apr 26, 2003)
- 39: anhaga (Apr 26, 2003)
- 40: Hoovooloo (Apr 26, 2003)
More Conversations for Ask h2g2
Write an Entry
"The Hitchhiker's Guide to the Galaxy is a wholly remarkable book. It has been compiled and recompiled many times and under many different editorships. It contains contributions from countless numbers of travellers and researchers."