A Conversation for Design for a New Peer Review System
Peer Lottery Picking
Monsignore Pizzafunghi Bosselese Started conversation Sep 22, 2001
I'd like to a) share a few observations, and b) offer a proposal.
Observations (I'd love to be told that I'm wrong!):
1. Peer Review has calmed down a lot. There have been days or even weeks when PR was featured under the top 5 busies conversations, but not recently.
2. More and more threads are 'unscouted' and even un-commented upon.
3. There is a tendency to be overly, sometimes deadly, serious in the discussions.
4. Peers are screening the *first* page for interesting subjects and may or may not leave a comment there.
5. They don't go deeper.
6. Thread 'owners' are pushing up their threads just to get attention from the latter.
To sum it up, the first page of PR is the most 'interesting' one, for both authors and (non-Scout) peers. The proposal is aimed at
a) creating more traffic on PR,
b) luring peers deeper into PR,
c) stirring up the pot.
Proposal:
What if there was a lottery among the researchers who are active on PR... with the following rules:
1. Any posting to the PR is a lottery ticket, provided the following is true:
1.a. You posted to a thread which does not belong to one of your own entries.
1.b. At the time of posting, you are not a Scout.
1.c. The posting was made during the last 7 days, calculated from the time of drawing. However, a ticket will remain valid until at least one drawing is made.
1.d. You are replying to a posting from somebody else (consecutive postings count as one).
1.e. At the time of posting, the thread has not been 'seconded' by a Scout for removal from PR.
1.f. No other rules apply. A single smiley counts.
2.a. An anti-spamming rule may be put into place if there is reason to assume cheating (especially with regard to rule 1.f. above). Details of this rule are yet to be defined.
2.b. In order to distribute comments over all the threads, there may be further rules (perhaps, counting only "x" postings per thread and day as tickets).
3. A ticket counts double (that is, gives you two chances to be drawn) if the previous posting was older than four days.
4. If one of your tickets is drawn, then:
4.a. Your price is to be awarded a Scout's pick, ie: to select an entry for inclusion into the Edited Guide.
4.b. The Scouts' picking rules apply (none of your own/close friends' entries, 1 week on PR, etc.).
4.c. The winning ticked is taken out of the pool and won't participate in further drawings.
4.d. Any other of your tickets retain their chance as long as they aren't taken out according to rule 1.c.
4.e. Your pick is submitted by email to [Chris?]
4.f. If your pick should be rejected by the Towers then you may have further tries.
5. A drawing will be done as soon as a Lottery Pick is available. Picks are made available to the lottery by
5.a. Donation from a Scout.
5.b. A Scouts inability or failure, for whatever reason, to submit their pick in time.
5.c. Reducing Scouts' quota down to *two* picks per turn. (this would hurt )
5.d. Inserting two or three 'dummy' Scouts into the roster, in place of two 'real' Scouts, at the next time a place is free. The due dates for their picks are distributed over the calendar such that at least one drawing per week can be made and it is guaranteed that every ticket gets at least one chance to be drawn.
----------
Q. to the Towers: is this feasible at all? There is some programming necessary to determine whether a posting is a valid ticket, to perform the drawings etc...
----------
Would the lottery only draw nonsense comments into PR? Is it desirable?
Your opinions please
Bossel
Peer Lottery Picking
Monsignore Pizzafunghi Bosselese Posted Sep 22, 2001
Now it's my time to say .
I'm afraid the times are coming back which made me write this one: http://www.bbc.co.uk/h2g2/guide/A487253
Peer Lottery Picking
Azara Posted Sep 22, 2001
I'd noticed that Peer Review had fallen out of the top 5 conversation list in the last while (I joined in early May, and I think that for May, June and July PR was nearly always in the top 5 list). I wasn't sure if that was just a temporary thing, though - September is back to school or college time for a lot of researchers, who may take a little while to settle in and start contributing again. And current events have been fairly distracting lately!
Anyway, Bossel, I think that your lottery would be a great idea, if it was possible to implement fairly easily. I spend most of my time here on Peer Review, but I don't want to become a scout. But the chance of making an occasional pick (without carrying the scout's responsibility permanently) sounds like a lot of fun.
Azara
Peer Lottery Picking
HappyDude Posted Sep 23, 2001
I think the problem is that the non-frames option is the default setting - many new researchers may not realize the depth of the hread.
Peer Lottery Picking
I'm not really here Posted Sep 24, 2001
I think frames is the default setting, I had never even seen no frames until I swopped after the BBc took us over.
Peer Lottery Picking
Mark Moxon Posted Sep 24, 2001
Non-frames is the default setting and has been since we joined the BBC, but both types of display suffer from the same issue, which is not going deeper than the first page.
This is a fun idea, Bossel, but we won't be doing any programming on Peer Review until the new Review Forums system is in place. We are about halfway through implementing this, and it's looking good. We're hoping that the various ordering options available in Review Forums will get round the problem you describe, which is a very real one; currently you are stuck with one order (date of last posting) and it's a drag to have to sift through all the Conversations.
Without a technical solution to the lottery, then, would it be too confusing for Scouts' picks to be changed from their simple, regular times (as at the moment) to a more random, lottery-based system like this? If it would be wasy to implement this totally manually without any extra effort for the staff, then we'd be only too happy to consider implementing it, but having had a look, I get the feeling that it might be too confusing for us at this end.
What do you think? Any ideas that can help out Peer Review through the traditional quiet period of the summer are really welcome, but our internal efforts are currently on implementing the scheme describe in the entry this Conversation is attached to, and we're hopeful that the ordering system described in the proposal will help to change the way Peer Review functions.
Anyone any ideas on this score?
Peer Lottery Picking
. Posted Sep 25, 2001
I think it sound all right, but confusing.
I don't have the faintest clue how it would work, but there would need to be some good way of telling the scouts when to make picks. With the current system I know that every four weeks I can say to my mum "Budge up I'm going to spend hours in front of the computer now 'cos if I don't recommend some entries by tomorrow I'll be in big trouble". Sometimes I'm away from h2g2 for a long time though, and if I "won the lottery" and had to pick entries during that time, I would be away and wouldn't know I had to do it.
Or something...
Peer Lottery Picking
Martin Harper Posted Sep 25, 2001
It's a shame Fragilis's "view at h2g2" seems to have gone missing... They were really useful...
Counting threads, there are about the same number in the last View(25 July) as now, but there are about fifty fewer in Peer Review. That's mostly an artefact of the new system, I expect.
The longest a thread has gone without comment is twelve weeks, compared to four weeks when the stat was first gathered, and six weeks on 25 July. And 61 threads have received a comment in the last week, compared to 80 on 25 July. Ouch.
Over to you...
Peer Lottery Picking
Mark Moxon Posted Sep 25, 2001
Nadger - we're working on an automated reminder system for Scouts' picks, but it won't happen until after the Peer Review system is in operation. It's on the cards!
Lucinda - we've been specifically not putting Scout-found entries into Peer Review in an attempt to drive some of the more ancient Peer Review entries into the Edited Guide (and it seems to be working). Coupled with the increased ease of moving threads into the other Review Forums, that hopefully accounts for the lower number of threads. As for the lower figures in Scout commenting, though, that's something we should look at...
And you're right, it's a shame Fragilis' columns have disappeared. They were invaluable.
Peer Lottery Picking
. Posted Sep 26, 2001
I thought there was already an automated reminder system, doesn't the yahoogroups stuff count?
Unless you mean for the lottery...
Peer Lottery Picking
Mark Moxon Posted Sep 26, 2001
Yeah, there's the automated reminder on Yahoo Groups, but it doesn't chase you up if you then fail to make your picks... which is the extra thing we're thinking of adding.
Small thing, very useful though!
Key: Complain about this post
Peer Lottery Picking
- 1: Monsignore Pizzafunghi Bosselese (Sep 22, 2001)
- 2: HappyDude (Sep 22, 2001)
- 3: $u$ (Sep 22, 2001)
- 4: Monsignore Pizzafunghi Bosselese (Sep 22, 2001)
- 5: Azara (Sep 22, 2001)
- 6: HappyDude (Sep 23, 2001)
- 7: I'm not really here (Sep 24, 2001)
- 8: Mark Moxon (Sep 24, 2001)
- 9: . (Sep 25, 2001)
- 10: Martin Harper (Sep 25, 2001)
- 11: Mark Moxon (Sep 25, 2001)
- 12: . (Sep 26, 2001)
- 13: Mark Moxon (Sep 26, 2001)
- 14: . (Sep 27, 2001)
More Conversations for Design for a New Peer Review System
Write an Entry
"The Hitchhiker's Guide to the Galaxy is a wholly remarkable book. It has been compiled and recompiled many times and under many different editorships. It contains contributions from countless numbers of travellers and researchers."