A Conversation for Design for a New Peer Review System

Peer Lottery Picking

Post 1

Monsignore Pizzafunghi Bosselese

I'd like to a) share a few observations, and b) offer a proposal.


Observations (I'd love to be told that I'm wrong!):

1. Peer Review has calmed down a lot. There have been days or even weeks when PR was featured under the top 5 busies conversations, but not recently.

2. More and more threads are 'unscouted' and even un-commented upon.

3. There is a tendency to be overly, sometimes deadly, serious in the discussions.

4. Peers are screening the *first* page for interesting subjects and may or may not leave a comment there.

5. They don't go deeper.

6. Thread 'owners' are pushing up their threads just to get attention from the latter.


To sum it up, the first page of PR is the most 'interesting' one, for both authors and (non-Scout) peers. The proposal is aimed at

a) creating more traffic on PR,
b) luring peers deeper into PR,
c) stirring up the pot.


Proposal:

What if there was a lottery among the researchers who are active on PR... with the following rules:

1. Any posting to the PR is a lottery ticket, provided the following is true:

1.a. You posted to a thread which does not belong to one of your own entries.

1.b. At the time of posting, you are not a Scout.

1.c. The posting was made during the last 7 days, calculated from the time of drawing. However, a ticket will remain valid until at least one drawing is made.

1.d. You are replying to a posting from somebody else (consecutive postings count as one).

1.e. At the time of posting, the thread has not been 'seconded' by a Scout for removal from PR.

1.f. No other rules apply. A single smiley counts.


2.a. An anti-spamming rule may be put into place if there is reason to assume cheating (especially with regard to rule 1.f. above). Details of this rule are yet to be defined.

2.b. In order to distribute comments over all the threads, there may be further rules (perhaps, counting only "x" postings per thread and day as tickets).


3. A ticket counts double (that is, gives you two chances to be drawn) if the previous posting was older than four days.


4. If one of your tickets is drawn, then:

4.a. Your price is to be awarded a Scout's pick, ie: to select an entry for inclusion into the Edited Guide.

4.b. The Scouts' picking rules apply (none of your own/close friends' entries, 1 week on PR, etc.).

4.c. The winning ticked is taken out of the pool and won't participate in further drawings.

4.d. Any other of your tickets retain their chance as long as they aren't taken out according to rule 1.c.

4.e. Your pick is submitted by email to [Chris?]

4.f. If your pick should be rejected by the Towers then you may have further tries.


5. A drawing will be done as soon as a Lottery Pick is available. Picks are made available to the lottery by

5.a. Donation from a Scout.

5.b. A Scouts inability or failure, for whatever reason, to submit their pick in time.

5.c. Reducing Scouts' quota down to *two* picks per turn. (this would hurt smiley - yikes )

5.d. Inserting two or three 'dummy' Scouts into the roster, in place of two 'real' Scouts, at the next time a place is free. The due dates for their picks are distributed over the calendar such that at least one drawing per week can be made and it is guaranteed that every ticket gets at least one chance to be drawn.

----------

Q. to the Towers: is this feasible at all? There is some programming necessary to determine whether a posting is a valid ticket, to perform the drawings etc...


----------
Would the lottery only draw nonsense comments into PR? Is it desirable?

Your opinions please smiley - smiley


Bossel


Peer Lottery Picking

Post 2

HappyDude

smiley - huh


Peer Lottery Picking

Post 3

$u$

I have to concur with HappyDude here...smiley - huh


Peer Lottery Picking

Post 4

Monsignore Pizzafunghi Bosselese

Now it's my time to say smiley - huh.

I'm afraid the times are coming back which made me write this one: http://www.bbc.co.uk/h2g2/guide/A487253


Peer Lottery Picking

Post 5

Azara

I'd noticed that Peer Review had fallen out of the top 5 conversation list in the last while (I joined in early May, and I think that for May, June and July PR was nearly always in the top 5 list). I wasn't sure if that was just a temporary thing, though - September is back to school or college time for a lot of researchers, who may take a little while to settle in and start contributing again. And current events have been fairly distracting lately!

Anyway, Bossel, I think that your lottery would be a great idea, if it was possible to implement fairly easily. I spend most of my time here on Peer Review, but I don't want to become a scout. But the chance of making an occasional pick (without carrying the scout's responsibility permanently) sounds like a lot of fun.

Azara
smiley - rose


Peer Lottery Picking

Post 6

HappyDude

I think the problem is that the non-frames option is the default setting - many new researchers may not realize the depth of the hread.


Peer Lottery Picking

Post 7

I'm not really here

I think frames is the default setting, I had never even seen no frames until I swopped after the BBc took us over.


Peer Lottery Picking

Post 8

Mark Moxon

Non-frames is the default setting and has been since we joined the BBC, but both types of display suffer from the same issue, which is not going deeper than the first page.

This is a fun idea, Bossel, but we won't be doing any programming on Peer Review until the new Review Forums system is in place. We are about halfway through implementing this, and it's looking good. We're hoping that the various ordering options available in Review Forums will get round the problem you describe, which is a very real one; currently you are stuck with one order (date of last posting) and it's a drag to have to sift through all the Conversations.

Without a technical solution to the lottery, then, would it be too confusing for Scouts' picks to be changed from their simple, regular times (as at the moment) to a more random, lottery-based system like this? If it would be wasy to implement this totally manually without any extra effort for the staff, then we'd be only too happy to consider implementing it, but having had a look, I get the feeling that it might be too confusing for us at this end.

What do you think? Any ideas that can help out Peer Review through the traditional quiet period of the summer are really welcome, but our internal efforts are currently on implementing the scheme describe in the entry this Conversation is attached to, and we're hopeful that the ordering system described in the proposal will help to change the way Peer Review functions.

Anyone any ideas on this score?


Peer Lottery Picking

Post 9

.

I think it sound all right, but confusing. smiley - smiley

I don't have the faintest clue how it would work, but there would need to be some good way of telling the scouts when to make picks. With the current system I know that every four weeks I can say to my mum "Budge up I'm going to spend hours in front of the computer now 'cos if I don't recommend some entries by tomorrow I'll be in big trouble". Sometimes I'm away from h2g2 for a long time though, and if I "won the lottery" smiley - laugh and had to pick entries during that time, I would be away and wouldn't know I had to do it. smiley - erm

Or something...smiley - weird


Peer Lottery Picking

Post 10

Martin Harper

It's a shame Fragilis's "view at h2g2" seems to have gone missing... smiley - sadface They were really useful...

Counting threads, there are about the same number in the last View(25 July) as now, but there are about fifty fewer in Peer Review. That's mostly an artefact of the new system, I expect.

The longest a thread has gone without comment is twelve weeks, compared to four weeks when the stat was first gathered, and six weeks on 25 July. And 61 threads have received a comment in the last week, compared to 80 on 25 July. Ouch.

Over to you... smiley - winkeye


Peer Lottery Picking

Post 11

Mark Moxon

Nadger - we're working on an automated reminder system for Scouts' picks, but it won't happen until after the Peer Review system is in operation. It's on the cards! smiley - biggrin

Lucinda - we've been specifically not putting Scout-found entries into Peer Review in an attempt to drive some of the more ancient Peer Review entries into the Edited Guide (and it seems to be working). Coupled with the increased ease of moving threads into the other Review Forums, that hopefully accounts for the lower number of threads. As for the lower figures in Scout commenting, though, that's something we should look at... smiley - erm

And you're right, it's a shame Fragilis' columns have disappeared. They were invaluable. smiley - sadface


Peer Lottery Picking

Post 12

.

smiley - erm I thought there was already an automated reminder system, doesn't the yahoogroups stuff count?

Unless you mean for the lottery...smiley - erm


Peer Lottery Picking

Post 13

Mark Moxon

Yeah, there's the automated reminder on Yahoo Groups, but it doesn't chase you up if you then fail to make your picks... which is the extra thing we're thinking of adding.

Small thing, very useful though! smiley - smiley


Peer Lottery Picking

Post 14

.

Oh a chase-up-device-thingy would be very very useful...maybe...smiley - smiley


Key: Complain about this post