A Conversation for Design for a New Peer Review System

Some First Thoughts

Post 1

Monsignore Pizzafunghi Bosselese

Still have to sleep a night over it, but here goes...


I'd like to have a Sort-on-thread-number feature. IMHO, sorting by day of submission is somewhat too coarse for housekeeping purposes which might still be necessary.

'Works in Progress' -- What if authors leave, and leave a 'WIP' behind? There must be a way to put it up for review (at least to the Flea Market) if a researcher or scout thinks it's good enough. Needs a proper definition of AWOL...


The submit button should be disabled as long as the entry is in any of the review places, to prevent it from appearing in different places simultaneously.


How about an automatical 'Scouted thread' feature, which takes into account whether a scout has replied to a thread. In the case of a scout as the initiator, the feature should show correctly that *another* scout has (or has not) visited the thread.


All in all: sounds good smiley - ok


Some First Thoughts

Post 2

Mikey the Humming Mouse - A3938628 Learn More About the Edited Guide!

smiley - biggrin This looks amazing.....

If I didn't live on the opposite side of the globe, I'd be putting a big tin of chocolate chip cookies in the mail for Jim and Mark and everyone else! (As is, they would be nothing but stale crumbs by the time they got there...)

smiley - smiley
Mikey


Some First Thoughts

Post 3

Mark Moxon

Mikey - if only all proposals got that sort of response. smiley - winkeye

Bossel - good points! I'll see if I can work them in there when I next get a moment.


Some First Thoughts

Post 4

Blatherskite the Mugwump - Bandwidth Bandit

First of all, let me say that I absolutely love most of these proposals. Anything that I don't discuss here can be interpreted to have my full support. smiley - ok

One concern I have is with giving authors the ability to move their threads at will. I can see the potential for abuse here... an intractable author could keep posting the same unchanged article into Peer Review, after being told that it isn't acceptable as is. My suggestion to avoid this is to give the Scouts the tools to do this, and we could take care of it for them. Authors should have the ability to pull their articles at whim, or move them to the Workshop, but not Peer Review.

It might also be worth considering how much authority we grant to Scouts in respect to their own submissions. If we did implement this sort of thing, though, it could cause technical problems for Scouts who post the work of others to Peer Review... would they be prevented from recommending articles that they found?

We've had a gentlemen's (and gentlewomen's smiley - winkeye) agreement to refrain from picking articles that have been in PR for less than a week... but that hasn't always been honored. I'd prefer to see the recommendation button hidden for a week.


Some First Thoughts

Post 5

Monsignore Pizzafunghi Bosselese

Good points Colonel!

As to the 'Scouted' feature: a thread should be counted as 'visited by a Scout' only if the scout in question actually *is* a scout, and has not left the roster in the meantime. This would imply that the whole counting/checking thing be triggered whenever the Scouts' list changes.


Some First Thoughts

Post 6

Dancer (put your advert here)

CS said: "One concern I have is with giving authors the ability to move their threads at will."

I fully agree, I think a move to the workshop can be done by the author/submitter, butnot to PR, I think a move to PR should require a fresh thread. this'll help keeping things orderly. I don't think once in the workshop, a thread should be moved back to PR (at least in most cases, on other cases, the Scouts might be the right people to do this).

Also, there's a question of submitter vs. author.
if someone submits someone elses entry, can they pull it out then?
I think that as long as the author of the entry didn't post, they can. After Author's firs posting to thread, author should get the control.

I'll post again when I get some more thoughts together on the subject. Going home to bed now.


Check out all the new Smilies smiley - wizard,
Yours,
smiley - hsif
Dancer


Some First Thoughts

Post 7

SchrEck Inc.

On the moving threads/removing threads discussion - the main objective seems to be that the old discussion thread is closed and moved to the article and that a new one is created in the new forum. I'd consider this a good thing, too. Keep PR tidy! smiley - winkeye

What I'm trying to suggest is, if this could be done automatically when an entry is moved, removing and resubmitting wouldn't be necessary. Furthermore, if moving entries weren't allowed (if the 'move entry' button was just for the scouts), ordinary users could still move it by removing/resubmitting it.


Some First Thoughts

Post 8

Mark Moxon

Bossel...

"Sort-on-thread-number feature" - good idea, I'll add it.

"What if authors leave, and leave a 'WIP' behind" - good point. I've put it in as a question about whether the WIP setting should expire if the Researcher leaves and doesn't come back. Might be worth discussing in a new Conversation here (don't want things to get too clogged up!).

"The submit button should be disabled as long as the entry is in any of the review places" - it already is, though perhaps the wording isn't totally clear...

"whether a scout has replied to a thread" - great idea, I'll add it.


Some First Thoughts

Post 9

Mark Moxon

"What I'm trying to suggest is, if this could be done automatically when an entry is moved, removing and resubmitting wouldn't be necessary. Furthermore, if moving entries weren't allowed (if the 'move entry' button was just for the scouts), ordinary users could still move it by removing/resubmitting it."

This might be a good solution. To recap:

Everyone can move their entries between Review Forums, but moving is (technically speaking) exactly the same as removing and re-submitting. In other words, when someone moves their entry, we ask them for a new introductory paragraph (as the Forum is different and has different rules), and then we move the old Conversation to the entry, and start a new one.

I think everyone should be able to do this if it's their entry - it is, after all, the same as removing and resubmitting, but a bit slicker. The question is: should Scouts be able to do this themselves, or should we retain it as an in-house tool that the Editors implement when the Scouts propose and second thread moving, as is currently the case? I don't have an opinion, to be honest: as long as the entry moving tool posted the right messages to the author, it would probably be OK.

What do you guys think?


Some First Thoughts

Post 10

cafram - in the states.

For the WIP expiry thing, you'd have to define 'leaving and not coming back' - there're people who registered in the first couple of months of h2g2 and who've never been back, who still have accounts (obviously pre-beeb, so not activated) - how are you going to set the deadline of "no, you've been gone too long, don't come back"??

Mark - please check your email so I can send my batch and go to bed!!

smiley - biggrin


Some First Thoughts

Post 11

Mark Moxon

Email checked. :-) This question is now in a new Conversation at http://www.bbc.co.uk/h2g2/guide/F72126?thread=127349 - see you there for the discussion!


Some First Thoughts

Post 12

Mikey the Humming Mouse - A3938628 Learn More About the Edited Guide!

As far as the scouts moving threads themselves, I don't know. It sounds like it would be handy and save us all some time. We would definitely want to make sure that the scouts still waited for either agreement from the author or seconding from another scout before we made the move. However, there have still been situations where 2 scouts have agreed on a move, but have been overruled by y'all italics -- rare, but it happens.

Despite all this, I do think it would be easier for us to keep PR neat and tidy if we didn't need to send an email about each one that needed to be moved.

smiley - smiley
Mikey


Some First Thoughts

Post 13

Blatherskite the Mugwump - Bandwidth Bandit

I think it would be a wonderful time-saver to give the Scouts the ability to move the threads, but it would have to be done in a way that ensures that more than one Scout has agreed to it. Otherwise, we could have things moved willy-nilly, and we'd really p**s off our writers. In regards to the current setup for moving things out of Peer Review, I made this anaology in my Post interview: it's like shoveling snow in a blizzard... it makes a difference, but not much, and not for very long.


Some First Thoughts

Post 14

Mycroft

I can see the potential for abuse with authors being able to move entries into peer review at will, but is there actually any more potential than under the current system? As I understand it a thread currently stays in peer review until it's either moved to the sin bin or the entry gets recommended, but there's nothing to stop anyone creating a new thread for an entry (or simultaneous threads) in peer review as soon as the old thread gets binned. I assume that this isn't being widely abused at the moment, and if I'm right in my assumption is there any reason to believe that this will change if a new system is implemented?


Some First Thoughts

Post 15

Monsignore Pizzafunghi Bosselese

Good point Mycroft, and well pointed out. We shouldn't get paranoid about that.


Some First Thoughts

Post 16

Blatherskite the Mugwump - Bandwidth Bandit

People get weird when things are made easier for them. If they have to make a new conversation, it's more bother than it's worth. If they could move the same conversation thread back to Peer Review with a click of the button, then it's more prone to abuse.

But now I'm getting the feeling that I've misinterpreted things, and that it doesn't work quite like this. Can we have some clarification on exactly how things are moved, Mark?


Some First Thoughts

Post 17

Monsignore Pizzafunghi Bosselese

Oh, I think that was explained at ... darn! Dunno! The 'move' actually is two things:
a) move the existing thread, from wherever it is, to the entry
b) creation of a *new* one at the suggested place

That way, you don't have to wade through 'old' postings which most likely don't reflect the current contents of the entry in question smiley - smiley


Some First Thoughts

Post 18

Monsignore Pizzafunghi Bosselese

Yeah, the 'move' thing is explained in posting 9 above smiley - smiley


Some First Thoughts

Post 19

Mycroft

And that's another thing... I know things could be easier if the thread got created anew whenever an entry got moved, but it might also cloud the issue. I suspect the situation will frequently arise where suggestions which have already been discounted in another peer review thread will independently get repeated by someone else because almost no-one would check more than one thread to see if they're going over old ground. And what happens to the redundant thread when the entry moves to another review forum? Couldn't this mean entry forums ending up with half a dozen identically named threads? I think I'd prefer to see a single thread which follows an entry wherever it goes but I doubt I've thought it through sufficiently.

Whichever way it goes, I've a couple more ill-thought out ideas for discussion. I've a vague feeling that it might make life easier to mirror review threads, so that an entry's review thread is accessible both from the entry's forum and the relevant review forum. Also, if there are to be multiple review threads which end up getting place in the entry's forum, it might be useful to append the review area they came from to the thread name.

Incidentally, is there any mechanism for adding to edited entries? I occasionally come across existing entries about which I know a fair bit more than is already there, but I've no idea where or to whom I should post to update the entry, or even if it's possible.


Some First Thoughts

Post 20

Mikey the Humming Mouse - A3938628 Learn More About the Edited Guide!

Mycroft wrote: "Incidentally, is there any mechanism for adding to edited entries? I occasionally come across existing entries about which I know a fair bit more than is already there, but I've no idea where or to whom I should post to update the entry, or even if it's possible."

The most common way is to post a comment at the bottom of the entry. Then those who read the entry will also be able to read your comment. If and when the entry ever gets updated by a subeditor, they will have your comment available and will include it if they think appropriate (and give you credit where appropriate as well).

If you've really posted a lot of suggested fixes, additions, etc. to the entry, and think that the whole thing should get a rehaul, you can post the entry to the Update HQ (run by Jimi X with some occasional help from me).

http://www.bbc.co.uk/h2g2/guide/A496451

smiley - smiley
Mikey


Key: Complain about this post

Write an Entry

"The Hitchhiker's Guide to the Galaxy is a wholly remarkable book. It has been compiled and recompiled many times and under many different editorships. It contains contributions from countless numbers of travellers and researchers."

Write an entry
Read more