This document aims to describe a system to replace the currently very clunky and manually-run Peer Review system. It's an open forum for discussion about the system, and the aim is to end up with a specification that will work for the in-house staff and for the users of the site.

If you have any comments, questions or ideas, please post them in the Forum below. Feel free to say what you think!

The Family of Review Forums

The system is based round a whole family of Review Forums, which all work in the same sort of way. Currently we only have the Review Forums (if you ignore the Sin Bin, which this system aims to make a thing of the past):

• Peer Review
• The Writing Workshop

but in the new system we will be able to add multiple pages to the system, each covering a different aspect of collaborative reviewing and writing. Suggestions for the initial list are (mainly thanks to Bossel and Lucinda's suggestions):

• Peer Review, for entries that are finished in the author's eyes and are ready for the Edited Guide.

• The Collaborative Writing Workshop, for entries that are currently at very early stages (like the skeleton pages used to kick off This Week's Topic). The ensuing discussions can be incorporated into the entry by the original author, and the relevant Researchers credited, and eventually genuinely collaborative entries will appear.

• The Flea Market, which is where entries from other Review Forums can end up if their author has disappeared and we're looking for people to step in to make them excellent entries... or, indeed, entries from the rest of the Guide which are nearly there, but which need help finishing off.

• The Alternative Writing Workshop, for entries which don't fit the guidelines for the Edited Guide, but which the authors would like discussed/added to anyway.

• This system also allows Peer Review to be split up in the future into multiple subject areas, say one for each major subject in the Life/Universe/Everything system. This doesn't mean this is a good idea, but it does mean we can debate it when Peer Review gets too big to be manageable in one Forum.

Not for Review

Currently anyone can put anything into a Review Forum, which works well unless the author doesn't want their entry to be put into a Review Forum in the first place (which does happen, albeit rarely).

To get around this, we add a tickable box to the entry editor to enable authors to mark an entry as 'Not for Review', in which case nobody can put it into a Review Forum except the author.

Putting Entries into the System

Assume we are looking at an entry. One of three things happens:

• If the entry is not available for review, 'Not for Review' is displayed in the Entry Data box.

• If the entry is already in a Review Forum, it says the Forum name in the Entry Data box, along with a link to that Review Conversation.

• Otherwise there's a 'Submit for Review' button in the Entry Data box.

Clicking the 'Submit for Review' button takes you to the Review Forum submission page. This consists of:

• A text box, into which I type the relevant explanation of why I'm putting this entry into a Review Forum (eg 'I could do with some input', 'I'd like to put this up for the Edited Guide' etc etc). No other information is required - no links or entry titles, unlike the current Peer Review system! This text box must be filled in - it can't be left blank.

• A drop-down menu, initially set to 'Choose one', which lists all the available Review Forums. I have to pick the Review Forum into which I want to submit my entry.

• Some explanatory blurb, which briefly describes the function of each Review Forum, along with some handy links (eg to Writing Guidelines and so on).

• A 'Submit' button, which submits the entry to the relevant Review Forum.

Clicking on the 'Submit' button with all the relevant information correctly filled in will put that entry into the relevant Review Forum. If the entry is recommended by someone other than the author, it will also post a message to the author's Personal Space to inform them that their entry is being reviewed, and which will point to the Review Conversation.

Moving, Removing and Re-starting Reviews

Those who put their entries into Review Forums should definitely be able to remove them too, in which case the Review Conversation simply gets moved to the Conversation Forum for that entry. This should be done if the author wants to do some work on the entry and doesn't want it to be picked from Peer Review before they've done it - we need Peer Review to only be populated by Conversations about entries that are pickable, as far as the author is concerned.

Using this system, people can 'restart' Review Conversations by removing their entry from Peer Review, and then resubmit it. This would have the effect of moving the existing Review Conversation to the entry itself, and then starting a new, blank Review Conversation in the relevant Review Forum. This is particularly handy when something is moving from a collaborative writing environment into Peer Review itself, when (in theory) all the suggestions have been take on board and aren't relevant to the 'final' piece being put up for review.

Whenever a Review Conversation is moved to the entry (whether because the author has removed the entry from a Review Forum, or because a Scout has recommended it), the subject of the Conversation will be appended with the name of the original Review Forum. So 'A123456 - Entry Title' in Peer Review would be renamed 'Peer Review: 123456 - Entry Title' when moved.

Review Forums

Now for the meat of the system: the Review Forums.

Going to a Review Forum (eg /PeerReview) will display a page containing a list of the entries in that Forum (probably in batches of 25 or so, much like the Conversation list for normal Forums). This list will initially be shown with the most recently-posted-to Conversations at the top, so visitors can easily jump to the latest action, but it can also be ordered by any of the items of information shown in the list. This includes:

• The entry's A123456 number
• The entry's title
• The author's name
• The author's number
• Date of last posting in the Review Conversation
• Date that the entry was put into this Review Forum
• Unique Conversation ID (handy for housekeeping)
• Whether a current Scout has posted to those Review Conversations (not including the initial Posting so this also flags unscouted Conversations for entries written by Scouts) - this part only viewable by Scouts, as it's a tool for scouting, not for reviewing.
• A totally random ordering, to facilitate Scout picking.

The ordering should be specifiable in the URL (like ?order=DateCreated) so Researchers can add those specific orderings to their browser's hotlists, removing the need for ordering preferences to be implemented.

When an entry is submitted to a Review Forum, a new Review Conversation is created, much like starting a new Conversation in a traditional Conversation. The subject of the initial posting is set to 'A123456 - Entry Title', a link to the entry is automatically inserted at the start of the initial Posting, and the text that the submitter typed in before is put in after that.

Although the Review Forum looks and acts like a normal Forum, it won't have a 'New Conversation' button; the only way to create a new Conversation is to submit one's entry through the proper channel.

Recommending Entries

Some Review Forums, notably Peer Review, exist to provide entries for Scout recommendation. For entries that have been put into certain Forums (such as Peer Review), a 'Recommend Entry' button will appear in the Entry Data box for that entry, but only for Scouts. Clicking this will pop up a box into which the Scout can type his/her comments, and this will then feed into an in-house tool for the Editors to process that pick.

It would be easy to prevent the 'Recommend Entry' button from appearing for entries that have only been in the relevant Review Forum for a certain period, eg less than a week. This amount of time will be variable, so we can 'force' picks to be done from the older Review Conversations if we need to.

If the recommendation is approved by the Editors, then the effect will be to put the entry into the Subs' queue, and to move the relevant Review Conversation to the entry itself. An automatic message will also be posted to that Conversation to explain that it has been picked.

Other Things to Add

Most of the in-house system to automate the rest of the process has been written and will be appearing very shortly, but the following have not been looked at yet:

An automatic 'Coming Up' page that displays all entries in the Editorial Process, and their status (eg recommended entries that aren't with the Subs, entries that are with the Subs, entries that are back from the Subs but haven't been published yet).

We need a system to automate the quota of Scouts' picks, as currently the dates and quotas are completely manually maintained (and it's a pain!).

We need a system that automatically flags entries that have been out with the Subs for too long.

We could do with a system to automate the Picture Library, as this is one of the few manual aspects of the Editorial Process left.