Re: [boost] Re: [administrative] Formal Review Process
On Mon, 26 Jan 2004 22:10:23 -0800, Thomas Witt wrote
Jeff Garland wrote: I don't like the idea of a mailing list to solve this problem.
Why? Really I just want to know.
Because the primary issue is a lack of time and I don't see how complicating things with another mailing list solves that problem. Also, if review comments either intentionally or accidently wind up on a review list then there is a third list I now have to search when looking into the archive -- I've had to look on the dev and user groups a few times because I can't remember where a particular discussion occurred.
There is more to the filtering than meets the eye. The usual process up to know was that somebody posted a request. That request usually lead to some discussion. As a result in quite a few cases the
Which is one reason why I believe this needs to stay on the dev list...
library author decided to tackle the issues before the request. I've been following these discussions in order to see whether the request is likely to be withdrawn or not. The net effect of all this a lot of postings use review in the subject line and are neither requests nor formal review related. I am not criticising poeple for doing this I just try to present the current situation.
Yep.
Another issue that is related to this problem is that we are already close to having more review requests than we can handle. (I am not talking about the wizard here, just the list and the reviewers).
I don't agree. There were more reviews in 2002 (15) than 2003 (~10) by my count. My guess is that about 1 full review per month is about what can be reasonably done. I, for example, have not been a review manager for a year so I don't feel over-worked.
In the past we already had reviews where there was very little feedback.
That is an issue for the review manager to handle. I would leave it up to the judgement of the review manager to decide if there has been enough participation. If the participation is too low then the library should probably be rejected for lack of interest. But again that is the judgement of the review manager to make. As for the number of review requests,
In my opinion we need some kind of staging in order to streamline the influx of libraries in the formal review process and to improve the quality of reviewed libraries. Thus making failure for trivial reasons more unlikely.
I don't recall any specific examples of this. We already have a multi-stage review process. Sure there are examples of people just joining and throwing out a library for formal review before it is ready, but someone then provides guidance on the process.
I think the idea of a separate mailing list fits into this staging strategy. I.e. the review list would be reserved for review and review only. One idea might be to require a pre review on the developers list or required 2-5 people to second the review request.
Sure, but it complicates life for users trying to decide which lists to subscribe to, post to, and search. There won't be anything to stop someone from posting a half-baked library on the new list or accidentally posting a review request on the dev list...
I think some simple additions to the submission process would solve the 'heads-up' to the review manager problem: A copy of the Review Request posting should be sent directly to the review wizard (email here). If the review wizard does not respond with an acknowledgement within 48 hours another request should be made.
As explained not every request leads to a review.
Sure, but at least the submitter would be aware that you are paying attention. So the acknowledgement is not a date for the review, but just a little "hello, I see your request, have you read the submission guidelines" sort of mail.
Also, note that the entire discussion of the Review Wizard is on the Formal schedule the review.
I would really appreciate it if you could make that change.
Sure.
Finally, it seems to me the real solution is to either split the duties or find another review wizard volunteer.
As of now I don't have a good idea how things can be split up.
Perhaps you can offload the monitoring or organization of a particular upcoming review -- I'm sure we can figure something out.
I'm willing to spend a couple hours a month on this if it would help, so Thomas send me an email off-list if you have something you want to offload on me.
Thanks Jeff! I'll shoot you an email as soon as I see clearer.
Let me know. Jeff
"Jeff Garland" <jeff@crystalclearsoftware.com> writes:
On Mon, 26 Jan 2004 22:10:23 -0800, Thomas Witt wrote
Jeff Garland wrote: I don't like the idea of a mailing list to solve this problem.
Why? Really I just want to know.
Because the primary issue is a lack of time and I don't see how complicating things with another mailing list solves that problem.
*Simplifying* things with another mailing list might help to solve that problem. I know I have a much easier time searching for Boost.Python-specific issues because it has its own list and isn't grouped together with the rest of the Boost list. Likewise, I'm glad that Boost has a separate list from comp.lang.c++.moderated. ;->
Also, if review comments either intentionally or accidently wind up on a review list then there is a third list I now have to search when looking into the archive -- I've had to look on the dev and user groups a few times because I can't remember where a particular discussion occurred.
I guess the question is whether you're more likely to forget. I'm more likely to remember a discussion's list context. But lists.boost.org is indexed by Google so if you want to search all of the lists you know where to turn ;-)
There is more to the filtering than meets the eye. The usual process up to know was that somebody posted a request. That request usually lead to some discussion. As a result in quite a few cases the
Which is one reason why I believe this needs to stay on the dev list...
I don't understand why discussions that are part of a review shouldn't take place on a review list.
library author decided to tackle the issues before the request. I've been following these discussions in order to see whether the request is likely to be withdrawn or not. The net effect of all this a lot of postings use review in the subject line and are neither requests nor formal review related. I am not criticising poeple for doing this I just try to present the current situation.
Yep.
Another issue that is related to this problem is that we are already close to having more review requests than we can handle. (I am not talking about the wizard here, just the list and the reviewers).
I don't agree. There were more reviews in 2002 (15) than 2003 (~10) by my count. My guess is that about 1 full review per month is about what can be reasonably done. I, for example, have not been a review manager for a year so I don't feel over-worked.
In the past we already had reviews where there was very little feedback.
That is an issue for the review manager to handle. I would leave it up to the judgement of the review manager to decide if there has been enough participation. If the participation is too low then the library should probably be rejected for lack of interest. But again that is the judgement of the review manager to make.
Agreed.
As for the number of review requests,
In my opinion we need some kind of staging in order to streamline the influx of libraries in the formal review process and to improve the quality of reviewed libraries. Thus making failure for trivial reasons more unlikely.
I don't recall any specific examples of this. We already have a multi-stage review process. Sure there are examples of people just joining and throwing out a library for formal review before it is ready, but someone then provides guidance on the process.
I agree again.
I think the idea of a separate mailing list fits into this staging strategy. I.e. the review list would be reserved for review and review only. One idea might be to require a pre review on the developers list or required 2-5 people to second the review request.
Sure, but it complicates life for users trying to decide which lists to subscribe to, post to, and search. There won't be anything to stop someone from posting a half-baked library on the new list or accidentally posting a review request on the dev list...
I think we could reasonably provide a more-complete tutorial guide to list subscriptions that would walk people through the subscription steps they'll want to take.
I think some simple additions to the submission process would solve the 'heads-up' to the review manager problem: A copy of the Review Request posting should be sent directly to the review wizard (email here). If the review wizard does not respond with an acknowledgement within 48 hours another request should be made.
As explained not every request leads to a review.
Sure, but at least the submitter would be aware that you are paying attention.
That is really, really important.
I'm willing to spend a couple hours a month on this if it would help, so Thomas send me an email off-list if you have something you want to offload on me.
Thanks Jeff! I'll shoot you an email as soon as I see clearer.
Let me know.
Yes, double/triple thanks! -- Dave Abrahams Boost Consulting www.boost-consulting.com
Jeff Garland wrote:
On Mon, 26 Jan 2004 22:10:23 -0800, Thomas Witt wrote
Jeff Garland wrote: I don't like the idea of a mailing list to solve this problem.
Why? Really I just want to know.
Because the primary issue is a lack of time and I don't see how complicating things with another mailing list solves that problem.
I think providing more structure does not neccessarily complicate things.
Also, if review comments either intentionally or accidently wind up on a review list then there is a third list I now have to search when looking into the archive -- I've had to look on the dev and user groups a few times because I can't remember where a particular discussion occurred.
As Dave said Google might help with that.
There is more to the filtering than meets the eye. The usual process up to know was that somebody posted a request. That request usually lead to some discussion. As a result in quite a few cases the
Which is one reason why I believe this needs to stay on the dev list...
I don't think I've said this clearly before. The basic idea is to actually require a posting to the dev list prior to the review request on the review list. I am thinking of something like this: 1) The lib author posts a pre-review request to the dev list asking for interest and comments and people willing to second the request. 2) The author and those who second the request post a review request to the review list. * From this point in time the discussion on the review list is purely administrative. If somebody makes a request the review will be done unless the review manager rejects the library for technical reasons(with the pre-review this should be highly unlikely). 3) A review announcement is made (announce/dev/review) and the review takes place. * I am fine with having the technical discussion on the dev list. 4) The review manager posts the review results (announce/dev/review) The reason why I think the separate list is a good idea is that the administrative discussion is open. This makes it at least possible to spread the workload. Another point is that it should be easier for somebody else to pickup the pieces if somebody gets busy.
library author decided to tackle the issues before the request. I've been following these discussions in order to see whether the request is likely to be withdrawn or not. The net effect of all this a lot of postings use review in the subject line and are neither requests nor formal review related. I am not criticising poeple for doing this I just try to present the current situation.
Yep.
Another issue that is related to this problem is that we are already close to having more review requests than we can handle. (I am not talking about the wizard here, just the list and the reviewers).
I don't agree. There were more reviews in 2002 (15) than 2003 (~10) by my count. My guess is that about 1 full review per month is about what can be reasonably done.
Agreed.
I, for example, have not been a review manager for a year so I don't feel over-worked.
My major concern is that the reviewers get overworked. I doubt most people can handle a review per month. So having more than one review per month means getting more reviewers.
In the past we already had reviews where there was very little feedback.
That is an issue for the review manager to handle. I would leave it up to the judgement of the review manager to decide if there has been enough participation. If the participation is too low then the library should probably be rejected for lack of interest. But again that is the judgement of the review manager to make.
My point is not about rejecting or not. The point is that little feedback might be a result of overworked reviewers
As for the number of review requests,
In my opinion we need some kind of staging in order to streamline the influx of libraries in the formal review process and to improve the quality of reviewed libraries. Thus making failure for trivial reasons more unlikely.
I don't recall any specific examples of this.
I was not referring to bad libs. The idea is that some basic technicla problems (naming doc structure ...) can be dealt with before the review.
We already have a multi-stage review process. Sure there are examples of people just joining and throwing out a library for formal review before it is ready, but someone then provides guidance on the process.
Let's put it this way. what I want to do is formalize what happens most of the time anyway. Thomas
Jeff Garland wrote:
On Mon, 26 Jan 2004 22:10:23 -0800, Thomas Witt wrote
Jeff Garland wrote: I don't like the idea of a mailing list to solve this problem.
Why? Really I just want to know.
Because the primary issue is a lack of time and I don't see how complicating things with another mailing list solves that problem.
I think providing more structure does not neccessarily complicate things.
Also, if review comments either intentionally or accidently wind up on a review list then there is a third list I now have to search when looking into the archive -- I've had to look on the dev and user groups a few times because I can't remember where a particular discussion occurred.
As Dave said Google might help with that.
There is more to the filtering than meets the eye. The usual process up to know was that somebody posted a request. That request usually lead to some discussion. As a result in quite a few cases the
Which is one reason why I believe this needs to stay on the dev list...
I don't think I've said this clearly before. The basic idea is to actually require a posting to the dev list prior to the review request on the review list. I am thinking of something like this: 1) The lib author posts a pre-review request to the dev list asking for interest and comments and people willing to second the request. 2) The author and those who second the request post a review request to the review list. * From this point in time the discussion on the review list is purely administrative. If somebody makes a request the review will be done unless the review manager rejects the library for technical reasons(with the pre-review this should be highly unlikely). 3) A review announcement is made (announce/dev/review) and the review takes place. * I am fine with having the technical discussion on the dev list. 4) The review manager posts the review results (announce/dev/review) The reason why I think the separate list is a good idea is that the administrative discussion is open. This makes it at least possible to spread the workload. Another point is that it should be easier for somebody else to pickup the pieces if somebody gets busy.
library author decided to tackle the issues before the request. I've been following these discussions in order to see whether the request is likely to be withdrawn or not. The net effect of all this a lot of postings use review in the subject line and are neither requests nor formal review related. I am not criticising poeple for doing this I just try to present the current situation.
Yep.
Another issue that is related to this problem is that we are already close to having more review requests than we can handle. (I am not talking about the wizard here, just the list and the reviewers).
I don't agree. There were more reviews in 2002 (15) than 2003 (~10) by my count. My guess is that about 1 full review per month is about what can be reasonably done.
Agreed.
I, for example, have not been a review manager for a year so I don't feel over-worked.
My major concern is that the reviewers get overworked. I doubt most people can handle a review per month. So having more than one review per month means getting more reviewers.
In the past we already had reviews where there was very little feedback.
That is an issue for the review manager to handle. I would leave it up to the judgement of the review manager to decide if there has been enough participation. If the participation is too low then the library should probably be rejected for lack of interest. But again that is the judgement of the review manager to make.
My point is not about rejecting or not. The point is that little feedback might be a result of overworked reviewers
As for the number of review requests,
In my opinion we need some kind of staging in order to streamline the influx of libraries in the formal review process and to improve the quality of reviewed libraries. Thus making failure for trivial reasons more unlikely.
I don't recall any specific examples of this.
I was not referring to bad libs. The idea is that some basic technicla problems (naming doc structure ...) can be dealt with before the review.
We already have a multi-stage review process. Sure there are examples of people just joining and throwing out a library for formal review before it is ready, but someone then provides guidance on the process.
Let's put it this way. what I want to do is formalize what happens most of the time anyway. Thomas
participants (2)
-
David Abrahams -
Jeff Garland -
Thomas Witt