Every year we encounter papers submitted to the CHI conference that are very difficult to review. They are missing information that is essential to giving the paper a fair review, or the information is buried deep in the paper where many reviewers miss it. The authors clearly have not had the reviewer's task in mind when they submit the paper. Reflecting on this made us realize that most submitters, even those who have been previously successful, are not aware of what goes on during the CHI review process. Authors can influence their chances for acceptance by writing in a way that makes the reviewers' and the Papers Chairs' job easier.
When you look at the papers that make it, it may be easy to see why they were chosen -- typically they are well-written, cover interesting material and just plain old "look nice". But why these? How are other papers weeded out? We have participated in a number of program committees over the years, so we figured we'd take a minute to put down a description of how we expect things to work during the CHI96 paper selection process.
The CHI papers review process is far from perfect, even though many volunteers (300 or so) are giving it their best effort. But because of time constraints, reviewers will often simply be unable to spend lots of time on any one paper -- in writing your paper, you must keep this in mind at all times.
There are several phases in the ultimate decision made concerning your paper. Let's look at each of them and see what you might do to help or hurt yourself.
Each year it is up to the Papers Chairs to find a mechanism for objective assignment of papers to reviewers. This year, assignments will be determined mostly by the matching phrases you select. Each reviewer and meta-reviewer has been asked to indicate matching phrases they wish to cover. You can do nothing about the reviewers' self-classification, but you should think carefully about the phrases you provide. Don't choose too many phrases, or your paper will match no reviewer well, and every reviewer so-so. Pick a set that BEST characterizes the way you want your work reviewed.
A secondary input into assignments will be title and abstract. These will be used to assign meta-reviewers and in cases when the matching phrases aren't enough to produce a decision. So make sure that the title is revealing, and that the abstract provides a concise and accurate summary of the paper content.
Our goal is to get your paper reviewed by at least 5 reviewers; however, it may be "optionally" reviewed by 2-4 others. Generally, it seems that more reviews are better; it gives the meta-reviewers more to work with, a more balanced picture of the paper. In addition to the papers they must review, reviewers are sent copies of 5-6 additional papers, from which they choose 3 to do optional reviews of. You can make your paper more likely to be selected for optional review by working carefully on the title, abstract, and visuals, as these are the features a reviewer is likely to browse in deciding to read/review your paper.
Finally, you should try to make sure that any competent CHI expert can at least grasp the main point of your paper. If you are working in a particularly new or narrow subfield, or if by an unfortunate failure in the matching process, your paper is not assigned to any of the experts in your subfield, you want to ensure that the reviewers assigned to your paper can understand it. In fact, the CHI review process is designed to include some reviewers who are not experts in your field. Their role is to make sure that the paper makes sense to the hypothetical "reasonably knowledgeable CHI attendee." Since almost 2500 people attend the CHI conference each year, we don't have room on the program for papers that will only be understood by a few attendees. So make sure that your paper is understandable by both experts in your subfield and by the larger CHI community (a good way to do this is to have colleagues outside your subfield read and comment on the paper before you submit it).
Before you write your paper, look at the form reviewers will be filling out when they evaluate it. This review form is based on the comments actually made by reviewers of last year's papers. Try to imagine how a reviewer might answer each question on the review form concerning your paper: Do you like the result? If not, how can you change the paper to get a better response?
The submission types are not driven by any data or classification scheme for CHI papers, but rather by the intuitions of past and present Papers Chairs. This is something that we hope to collect data about in the future; however, right now you are casting your paper into a classification scheme that is based completely on intuition. Your paper may not fit cleanly into any single submission type. If this is the case, you need to figure out whether you are trying to do too much and thus need to refocus the paper more clearly, or whether you really have a hybrid (or completely novel) paper category.
It's important that you look at your paper objectively. You may think that you are approaching something in a completely novel way, but your reviewers may see this work as a variant of an existing submission type. In that case, they will likely judge the paper according to the criteria for that type, no matter what you say. For example, most reviewers believe that a systems paper should be judged substantially on whether the ideas are implemented as opposed to simply designed (this is a broadly held belief within the CHI community -- that system design is deeply influenced by feedback from the implementation, and that a system concept is very different from a system backed up by implementation). You are not likely to avoid that criticism by calling your paper about an unimplemented system a "system design" paper. On the other hand, if you, for example, applied a business case analysis to some area of CHI, this might be different enough from the existing categories to be worth calling it a new submission type. If you feel that your paper does not fit any existing submission type, please provide a cover letter with some background on the research traditions your paper draws on, so that the Papers Chairs can try to find reviewers who will be able to judge your paper fairly.
You may instead feel that your paper straddles multiple categories. This is permitted, but it is often a sign that your paper is trying to do too much in the limited space available. For example, suppose your paper is a empirical evaluation of a system you have implemented. You may be uncertain whether to call it a systems paper or an empirical paper (or both). It's very unlikely that you can do a good job of describing both the evaluation and the system being evaluated in 8 pages, in a way that reviewers can judge the originality of your system and the rigor of your evaluation. You will most often be better off to choose one or the other as the focus of your paper (possibly submitting two papers), making sure that you cover the essential issues required of that type of paper. You can use whatever space you have left to make connections to the other aspects of your work.
One exception to this rule is if your paper seems to straddle the line between theory and methodology. We provide both submission types because there are papers that are clearly one or the other, but often a good paper in this area will attempt to provide both a methodology and an underlying theory to justify the method.
The next question asks about the mechanics of the paper (writing quality, figures, etc.). CHI papers do not have the luxury of a revision cycle where such mechanical problems can be addressed, so reviewers have pretty high expectations that they are seeing a polished work. Mechanical problems alone should not cause a paper to be rejected, but if the paper is otherwise on the edge, or the reviewer feels that these problems make it too hard for someone to understand the paper, they can make the difference between acceptance and rejection.
The reviewers are also asked to describe what led to their recommendation. This serves as feedback to you about which of the comments the reviewer made are most important for you to deal with. It also provides important information to the meta-reviewer. If the various reviewers disagree about your paper, its helpful for the meta-reviewer to understand which aspects of the paper most influenced their rating.
In addition to synthesizing the substantive comments, your meta-reviewer will generate a summary numerical rating for your paper. In many cases this will be exactly (or nearly) the same as an average of the individual reviewer ratings. However, one of the meta-reviewer's responsibilities is to allow for some sort of weighting based on the soundness of the various reviewers' comments and their reported expertise in the area.
The meeting begins with a list of papers sorted by the meta-reviewer's summary numerical rating. Right next to this rating is the average of the ratings made by all reviewers, along with a measure of rating variability and average reviewer expertise. We usually start by setting some ground rules, e.g., that probably the top 20 papers are "in", and that papers below some moderate cutoff are "out". The specific cutoffs depend a lot on what was submitted and how the papers fared in the review process.
We usually try to deal with the easy cases first. Meta-reviewers are asked to identify any papers below the informal cut-off that they feel warrant discussion, and these are added to a "make sure we discuss" list. Then we start working down from the top. Each paper is summarized by the assigned meta-reviewer, with reference to the meta-reviews. Special attention is paid to papers where the meta-reviewer's summary rating diverges from the average reviewer ratings; the meta-reviewer should have a clear rationale for such cases, and if the committee is not satisfied, the paper will be tabled for further discussion.
The most difficult decisions concern the borderline papers. It is not impossible for a highly rated paper to be rejected, but unless the meta-reviewer missed a fatal flaw, this is very unlikely. It is also not impossible for a low-rated paper to be accepted: a meta-reviewer may give a paper a relatively low rating, but believe that the ideas have enough redeeming value that perhaps it should be accepted, conditional on someone signing up to "shepherd" it (i.e., work with the authors in specific ways to improve this -- constituting an exception to the as-is rule). However, most of the discussion will center on papers with middling ratings. For example, we might end up with 40-60 papers to discuss, from which we want to accept 25-30.
After making the easy decisions, we begin going through these less clear cases. For each paper, the meta-reviewer presents the paper's case as best as possible. In some cases, the committee will feel strongly that the arguments presented warrant acceptance or rejection, and a decision will be made. More often, the committee will table the discussion to get more input. Once we have gone through each paper briefly (e.g., 5 minutes), we break up into smaller groups, where individual papers are discussed at more length. Some meta-reviewers will go off to read a paper in question to provide another opinion; often the paper will be browsed by several members of a sub-group.
This is a critical stage for your paper. Somehow your paper needs to make enough of an impression on one or more meta-reviewers that they are willing to argue your case in front of the group. They may not have the time to really study your paper, so accessibility of the main contributions and arguments is key. This is where writing quality can make or break you. Including material that is easily browsable -- e.g., tables, graphs, figures of one sort or another -- can make a big difference.
At the end of the first day, the committee reconvenes to see where things stand. Hopefully, a few more decisions will be prompted by the sub-group discussions. Other papers will need more work, and normally many papers go back to hotel rooms with meta-reviewers for further study. These papers are often topics of discussion over dinner. Here, making your material seem interesting can be a big win: if a meta-reviewer is interested, he or she is much more likely to bring it up in conversation and either intentionally or not help to make a case for it. Papers that can stir controversy do well here -- if the meta-reviewers can get into a good argument about a paper over dinner, they may conclude that it is a good bet for livening up the conference.
The next morning the committee meets again as a group. Usually we are able to make a few more decisions first thing, based on additional reviews and discussion. In all cases, when a decision is made that is at odds with the meta-review, that meta-reviewer is sent off to update the meta-review so that it reflects the group discussion. During the night, the Papers Chairs will have gone through the remaining candidates and considered issues of balance (e.g., among the various paper categories, to some extent among content, as well as looking for "new blood"), and this input is also factored in.
One sticky issue in the discussions of borderline papers concerns authorship by meta-reviewers. These papers are treated like any other, except that when they come up for discussion, the meta-reviewer author is asked to leave the room; papers written by colleagues of a meta-reviewer are treated in the same fashion. Such papers are considered especially carefully and tend to be held to slightly higher standards. For example, if a paper authored by a meta-reviewer is considered to be in a "tie" with a non-meta-reviewer paper, the committee will usually decide to go with the latter.
We try to finalize our list the morning of the second day. In the afternoon, many meta-reviewers start peeling off, trying to make air connections and such. Those who remain help the Papers Chairs to organize sessions and provide feedback on the process that they think might help next year's chairs improve the process.
The following week there is a meeting of Technical Program Chairs where the content of the conference is finalized. This meeting explores ways to combine multiple submissions to a conference to create the most effective presentation of the work for the conference attendees. After this meeting all submitters are notified about their submissions.
If your paper is accepted, this is only the beginning. You will have about 6 weeks to get a camera-ready copy of your paper to the Publication's Chair for inclusion in the Proceedings. Keep in mind, however, that this period includes the December holidays. Even though your paper is accepted, there are bound to be comments about ways you can improve it. Use the reviewers' comments, tempered by the meta-reviewer's assessment of which of those are the most serious, to improve your paper as much as you can in that relatively short time. The final draft is due in early January. After that you will hear from us a few more times, about your presentation session, your slides, the format of your session, and other details specific to presenting your paper.