Now, that I am partially confident about putting the guidelines I used out, here it is. This guide is aimed at the structural aspects of a good review. I agree that we should keep undergrads like me out of the reviewing process, since we lack the domain knowledge aand experience to provide a good review.

These guidelines are directly sourced from the incomplete list in the references section. I’ve added the structure after I decided to post this online, so if I missed your blog/slides but sourced from it, please let me know. I’ll gladly cite it. Apologies!

Table of Contents

• Table of Contents
• Bare Basics
  • Audience
  • Ethics
• Philosophy
  • Prevent Bad Work
  • Writing the review
  • Evaluation Aspects
• Paper Structure
  • Abstract
  • Introduction
  • Related Work
  • Proposed Approach
  • Results
    • Common issues faced:
  • Conclusions
• Review Structure
  • Summary
    • Common mistakes:
  • Positives
  • Negatives
    • Major Issues
    • Minor Issues
    • How to structure?
  • Avoid in reviews
  • Re-review process (Post-rebuttal Phase)
  • Handling Policy Violations
  • Supplemental material
• References

Bare Basics

Audience

There are three audiences for your review: Area Chairs, Authors and Community.

Area Chairs (Primary): They need clearly justified guidance for paper accept/reject decisions and critical reviews.

Authors (Secondary): They need fair consideration and constructive feedback on what directions should they build upon (regardless of acceptance).

Community (Long term): You should help your community, ensure every submitted paper teaches something worthwhile.

Ethics

No conflict of interest: Before accepting an invitation to review, you better be confident in your ability to assess the paper impartially. You cannot impartially review of your advisor’s paper, your current or recent boss’ paper. You should be impartial towards competing groups’ work, or a colleague’s paper etc. The best method is to put conflict domains as per guidelines to partially automate some aspects of this.

Impartiality: You might be be asked to review a paper that deals with a problem that you would like to be solved. Obviously, it is a huge breach of confidentially, etiquette, and professional standards, not to mention being incredibly nasty, to steal an idea from a paper that you are asked to review. The community needs to respect everyone’s contributions to the highest degree. Please, don’t be that reviewer!

Epistemic Kindness: Be kind to ideas you disagree with. Kindness to positions you want to dismiss as crazy and dismiss with insults and mockery. Kindness that breaks you out of your own arrogance, makes you realize the truth is more important than your own glorification, especially when the authors have spent months of effort, there is a lot at stake. When a paper proposes an idea which you disagree with, before you dismiss and slap yourself five for your brilliant takedown, take a second to consider it fairly on its own terms.

Philosophy

Remember that someone else put a huge amount of work into this paper and their career/livelihood depends on them getting papers published in a reasonable amount of time. If you think the paper should be rejected, do it quickly! If you think it should be accepted, do it quickly!

Prevent Bad Work

• Papers which have methods, design, results, or claims are blatantly false.
• Papers which have major flaws that cannot not be corrected.
• Papers which are clearly far inferior than the current state of the art.

Put a strong reject if: You think the paper is uninteresting and you wouldn’t accept it even if they did everything you recommend. This helps authors explore better directions than the current one.

Writing the review

I saw people generally follow these guidelines:

• Being concise - Nothing extraneous
• Being precise - State specific problems with the manuscript, be concrete!
• Being constructive - State how problems you found can be addressed by authors
• Being polite - Focus on real issues rather than pet peeves.

Specifics are discussed later, for eg: Don’t point out typos (rule 4).

Evaluation Aspects

Evaluate aspects of paper and give concrete feedback to make the work better:

• Is it in line with the journal/conference’s scope?
• Claims are new and unambiguously stated.
• Acknowledgement is given to the existing body of work.
• The improvements can be attributed to the method and not some other change.
• Results presented have been interpreted correctly and all interpretations considered.

Other guidelines:

• Ensure that the results are not too preliminary or too speculative, but at the same time try to not block innovative new research and theories.
• Does your expertise cover all aspects of the article? If not, describe which sections you can respond to and why?
• Is it useful to you (A proxy for the community)?
• Do the claims make sense intuitively?

Paper Structure

First step is trying to understand the paper and what it says.

Here are some expectations to be met by each section of the manuscript, which are checked by a reviewer.

Abstract

• ?

Introduction

This section should motivate the need for the proposed approach.

• Does the introduction clearly identify (1) need and (ii) relevance of the proposed work?

Related Work

This section should convince readers of novelty if it’s not obvious.

• Has the authors covered very similar works and differentiated their work from the ones that look similar?

Proposed Approach

• Does the methodology target the main claims appropriately?
• How complex is the presented model?

Results

• Are the results clearly and logically presented
• Are they justified by the data presented?
• Are the figures clear and fully described?

Common issues faced:

• Insufficient data
• Statistically non-significant variations
• Unclear data tables
• Contradictory data (Not self-consistent or disagree with the conclusions)
• Confirmatory data that adds little, if anything, to current understanding (unless strong arguments for such repetition are made)

Conclusions

• Do the conclusions justifiably respond to main questions the author(s) posed?
• Do the Conclusions go too far or not far enough based on the results?

Review Structure

A review would typically consist of 4 parts: (i) Summary (ii) Positives (iii) Negatives (iv) Overall

Is the manuscript’s story cohesive and tightly reasoned throughout?

If not, where does it deviate from the central argument?

Summary

• You should find the parts you think are most relevant and those should constitute the summary.

Common mistakes:

• The summary should absolutely not be a restatement of the abstract of the paper
• Summary is critical because if you can’t distill the ideas down then you haven’t really understood the paper

Positives

Negatives

Try and separate your points into “Major” or “Minor” issues.

Major Issues

Common major issues:

• Claims not supported by the data.
• Methods or result which appear completely incorrect.
• Critical missing pieces of information.
• A paper that is not readable by a person trying their best to understand it.

Minor Issues

Common minor issues:

How to structure?

• Point to specific figures, paragraphs, or results for each major issue and be concrete about the problems. Vague criticisms are unacceptable. When possible, use references from previous literature to back up your claims.
• Not making all data and code available with a specific link and instructions, is a major issue.
• Minor issues should also be a bulleted list. There are a much broader range of minor issues that you may encounter. Some examples include simulations that miss some cases, figures that are missing axis labels, or the paper has extraneous results that aren’t relevant to the claims being made.
• Using a numbered list of points can help the author(s) keep track when responding to your review and/or suggestions.
• Make sure that your review is constructive and not offensive. Please change text that could be considered rude before you submit.

Avoid in reviews

Here are some things that your comments to the authors should not contain:

• A recommendation of whether to accept or reject the paper
• Requests for citations to a bunch of your papers (this will matter more later in your career)
• Requests for experiments/simulations that are unnecessary to justify the main points in the paper
• Insulting criticism or sarcasm
• Typos are not minor issues or major issues. It is not your responsibility to find them.

If there are a huge number of typos then that may be stated as a minor issue. If the paper is completely unreadable then that is a major issue. Completely unreadable means you could not follow the paper even after ignoring all typos.

Re-review process (Post-rebuttal Phase)

Unless the paper was outright rejected or accepted, the authors will have a chance to respond to your review. If you have followed the guidelines above, it should make the re-review process more straightforward: (Conference/Journal)

• If you said minor issues and they addressed your minor issues - Accept.
• If you said major issues and they addressed all your major/minor issues - Accept.
• If you said major issues and they didn’t do what you asked - Reject / major revisions with the outstanding issues
• If you said major revisions and their revision showed their method was incorrect/uninteresting - Reject

Handling Policy Violations

• Contact the program chairs with the paper number and explanation of the suspected problem.
• Review the paper as if there is no problem

Program chairs/ Editors will follow up on the issue, but it may take some time.

Supplemental material

The supplemental material is intended to provide details of derivations and results that does not fit within the paper format or space limit. It is not an extension of the deadline. The paper should indicate which materials are in the supplemental material, and you need consult only if you think it is helpful in understanding the paper and its contribution.

References

These guidelines are directly sourced from the incomplete following list. I was scared that I might give outright horrible reviews when I write them all by myself that I pieced together this guide, and after getting an outstanding reviewer mention decided that it might help people if I put this out.

[1] https://github.com/jtleek/reviews

[2] https://www.danielcolquitt.com/journal/2017/10/peer-reviewing-your-first-paper

[3] https://www.journals.elsevier.com/applied-soft-computing/news/tips-and-advice-when-you-review-a-scientific-paper

[4] http://www.cs.ucsb.edu/~mariya/courses/papers/ReviewingPapers.html

[5] https://authorservices.wiley.com/Reviewers/journal-reviewers/how-to-perform-a-peer-review/step-by-step-guide-to-reviewing-a-manuscript.html

[6] https://www.sciencemag.org/careers/2016/09/how-review-paper

[7] https://www.dropbox.com/s/725p60wcajbb8xh/How%20to%20Review%20for%20CVPR.pptx?dl=0#