Reviewing a Manuscript
for Publication
Allen S. Lee
Professor, Department of Information Systems
Eminent Scholar, Information Systems Research Institute
School of Business
Virginia Commonwealth University
http://www.people.vcu.edu/~aslee/
Published as an invited note in
Journal of Operations Management
Volume 13, Number 1 (July 1995),
pp.87-92.
If you copy, download, or circulate this
paper, please simply inform the author (at AllenSLee@alum.mit.edu)
that you are doing so.
This paper is based on remarks that the
author prepared for presentation at the New Faculty Workshop held at the
23rd Annual Meeting of the Decision Sciences Institute in Miami
Beach, Florida, November 22, 1991.
Abstract
This paper offers suggestions
about how to review a manuscript submitted for publication in the fields
of management information systems, organizational studies, operations management,
and management in general.
Rationales
for the suggestions and illustrative sample comments are provided.
Contents
- Abstract
-
Action
1:
Start out with your own summary
of the manuscript.
-
Action
2:
Let the editor and author know
what your expertise does, and does not, cover.
-
Action
3:
Give “action-able” advice.
-
Action
4:
Convince the authors by arguing
from their own assumptions and framework.
-
Action
5:
Provide both (1) your general,
overall reaction and (2) a list of specific, numbered point-by-point comments.
-
Action
6:
List the manuscript’s strengths.
-
Action
7:
Quote, give the page number,
or otherwise explicitly locate the parts of the manuscript to which you
are referring.
-
Action
8:
Offer comments on tables, figures,
and diagrams.
-
Action
9:
Be kind.
-
Action
10:
Be frank, in a tactful way,
about your own emotional reaction.
-
Action
11:
Do some of your own library
research.
-
Action
12:
If rejecting the manuscript,
suggest what future research efforts might examine.
-
Action
13:
If recommending a revision,
spell out alternative scenarios for how the revision could be done.
-
Action
14:
Provide citations or a bibliography.
-
Action
15:
Date your review.
-
Why
Review?
-
Conclusion
As management researchers,
we regard the behavior of managers, systems professionals, and other organizational
participants to be a manifestation of the values that they hold as members
of their organization and their
profession.
In
the same way, we may regard our own behaviors, as reviewers of manuscripts
in the “double blind” reviewing process, to be a manifestation of the values
that we hold as members of the community of
scholars.
As
an author and editor, I have seen our community manifest the best and the
worst of human values in the anonymous reviews offered on manuscripts submitted
for
publication.
Some reviewers rise
to the occasion and give extensive help, even though the anonymous reviewing
process promises them nothing in return for their
efforts.
Other
reviewers hide behind the anonymity of the reviewing process, offering
negative remarks that they would not have the courage to voice in
public.
My
immediate purpose is to offer suggestions, based on the reviews I have
seen as an author and editor, about how to provide useful, kind, constructive,
and responsible reviews of manuscripts submitted for publication.
I
offer these suggestions to my colleagues who review manuscripts submitted
for publication in research journals in management information systems,
organizational studies, operations management, and other fields of
management.
1.
Suggestions
for Reviewing a Manuscript
For many of the suggestions
below, I offer sample comments to illustrate my
points.
I
have based these comments on actual
reviews.
1.1
Start
out with Your Own Summary of the Manuscript
As a reviewer for a manuscript, I was
surprised, upon subsequently receiving the associate editor’s own review,
to see that he began with a summary of the
manuscript.
After
all, an author knows what his or her own manuscript is about, so why summarize
it?
Apparently, at least in this case, the
summary was provided for the benefit of the senior editor, not necessarily
the
author.
The associate editor’s
review was, I realized, as much a recommendation to the senior editor as
it was an explanation to the
authors.
Because
a reviewer’s review is, in the same way, a recommendation to an editor,
I have come to believe that a summary of the manuscript being considered
is no less useful in the reviewer’s
review.
I now believe that an opening summary may
also be useful to the manuscript’s author and to the reviewer himself or
herself.
For the author, how effectively
the reviewer’s summary does or does not capture the gist of the manuscript
may serve as one measure of how effectively the manuscript communicates
its
message.
For the reviewer, the
very exercise of composing a summary encourages and virtually assures a
thorough reading of the
manuscript.
Opening summaries are also useful to the
editor when the manuscript is
controversial.
Occasionally,
as an editor, I have wondered if the different reviewers assigned to a
controversial manuscript have actually been sent the same
manuscript.
An
opening summary of the manuscript, presented from the reviewer’s own perspective,
would be a big help to the editor when he or she is trying to reach a decision
on a manuscript that evokes controversial
reactions.
Some illustrative sample comments are:
-
This
paper represents a major effort to test two competing theories about user
satisfaction with electronic mail...
The
methodology of the paper consists of...
The
data were gathered from two field sites...
The
major finding was that...
The contributions
to theory and practice would appear to be...
-
This
manuscript pursues two somewhat conflicting
goals.
It
attempts to…, while it also tries to….
The
authors do a good job of the first one, but their treatment of the second
one raises more questions than it answers.
1. 2
Let
the Editor and Author Know What Your Expertise Does, and Does Not, Cover
By stating where you have expertise and,
no less important, where you lack expertise, you will be helping the editor
and author in their job of interpreting and weighing your
comments.
Reviewers,
in voluntarily identifying where their expertise may be lacking with regard
to the manuscript being reviewed, might even gain additional credibility
for their claims about where they do have
expertise.
-
For
the reader, such as myself, who is unfamiliar with concepts X, Y, and Z,
the authors present no helpful explanation of these concepts or justification
for their inclusion in the study in the first place…
-
Another
problem I had is that, probably like most of the people who read this journal,
I am not deeply read in all three of the research fields that the authors
draw
upon.
I cannot judge how well
this paper builds on past
research.
1. 3 Give “Action-able” Advice
Advice stated in the form of do-able
tasks is mutually advantageous to the author and the reviewer in the event
that the editor asks for a
revision.
For
the author, the advised actions point to a “fixed target” where he or she
may aim the
revision.
For the reviewer,
the advised actions (as further interpreted by the editor) may serve as
the criteria on which to judge the
revision.
In
contrast, a reviewer who offers vague generalities, and no action-able
advice, in his or her first review would have no real “handle” with which
to approve or disapprove the revision; such a reviewer might very well
find a revision returning to “haunt” him or
her.
-
If
my concerns can be addressed successfully in a revision, then I believe
the paper should be
published. I
have four major
concerns. They are…
-
Therefore,
I recommend rejection, but would be willing to review a revised version
if (1) … and (2) …
-
The
following suggestions are provided to improve the weaknesses pointed out
above:
-
Clearly
state the objectives, contributions, and limitations of the
study.
-
Provide
a definition of what you mean by Organizational Support System and use
it consistently throughout the
paper.
-
Using
this definition, narrow down the literature
review.
1. 4
Convince
the Authors by Arguing from Their
Own Assumptions and
Framework
A reviewer can always take issue with
a manuscript’s assumptions and
framework.
However,
disagreeing with the assumptions is not always an effective reviewing strategy
because, strictly speaking, all assumptions are incorrect for what they
assume
away.
An alternative strategy
is to accept the manuscript’s assumptions (if only for the sake of argument)
and then to point out any shortcomings in the manuscript by examining the
consequences that follow from these
assumptions.
(Indeed,
if the assumptions lead to no objectionable consequences, then the assumptions
might not be bad assumptions in the first
place.)
By casting the review
in terms of the authors’ own framework, the reviewer might then be more
likely to convince the authors by courting and affirming the authors, rather
than by disputing the authors.
-
On
the first page, the paper says that it will do the following…
The
rest of the paper, however, does not follow through adequately on what
it promised to
do.
In particular,
according to the standards of the research framework that the authors themselves
have chosen, the following things still need to be done or need to be done
better…
Still, there is much potential
value in what the paper initially proposed and I encourage the authors
to flesh out the paper’s ideas more
thoroughly.
Along
these lines, my suggestions are…
If the reviewer wishes
to suggest a different framework and set of assumptions to the authors,
this suggestion would be more convincing after the reviewer has
demonstrated that he or she has given due consideration to the authors’
original framework, rather than dismissing it
outright.
1.5
Provide
Both (1) Your General, Overall Reaction and (2) a List of Specific, Numbered
Point-by-Point Comments
As an author, I have received some reviews
consisting entirely of numbered, point-by-point comments that give
the impression that the reviewer was simply typing up his or her review
as he or she was reading my manuscript linearly, sentence-by-sentence,
turning it page-by-page.
Whereas
such a review might be detailed and even exhaustive, I have found that
such reviews sometimes negatively criticize me on matters that I actually
address satisfactorily later in the
manuscript.
These
reviewers do a good job of analyzing the words in my manuscript, but they
appear to put no effort into discerning what I meant by these
words.
My
impression has been that these reviewers considered the reviewing job to
be a burden and just wanted to get it
over.
I
have found that if there is no statement of an overall reaction from the
reviewer, I am sometimes left wondering about what the reviewer really
means.
In fact, in this situation,
I sometimes wonder if the reviewer himself knows what he
means.
For
these reasons, I believe that a general, overall reaction or overview from
the reviewer is needed as much as his or her specific, point-by-point
comments.
However, there is at least one occasion
in which a linear, sentence-by-sentence, and page-by-page reading might
be
useful.
When I am a reviewer,
I will occasionally amend my review by paging through the manuscript once
more and enumerating, point-by-point, any comments which I had planned
to make when I first read the manuscript, but which somehow did not make
their way into the main body of my
review.
Numbering the major points in a review
is helpful to the editor and
author.
For
instance, an editor could then conveniently say to the author, “Pay particular
attention to points 2, 3, and 5 by Reviewer
1.
”
1. 6
List
the Manuscript’s Strengths
Perhaps the most disheartening review
I have ever seen is one that began with the single-sentence paragraph,
“There are several problems with this paper,” and followed with a numbered,
blow-by-blow listing of all the alleged problems in the
manuscript.
An
accompanying listing of the manuscript’s strengths would have made the
review more palatable (and hence convincing) to the
author.
A listing of the manuscript’s strengths
takes on added importance when the reviewer’s recommendation is that the
manuscript should be
rejected.
Is
there really nothing in the manuscript that would make it worthy of a revision?
Making up a list of the manuscript’s strengths would help make sure that
no stone is left
unturned.
1.
7
Quote,
Give the Page Number, or Otherwise Explicitly Locate the Parts of the Manuscript
to Which You Are Referring
This will pinpoint what you find difficult
to understand, what you disagree with, or exactly what you believe needs
to be
changed.
Moreover, if the author
should disagree with your assessment, then the author may respond precisely
to your
objection.
-
In
the third paragraph on page 9, it is not clear to me that the authors understand
the concept of construct
validity.
-
On
page 3, in the literature review section, the paper says, “…only 12 percent
of the past studies examined the same factors we will be examining in this
study….” Exactly which studies were these?I
do not doubt your statement, but I would like to be able to evaluate it
for myself.
-
On
page 2, why does the prior research necessarily suggest that we need to
study this topic, as you claim?
1. 8
Offer
Comments on Tables, Figures, and Diagrams
Because tables, figures, and diagrams
often appear at the end of the manuscript, they often do not receive the
attention they
deserve.
However,
I believe that reviewing an illustration can be equivalent to reviewing
a thousand
words.
Because illustrations
are often overlooked in reviews, a detailed comment about an illustration
might favorably impress the author and editor, suggesting to them that
the reviewer is especially
conscientious.
Also,
suggesting a new table, figure, or diagram may encourage the author to
sharpen his or her argument.
1. 9
Be
Kind
There are tactful ways to express negative
criticisms.
For example, if you are
unsure what the contribution of the manuscript is, say “What’s new?”
instead of “So what?”I believe
that if the criticism cannot be stated in a kind and constructive way,
then the criticism might not be worth stating at
all.
Also,
unkind remarks in a review that is otherwise valid may create difficulties
for the editor who would like to persuade the author that the review does
have
merit.
1. 10
Be
Frank, in a Tactful Way, about Your Own Emotional Reaction
Some reviews tend to be
dry.
As
an author and editor, I find that any hint or explicit statement about
the reviewer’s feelings will help me to interpret what he or she
means.
1.
11
Do
Some of Your Own Library Research
In my experience as an author and editor,
I tend to give an extra measure of credibility to reviewers who have done
some library or other research for their
review.
This
effort makes the review appear sincere and
convincing.
A
quotation from a book or article that the reviewer has looked up can be
impressive.
1. 12
If
Rejecting the Manuscript, Suggest What Future Research Efforts Might Examine
Our own behavior as reviewers in the
“double blind” review process reveals our individual values, which may
include adversarial values and collegial
values.
Rejecting
a manuscript and offering only the reasons for rejection reveals a person
who has no contribution to make to the overall community of
scholars.
Rejecting
a manuscript, but also offering suggestions about what the author could
pursue instead or pursue differently in future research, reveals a person
who is integrated into the community of scholars and seeks to foster its
growth.
1. 13
If
Recommending a Revision, Spell Out Alternative Scenarios for How the Revision
Could be Done
Simply saying “this paper needs a good
re-write” is not, by itself, helpful, especially if it is
true.
Often,
there is more than one way to revise a
manuscript.
Suggest
two or more scenarios, mention what you believe to be the advantages or
disadvantages of each one, and leave the choice up to the author.
1. 14
Provide
Citations or a Bibliography
A citation that the author finds helpful
can be as valuable as a thousand or more words in the rest of the review.
1. 15
Date
Your Review
As an author and editor, I do not appreciate
late
reviews.
Once, I noticed that
a colleague of
min
e prominently displayed the current date at the top of
a review that he was about to send
in.
He
said that the date would let the authors of the manuscript know that, if
the overall cycle time on their manuscript was excessive, he was not the
cause.
I also suspect that a date
on a review can function as an incentive for subsequent participants in
the review process to act on the manuscript promptly.
2.
Why
Review?
I see four benefits
to engaging in the effort of reviewing a manuscript submitted for publication.
Benefits to the Reviewer in the Short
Run Typically, a reviewer will
receive the reviews by the other reviewers and the
editor.
Doing
a review therefore confers an insider’s view of the reviewing
process.
The
reactions of the other reviewers and the editor all contain potential lessons
for one’s own manuscripts to be submitted for
publication.
In
reviewing manuscripts, one also gains access to invaluable
bibliographies.
Access
to these bibliographies is sufficient justification, in itself, to find
the time to participate in the reviewing process.
Benefits to the Reviewer in the Long
Run Good reviewers are hard to
find.
A track record of good reviews
will enhance one’s reputation with editors, who may then serve (if need
be) as job contacts or outside reviewers in one’s tenure, promotion, and
re-appointment
process.
In this regard,
one’s performance in his or her review of a manuscript can be compared
to one’s performance in a job
interview.
Good
reviews can benefit one’s career.
Benefits to Others
Numerous
people have helped me launch my career as an university teacher and
researcher.
When
they ask me to review a manuscript for which they are the editor or track
chair, I regard their request as an opportunity for me to return some of
the help they have given
me.
In our
research culture, doing a review of a manuscript is a socially significant
gesture.
Benefits to One’s Own School of Thought
As
an author, I often have the experience in which reviewers, hostile to and
ignorant of the research traditions that I embrace, misreview my
submission.
Therefore,
whenever I find that I am a reviewer for a submission that falls in my
own school of thought, I expend extra efforts to give it a careful, constructive
review.
Realizing that the refereeing
process is political, I will do my best to be supportive and affirmingly
critical, drawing attention to any major significant points in the submission
and delineating in explicit, constructive, and “action-able” ways how the
author’s research can be
improved.
As
a result, the editor would, if necessary, have some “ammunition” with which
to neutralize any hostile and ignorant reviews and thereby to justify a
positive editorial decision on this submission.
3.
Conclusion
No review of a manuscript
must incorporate all the features I have described above.
I
am also confident that there are additional useful features I have not
yet
encountered.
I have identified
these features based on my own experience as a member of the management
research
community.
I encourage my
colleagues to do the same.
Do actual instances of good reviews follow
from rules for how to review a manuscript for publication, or do rules
for how to review a manuscript for publication follow from actual instances
of good reviews? I believe that there is some truth to
both.
Following
any set of guidelines for how to do a review may be helpful, but should
not dissuade the creative and caring reviewer from innovating additional
reviewing methods.
[
Back
to Allen Lee’s home page
]
|