Peer review guide

GUIDE FOR PEER REVIEWERS OF SCIENTIFIC ARTICLES IN ST-OPEN

 

This guide for peer reviewers was modified with permission from Marušić M, Sambunjak D, Marušić A. Guide for peer reviewers of scientific articles in the Croatian Medical Journal. Croat Med J. 2005;46:326-32 (http://www.cmj.hr/2005/46/2/15849858.pdf).

 

The peer review process has many imperfections and shortcomings. It is subjective and difficult to control and standardize. Critics claim that the peer review process is slow, expensive, partial, and subject to abuse. However, without peer review it would be almost impossible for editors to publish journals. Peer review is the pillar of scientific publishing, which in turn is a basis of accumulating human knowledge. Still it is not rare to hear that university personnel wonder why they would waste their precious time with doing reviews for journals in which they do not publish.

 

WHY PEER REVIEW?

The first reason to do a review is academic obligation: peer review is a part of scientific publishing; whoever wants to publish must be ready to peer review. The second is the personal benefit – increasing of knowledge and awareness, strengthening professional reputation, and the third is satisfaction – scientific debate, exchange of information, fulfilling the responsibility.

 

Benefit of peer review for authors of the reviewed article

A good review – one that grasps to the essence of a reviewed article, keeping its clarity and simplicity at the same time – can considerably increase the scientific merit of the reviewed article. The reviewer acts as an educator: his or her suggestions and comments enrich authors’ knowledge and ability to perform research and report about it.

 

Benefit of peer review for reviewers

Peer review brings direct benefits to the reviewer. It is a chance for learning, a valuable source of up-to-date scientific information, and actually an exciting job. It increases the reviewer’s knowledge, brings the pleasure and beauty of scientific debate, and creates a feeling of fulfilled responsibility.

Reviewers are privileged to have an insight into the latest research and still unpublished results in their scientific field. Reviewers also build up their ability to critically assess scientific papers, which may be useful in their own professional work and development. Writing high quality reviews strengthens a reviewers’ scientific reputation. Reviewing can also be a significant part of the curriculum vitae.

 

WHAT IS NECESSARY FOR A GOOD PEER REVIEW?

Responsibility. A prerequisite for a good reviewer is a strong sense of responsibility towards research and their colleagues. The reviewers assess the manuscript timely, fairly, and to the best of their abilities.

Conversance with the literature. The reviewers must be thoroughly conversant with the pertinent literature and be able to apply general scientific principles to the given problem. Good reviewers can place the article in the context of relevant previous research, recognize the limitations and weaknesses of the hypothesis, and understand the way in which the conclusions of the article can relate to clinical practice. Reviewers should also be acquainted with the guidelines for authors of the journal for which they are refereeing.

Time. Depending on the complexity of the reviewed article and relevance to the reviewer’s expertise, the time for a fair assessment of an article worth reviewing has been estimated to about three hours. Badly written articles increase the time needed for a review.

Knowing the journal. Different journals have different publishing priorities, review policies, and rejection rates. A good peer reviewer should know these aspects of the journal, so that the review process could identify the best articles for the journal.

Publishing priority of the ST-OPEN is helping students to transform their graduation theses into scientific articles. Our reviewers should keep it in mind while doing the reviews (Box 1).

 

Box 1. Reviewing for ST-OPEN –     Be mild and tolerant to beginner’s mistakes –     Be strict that the articles they receive for the review are true scientific research in given scientific areas –     Write clear and long reviews –     Write notes at the margins of the manuscript (use the “track changes” tool) –     Use every opportunity to send advise to the authors –     Check the list of references and recommend the most recent ones, and ones published in relevant and good scientific journals –     Be sincere and “fatherly”, bur recommend rejection when the article is hopeless (cannot be improved without additional research; still, for educational reasons, recommend the possible and useful additional research)

 

HOW TO REVIEW A MANUSCRIPT

The first principle is to be respectful but resolute. This entails demanding explanations, arguments, and clarity. The seriousness of peer review should not be watered down, inconsistencies should not be concealed, and the editor must be given a clear recommendation.

The process of peer review has a common structure: reading the abstract, reading the text of the article, final appraisal, and writting comments for authors and the editors.

 

Reading the abstract means checking the message of the article, recognition of the type of study, and formulation of broad questions.

In the abstract, authors disclose what they consider most important in their report. Therefore, the reading of the abstract can help the reviewer to look for the crucial elements of the study design, methods, results, and conclusions. At this point, it is good to note general, broad questions that arise from the abstract, such as “Is this really a research study?”, “What is new here?”, “Is the sample big enough?” or “Are the conclusions mixed up with repeating the data?”

 

First reading of the article is like a triage, where the reviewer decides on the importance and relevance of the study. The reviewer should try to understand the article and question all ambiguities, detect shortcomings and limitiations, ask specific questions, check the logic of “the story” and presentation of research data. It is best to write down all the questions in the text of the manuscript, on its margins.

In the first reading, the reviewer has to focus on the science of the article. If there is anything that reviewer does not fully understand, he or she has to think about it, examine the literature or discuss the problem (not the article!) with a more adept colleague. Specific questions can arise from any part of the article (Box 2).

 

Box 2. First reading of the article   Title: does it accurately reflect the content; does it specify the type and the setting of the study? Abstract: is it structured, is it concise, does it specify outcome measures, are numerical data presented, does the conclusion relate directly to the results of the study? Introduction: does it justify performing the study, does it end with the hypothesis, and does the hypothesis arise logically from the theoretical framework? Participants: is the sample and its formation described in detail, are inclusion and exclusion criteria stated, is there a study flowchart? Methods: are they supported by references? Statistical analysis: is the test suitable, presentation appropriate, and interpretation correct? Results: are they clear and convincing? Each table and figure has to be self-sufficient and carry a single message. Discussion: does it begin with the most important finding, does it relate exclusively to the results of the study, are the limitations of the study clearly stated? Conclusions: are they based only on the presented results. References: are they accurate and up-to-date, are they written according to guidelines for authors, are there any obvious mistakes?

 

Second reading of the article is a value assessment: novelty, scientific power and intelligibility.

The second reading should be done after a few hours or days, depending on the time available. It begins with checking the questions and remarks previously written on the manuscript. After that, the reviewer should assess the value of the article, keeping in mind several important points.

The reviewer should freely object to anything that disturbs him or her in reading and comprehending the article. In so doing it is not necessary to judge the general style of the article, because the tastes in that regard can differ. Also, the reviewer is not required to rectify the errors in grammar, spelling, and punctuation, but we appreciate doing it.

An assessment is made about the importance of the science in article. The reviewer’s judgment should not be biased with current popularity of some research areas, but depend upon the strength of the research methods, data, and conclusions. A good article is one that is scientifically sound and brings at least a small new information into the body of human knowledge.

Writing a peer review – comments for the editor up to 200 words), comments for the authors up to 1,000 words. Generally, the appraisal of the article can lead to different types of recommendations accept or minor revision or major revision or reject. Depending on the shortcomings detected during the first and second reading of the article, the reviewer will suggest the editor to accept or reject the article, or to send it back to the authors for revision. If the article presents an interesting idea, but is not sufficiently scientifically sound, the reviewer should suggest the authors how to improve it, and put forward the problem to the editor.

If the article has good science in it, but presents only a minor novelty, the reviewer should ask the authors to explain what they consider new in their work.

If the article is scientifically acceptable, but the text itself is poorly written, the reviewer can be tolerant, but only to a certain point: a carelessly written and messy article should be rejected.

Concrete reasons for recommending a revision of the article can be divied into problems with science (Box 3) and problems with presentation (Box 4). In principle, if the reviewer sees the opportunity for authors to improve the scientific value and data presentation in their article, they can be given a chance to do so.

 

Box 3. Scientific problems with an article –     Contradictions –     Ill-founded conclusions –     Groundless generalizing or attributing causality –     Inappropriate extrapolations –     Circular reasoning –     Studying irrelevant details –     Inconsistencies in classification and measuring

 

 

Box 4. Poor presentation of results   –     Redundancies –     Elaborating unimportant questions –     Imprecise use of words or phrases –     Ill-chosen words in translation –     Use of jargon and nonstandard abbreviations –     Tables and figures not corresponding with text, incorrect sums

 

Reasons for recommending rejection

In spite of being aware that every article submitted for publishing is the result of more or less long and arduous labour of its authors, the reviewer should not hesitate to recommend rejection if the limitations of the article are insurmountable (Box 5).

 

Box 5. Signs of a flawed study –     Does not bring anything new –     Unimportant subject matter –     Fundamentally flawed structure –     Major ethical doubts

The fundamental structure of the study can be flawed, for example when the study does not really test the hypothesis. Unacceptable ethical doubts regarding the study can also be a reason for recommending rejection. The reviewer has to bear in mind that the approval of an institutional ethical committee is not always a guarantee that the study is ethically acceptable. It is the reviewer’s duty to independently assess the ethical integrity of the study. The reviewer should also help in disclosure of plagiarisms and duplicate publications.

 

The article should be rejected if the authors did not use basic scientific principles (Box 6). Improper statistical analysis is not necessarily a reason for recommending rejection, because the authors can correct it. However, appropriate analysis often shows that there are no substantial differences needed to prove the hypothesis, which makes the article unacceptable for publishing.

 

Box 6. Unacceptable shortcomings in an article –     No hypothesis (unless a qualitative study) –     No control –     Weak evidences –     Inappropriate statistical analysis

 

WRITING A PEER REVIEW REPORT

A peer review report consists of two main parts – one for the editor, and the other for the authors. Additionally, the reviewer is usually asked to write comments for the editor and, separately, for the authors.

Comments for the editor

The part intended for the editor should be brief, approximately 200 words. Iit is useful to divide the remarks into general and specific. The reviewer should explain why he or she considers certain objections and questions important, and suggest the way the authors could work them out. At this point one could also express any doubt as to whether authors would be able to satisfactorily resolve the problems. Finally, this is the place for possible praise or recommendation, for example: “This is an original idea, so in spite of the shortcomings of the article, it deserves to be revised instead of rejected”.

Comments for the authors

If the editor decides that the article should be revised before publishing (which is usually the case), he or she will send the reviews to the authors. Although the identity of the reviewer usually remains unknown to the authors, the review should be written as though it would be signed – politely, constructively, and collegially. ST-OPEN leaves an option for the reviewer to sign his or her comments for the author.

The part intended for authors can be as long as 1,000 words or more, but length itself does not always guarantee quality. A few clear, well thought out, and focused questions can be more than enough to help authors to improve the article.

The reviewer should avoid any kind of censure, but also any kind of praise. In the first paragraph the authors might find it useful to see what the reviewer understood as the main message of their article.

If the reviewer could not evaluate certain aspects of the article, he or she should openly admit it.

The comments for authors should be divided and numbered so that the authors can clearly answer each one of them.

Major comments. The reviewer should first state the comments which were described to editor as the most important. Every comment or question should be well-explained and well-founded. Instead of general remarks like “sampling was bad,” it is necessary to clarify why certain aspects of the article are problematic. It is crucial to write precisely and to make clear if the comment is the result of personal reasoning or it is based on available scientific evidences.

Minor comments. The reviewer finally mentions minor faults like unnecessary repetitions, incorrect symbols, or abbreviations. They should be ordered in the same way they appear in the text, and identified by page, paragraph, and line.

A note for reviewers. It is important to finish the review in the time limit set by the editor. If for some reason the reviewer cannot do so, he or she should immediately inform the editor and agree whether the editor will wait longer or send the manuscript to someone else, in which case the reviewer can recommend some less busy colleagues.

It is also important to recognize possible conflicts of interest and, if necessary, decline reviewing the article, with an appropriate explanation to the editor.

Peer review should not be abused as an opportunity for revenge. Any kind of personal remarks are utterly inappropriate and editors usually do not convey them to the authors.