Evidence-based medicine refers to the application of the best available evidence in making decisions about the care of patients.1 However, in some instances, the current evidence is insufficient or of poor quality. In these cases, the expert consensus becomes a useful tool to offer unified answers and reduce variability in clinical practice.

A consensus document is a report developed by a group of experts in a rigorous process that involves the application of a formal and standardized consensus method to answer a clinical question.2 Informal consensus exercises may be carried out sometimes, but they are more likely to have drawbacks, such as the potential for particular individuals to dominate the discussion and decision-making, the influence of external pressures or extreme decisions being made by experts with very strong opinions. Table 1 presents scenarios in in which expert consensus may be used in the health care field.

The development of a consensus document involves the selection of a topic for which there is no current evidence or there is controversy requiring clarification (clinical question). An exhaustive review of the scientific literature should be carried out before starting the consensus exercise (there may be consensus based on evidence or not based on it, if there is none, but the literature must be reviewed nonetheless), followed by the definition of the consensus development method to be implemented and the formation of a group of experts. Lastly, the group must write the conclusions/recommendations and draft the document for publication, ensuring transparency and rigor.3–6

There are certain limitations to the expert consensus process. It may just reflect what is known as “collective ignorance”, that is, yield agreements that, while representative of the group’s views, lack a solid scientific basis. In addition, it is not always possible to solve deep disagreements, especially when the experts hold different opinions. In consequence, expert consensus should be considered a complement to the best available evidence, and never a substitute for it.7

The most commonly used methods for social- and health-related topics are the Delphi technique, the nominal group technique, the RAND/UCLA appropriateness method, the consensus development conference, consensus meetings and focus groups (Table 2). The choice of method depends on the type of question, the available time and technical limitations or feasibility.

The Delphi method4,8 is the most structured and most widely used approach. The main drawback is that it requires more time. Questionnaires are sent through mail/email (anonymity), and it is possible to engage a greater number of participants without geographical barriers, but this also increases the probability of losing participants before completion9 (Fig. 1) and precludes the direct interaction of participants during the consensus process. A steering group selects a panel of experts (preserving anonymity), striving to represent the full spectrum of professional, socioeconomic and geographical diversity. There is no agreement regarding the most suitable number of experts, but in most cases between 20–30 participants are involved in this technique. The expert panel does not include the steering group members. The steering group may carry out a systematic review on the topic before the consensus development process, crafting a preliminary list of questions (modified Delphi),8 or simply survey the opinions of experts through one or more open-ended questions on the topic (traditional Delphi). The results of this phase are converted into a list of statements/items in a questionnaire. The questionnaire is submitted to the experts, who have to rate their agreement with the items and provide further feedback through comments or open-ended questions. Agreement is rated on a Likert scale, usually with 3–9 answer choices. The same questionnaire is sent in the second round, this time including feedback on the answers of participants and the most relevant comments from the previous round. Items with the highest agreement (at least 80% of participants) in both rounds are selected. Items with insufficient agreement or for which the threshold was only reached in one round are reviewed and modified as needed. A third round is then carried out. After this vote, the steering group meets and produces the definitive document. Given the characteristics of the method, the agreement or consensus is considered inferior to the consensus achieved through the nominal group technique.

Figure 1.

Consensus methods: Delphi technique, RAND/UCLA appropriateness method, nominal group technique.

The nominal group technique, which, like the Delphi technique, is one of the most widely used structured consensus development methods, allows more interaction between experts on the topic through face-to-face or online meetings, with real-time feedback and a shorter overall duration4,10,11 (Fig. 1). Participation of 5–7 experts is recommended. The process starts with the selection of the professional with the greatest expertise on the topic, who is then responsible for the selection of the rest of the group. The facilitator provides information on the topic to be discussed to participants before they meet. The process comprises four stages: the first one involves individual reflection for 20 min. In the second, participants share their ideas one by one, taking turns, and without any discussion until all participants had provided their feedback (round robin). In the third stage, participants discuss the ideas shared in the previous stage, and the facilitator ensures that all panelists participate in the discussion of all the ideas. The initial ideas are then modified with the consent of the participants that had proposed them originally. In the fourth and last stage, the group votes on the proposed items as modified in the previous stage until a consensus is reached.

The RAND/UCLA appropriateness method4,12 is a hybrid of the two previous methods (Fig. 1). In this approach, the structure of meetings is customized based on the participants and the topic under discussion. The panel typically has 7–15 members. The process starts with a systematic review of the literature by the coordinating committee to establish a rigorous foundation for the discussion. A list of clinical indications is developed based on this evidence, ensuring that the list is comprehensive and the indications are mutually exclusive and applicable to clinical practice. The next step is the selection of an expert panel composed of specialists in different fields related to the topic at hand. The evaluation process is structured in three rounds. First round: Delphi technique questionnaire with individual and anonymous rating of the included items on a Likert scale. Second round: face-to-face meeting of the panel to discuss disagreements and re-rate the appropriateness of items to improve consensus. Third round: updating of items based on the preceding discussion and new round of anonymous voting with rating on a Likert scale. After the voting, each item is categorized as appropriate, inappropriate or uncertain, based on the group median rating. One of the advantages of this method is that it implements a more sophisticated analysis, using interpercentile ranges (raw or adjusted), palliating the drawbacks intrinsic to the variation in the number of experts and the effect of disagreements. There are multiple variants of this method.

The consensus development conference4 approach involves face-to-face meetings of a multidisciplinary group of experts. A steering committee selects a panel of approximately 10 experts. The process is more flexible compared to the previous methods. It starts with the formulation of questions regarding the topic of interest and the performance of a systematic review by a small group of experts that is not involved in the decision-making process. This group of experts present the results to the members of the panel and answer their questions. Subsequently, the panel members meet to deliberate on the issue under the direction of one of them, appointed to act as the chairperson, to reach consensus. This technique has the advantage of fostering dialogue and debate, but it also has significant drawbacks in terms of its cost and time constraints.

Consensus meetings6 and focus groups are less structured options. They consist in a meeting of experts to discuss specific topics in an organized manner in order to reach a consensus. They are based on discussion, negotiation and, sometimes, voting. During these sessions, the agreed-on recommendations and the arguments against certain points get documented. Their structured varies depending on the topic under consideration and the composition of the group.

In 1996, the CONSORT statement was published with the aim of improving quality in reporting randomized controlled trials. Since then, a growing number of reporting guidelines have been published for other methodological designs, many of which are available at the EQUATOR website.13 However, despite the importance of consensus-based guidance in many key health care decisions, a reporting guideline has not been available for consensus development exercises until 2024.

Per its authors, the ACcurate COnsensus Reporting Document (ACCORD) is a guideline developed to help report any consensus methods used in biomedical research, regardless of the health field, techniques used, or application2 (Fig. 2). As is the case of all other reporting guidelines, the purpose of the ACCORD guideline is to help researchers to be transparent in their reports, including detailed information about the materials, resources (both human and financial), and procedures used in their investigations. This way, readers, having access to all the necessary information, can judge the trustworthiness and applicability of their results/recommendations of the consensus document.

Figure 2.

General flow diagram of a consensus exercise.

We proceed to summarize the contents of this guideline. More detailed information, including examples and the original checklist, can be found in the published documentation available at EQUATOR and the ACCORD website.6 There are also free-access versions in Spanish of the checklist and an abridged glossary of terms (Tables 3 and 4).

Once they are published, it is essential to assess the quality of consensus documents through a critical appraisal, analyzing its validity, relevance and applicability to the specific issue or circumstance that prompted the exercise.14 If the report is written in adherence to the recommendations of the appropriate guideline (ACCORD,2 in the case of consensus exercises), it will probably provide sufficient information to answer the key questions in its appraisal.15

Few tools are available to assist the critical appraisal of consensus exercises. In the case of consensus development conferences, a set of quality criteria has been proposed taking into account previously validated critical appraisal approaches.3,16 With a broader scope, the Joanna Briggs Institute has developed a critical appraisal tool for documents reporting expert opinion17,18 named “Textual evidence: Expert opinion” (directions for its use in the Spanish language are provided in the Manual JBI para la Síntesis de la Evidencia).19 It is composed of 6 questions, with four possible answer choices (yes, no, unclear, not applicable). The questions explore the following aspects:

• •

  Clear identification of the source of the opinion

• •

  The knowledge and experience of the authors in the field, as well as their affiliations with any type of organization.

• •

  Whether the interests of the relevant population are the central focus of the opinion

• •

  Logical defense/rationale of the conclusions.

• •

  The reference to the extant literature.

• •

  The congruence/incongruence with the literature and other sources of the opinion.

As is the case of any type of study, consensus exercises are susceptible to potential biases, both during the consensus development process and the writing of the final recommendations. Biases can be implicit or explicit. Implicit bias refers to the unconscious expectations of the participants, whereas explicit bias refers to the conscious and overt bias/expectations of the sponsors, organizers and participants of the consensus process.20 Particular attention should be devoted to:

• •

  The employed consensus method, differentiating formal from informal approaches, as the latter, lacking a systematic methodology, are more prone to bias.

• •

  Whether an exhaustive literature review and synthesis was carried out and its results made available to participants to prevent recommendations that are inappropriate based on the current evidence.

• •

  The funding source, differentiating between public, nonprofit and private sponsorship. Industry-sponsored consensus exercises carry the highest risk of implicit and explicit bias.20

Information on the risk of bias must be provided and discussed in the Discussion section of the document, explaining the potential biases that may have affected the consensus and the measures the authors implemented to minimize them.

In this article, we used the terms response rate and completion rate for the sake of consistency, as they are the terms used in the literature on the subject of consensus exercises. Strictly speaking, however, these terms refer to proportions rather than rates.21

This research project did not receive specific financial support from funding agencies in the public, private or not-for-profit sectors.