KEY POINTS:

• The rational model of health policy analysis underscores the critical role of infor­mation in health policymaking.

• In the synoptic model, rational policymaking consists of a number of consecutive steps . Policymakers choose the policy alternative that yields the optimal result, given the best information available. 

• The deliberative model underscores the importance of argumentation, interpretation, multiple advocacy, and justification in making rational policy decisions.

• Policymaking involves sense-making, which can be described as inferring infor­mation from observations and imbuing information with meaning (interpret­ation).

• The synoptic version of rational policymaking puts analytical information based upon ‘objective’ analysis central. In the deliberative version, policymakers tap from multiple sources of information. 

• The call for evidence-based policymaking resonates with an optimistic belief in the power of science for resolving policy problems. 

• Science-based information has three main functions in health policymaking: an instrumental function, an enlightenment function, and a political function. 

• There are several limits to the scientification of health policymaking. Science cannot bridge the gap between ‘is’ and ‘ought’ and cannot cope well with the complex structure of many health problems. Another problem is lack of information. Policy­makers may also purposively pass over information.

• A logical gap exists between the ‘logic’ of science and the ‘logic’ of policymaking. 

• Uncertainty and risk are inherent to all health policymaking. Uncertain risks may involve substantial threats to public health.

• Strategies to deal with uncertain risks are: doing policy research, consultation, reducing complexity, doing by learning, applying the precautionary principle, building a resilient health system, covering up, and risk denial. 

8.1 Introduction

In Chapter 3, we have seen that Colebatch (2009) associates policy with order and consistency, expertise, and authority. The strategy of the Dutch government and governments in numerous other countries to manage the pandemic illustrates the reliance on expert knowledge to justify unprecedented decisions. State interventions should not be the outcome of political struggle, ideological convictions, or power conflicts but rest upon the best information available. Rational policymaking requires insight into the effectiveness and efficiency of the policy instruments, potential side effects, financial consequences, practical feasibility, and lawfulness. Health technology assessment must precede decision-making on the benefits catalog of public financing schemes (‘package decisions’). The effectiveness and safety of vaccines must be undisputed. Nothing would be more detrimental to public confidence in mass vaccination than fiddling around the effectiveness and safety of vaccines. Policy measures to discourage smoking, ensure food safety, or control healthcare expenditures should have a firm scientific basis, and so on.

The central claim of the rational approach is that health policymaking based on a systematic collection and well-crafted analysis and appraisal of information will yield the best policy results. Information-based health policy is superior to policymaking based on private interests, ideological contests, political games, and power.

This chapter discusses the rational model of health policymaking. The focus is on the role of information and analysis in health policymaking. The chapter starts with an overview and discussion of two alternative models of rational policymaking: the synoptic model and the deliberative model. Though noticeable differences, both models underscore the need for information. Next follows a discussion about the critical role of sense-making in policymaking. Sense-making is defined as the collective process of inferring information from data and imbuing information with meaning. The third theme concerns the fact that policymakers use multiple sources of information. Scientific (research-based) information is only one source of information and in many situations not the most important source. Hereafter follow three sections on the ‘scientification’ of health policy. What does the scientification of policymaking mean? How can science contribute to policymaking, and what are its limits in policymaking? The final part of the chapter discusses the problem of uncertainty and risk in health policymaking and describes several strategies policymakers practice to cope with uncertain risks.

8.2 Synoptic model 

The synoptic model (Braybrooke & Lindblom, 1963) is a good starting point in discus­sing rational policymaking. The model focuses on decision-making. Rational decision-making in the synoptic model consists of a number of steps logically following each other. The model assumes a problem that is not further problematized. The first step includes the formulation and ordering of policy goals and the second step an inventory of alternative policy instruments to attain the stated policy goals. Next follows an assessment of the expected effects of these instruments to find out which instruments or combination of instruments will best contribute to the attainment of the stated policy goals. The final step is to choose the instrument or combination of instruments that promises the best result, which is defined as the maximum difference between input (resources) and output (effects). Policy in the synoptic model is the outcome of rational choice.

The synoptic model does not describe how decision-making proceeds in practice but how it should be organized to achieve the best result. It is a prescriptive model for decision-making and assumes a simple relationship between policymaker and policy analyst. While policymakers carry responsibility for ultimate decision-making, the task of policy analysts is to feed them with the best information available.

Elements of the synoptic model are recognizable in how the Dutch government informed the nation about its policy measures to fight the pandemic. The government said to base its interventions upon the expert knowledge of the OMT. In turn, the OMT declared to base its policy recommendations upon the latest scientific insights and a complex quantitative disease model fed with the most recent updates on the spread and infection rate of the coronavirus. However, the OMT always asked explicit attention to uncertainty. The course of the pandemic, its impact on the healthcare system, the effects of policy measures, and particularly the effects of distinct policy measures (e.g. the extra effect of the curfew) could not be precisely forecasted. Estimates were surrounded by confidence intervals. Furthermore, the OMT concentrated on the epidemiological dimensions of the pandemic and its consequences for population health and the health­care system. The social-economic consequences, the consequences for mental health and patients on the waiting list due to lack of capacity, to mention only a few examples, were largely left out of consideration. Thus, the scope of expert information the government said to rely upon was limited.

In their seminal study ‘A Strategy of Decision’, Braybrooke and Lindblom discuss several reasons why the synoptic model has little prescriptive value for decision-making. First, the model assumes a given problem. This assumption ignores the multidimensional and unstructured nature of public policy problems. What is called the problem often consists of a cluster of interlocked problems with interdependent solutions and multiple dimensions. In other words: ‘the formulation of a wicked problem is the problem’ (Rittel & Webber 1973: p. 161).

Second, Bray­brooke & Lindblom refute the assumption of consensus on clearly defined and well-ordered policy goals. This assumption obscures the role of value pluralism and judgment pluralism in policymaking (chapter 9) and repudiates the multiplicity and ambiguity of policy goals. Even if policymakers say to agree on policy goals, they may nevertheless interpret these goals differently or set different priorities. The operationalization of abstract goals into concrete goals and activities is also frequently controversial. Besides, what is important today may be less important tomorrow. Weighing the costs and benefits using a well-defined and commonly accepted evaluative method is illusionary in public policymaking, despite the optimism on the merits of cost-benefit analysis.

Third, the synoptic model assumes a clear dividing line between facts and values. Facts belong to the sphere of activity of policy analysts, while policymakers are responsible for value judgments. However, a clear-cut dividing line between facts and values does not exist. Values and ‘facts’ may intersect each other in each stage of the policymaking process (chapter 9). The more policymakers lean on the input of policy experts, the greater the risk of a technocratic style of policymaking.

Fourth, the model reduces policymaking to a purely analytical and information-based activity to find the best or ‘optimal’ solution for a given policy problem. It assumes (near) complete information on policy instruments and their effects. However, even near complete information does not exist. Uncertainty is inherent to all policymaking and always confronts policymakers with the problem of how to cope with it. According to Nobel Prize winner Simon (1997), the synoptic model disregards the ‘boun­ded rationality’ of man. Policy­makers are unable to collect complete information. Neither can they deal with complete information because of cognitive limitations. Moreover, the collection of information is costly. In urgent situations, policymakers also miss the time to figure out which policy alternative will work best. They are expected to act immediately. For the most part, policymaking evolves as a process of trial and error or, in the terminology of Braybrooke and Lindblom, as a process of serial and remedial action.

Fifth, information is a potential source of confusion because of inconclusiveness and contradictions. Informational abundance has a similar effect. In practice, much of the struggle in policymaking concentrates on the validity and reliability of information and how information should be given meaning (section 8.3).

Sixth, the synoptic model of rational decision-making ignores the impact of interest conflicts, power relations, and the disjointed governance structure of public policymaking. Actually, the model assumes a neatly structured hierarchy for policy­making and features ‘a deep-seated suspicion of ‘politics’’ (Hajer & Wagenaar, 2003: p. 18).

Finally, the model assumes broad public trust in policymaking based on the best information possible. This assumption is problematic in the context of the declining level of public trust in public authorities and science-based policymaking. Rational policymaking does not guarantee public trust and legitimacy.

Despite these critical observations, many textbooks on policy analysis take the synoptic model as an analytical point of departure. For instance, policymaking should begin with investigating the scope and structure of policy problems to arrive at a common formulation of the problem, the policy goals, and the priority order. Policymaking also requires a systematic investigation of policy alternatives and their potential effects. Policy analysts have an extensive toolbox of methods and instruments for this task at their disposal. Examples are operations research, cost-benefit analysis, cost-effectiveness analysis, risk analysis, policy impact analysis, budget impact analysis, forecasting, disease modeling, simulations, strengths weaknesses estimates, opportunities/threats esti­mates, etc. The ideal of the synoptic model also resonates with the call for evidence-based health policymaking.

Braybrooke and Lindblom not only refute the prescriptive value of the synoptic model. They also observe a considerable gap between the synoptic ideal and the daily practice of decision-making. They describe policymaking as a process of ‘muddling through’ in which the challenge for policymakers is more on reaching an agreement through a process of mutual adjustment than on making rational means-ends choices. In practice, a great deal of policymaking consists of reacting to the moves of other actors (Lindblom, 1959).

Majone (who prefers the term decisionist model) criticizes the synoptic model for its exclusive focus on decision-making. Sometimes, prudent policymaking requires the postponement of decision-making, because the time is not yet ready for decision-making and the consequences of premature decisions can do more bad than good. A wait-and-see strategy can be preferable to respond adequately to unexpected developments. Furthermore, he criticizes the exclusive preoccupation of rational decision-making with outcomes and the neglect of the structure of the decision-making process. The acceptance of a policy not only depends on its outcomes but also on the organization of decision-making. Rational decision-making requires both output legitimacy (does it work?) and procedural legitimacy (is the organization of decision-making accepted as legitimate?). In fact, the decisionist model assumes a hierarchy-like organization of the decision-making process the outcomes of which are not questioned because they are assumed to be ‘optimal’. It is a top-down model of decision-making that leaves little or no room for bottom-up contributions (Majone, 1989).

8.3 Deliberative model

The deliberative model draws upon the insight that ‘policy analysis is more than data analysis or a modeling exercise: it also provides standards of argument and an intellectual structure for public discourse’ (Majone 1989: p. 7). The model underscores the crucial role of argumentation, interpretation, multiple advocacy, and justification in policymaking. Policy analysts play a supportive role in this process, but their role is not confined to feeding policymakers with information based upon abstract models and, preferably, quantitative analysis. Instead, their task is to support policymakers as ‘producer of argu­ments’ (p.23). Argumentation differs from formal demonstration. The formal demonstration that instrument X will produce effect Y or that Y will happen if no action is undertaken is insufficient to persuade. Policymakers need arguments to convince others in the health policy arena. The challenge of policy analysts is to provide policymakers with good arguments based on a critical analysis of policy assumptions, dilemmas, uncertainties, risks, longer-term consequences, and contextual factors. 

The critical role of argumentation in the deliberative model follows from the insight that policymaking takes place in an arena with multiple values, multiple views, multiple interests, multiple dilemmas, and multiple uncertainties (Hajer & Wagenaar 2003). There is no such thing as a ‘single truth’. Understanding the multi-faceted and interlocked structure and dynamics of public problems requires input from multiple sources. Deliberation requires an open debate on problems and solutions with room for alternative voices. Articulation and exchange of arguments are critical for arriving at reasonable decisions and an effective antidote to ‘policy myopia’. Arguments instead of power and vested interests should ultimately be decisive. Besides, argumentation is an effective strategy to question institutionalized belief systems and develop new ideas for policymaking.

The deliberative model underscores the critical role of information in policy­making. Policymaking without information or ignoring relevant information is a ticket to misery. However, the model postulates that information is not discovered but manufactured and that information must be interpreted to be meaningful for policy­making. Deliberation requires a critical inspection of information and how it is given meaning (see next section).

Furthermore, the model assumes a pluralist or democratic organization of the policymaking process. Nothing is more useful in policymaking than an exchange of information and viewpoints from different perspect­ives. According to Majone, multiple advocacy contributes to the legitimacy of policy decisions.

The deliberative model stresses the normative dimension of policymaking. Majone speaks in this respect about the critical role of norm-setting and norm-using in policymaking. Policymaking cannot be reduced to a mere ‘information process’. Rationali­ty should ‘not be defined in instrumental terms, but as the ability to provide acceptable reasons for one’s choices and actions’ (p. 23). Policy analysts and policy­makers must explain which moral judgments have directed their problem formulation and policy choices. Policymaking involves a complex balancing act between alternative normative viewpoints and criteria.

Finally, Majone distinguishes between the processes of discovery and justification. Discovery is concerned with how policy decisions have been reached, while justification includes persuading people of the necessity and reasonability of these decisions. Policymaking is not only a matter of well-reasoned decisions but also a matter of building public trust and using appealing symbols. Policies must be legitimized to be accepted. They require a convincing narrative.

Majone presents his deliberative model of policymaking as an alternative to the information-driven and ‘technocratic’ synoptic model. A critical aspect of the deliberative model is the assumption of an open mind. Policymakers and the wider public must be willing to listen to each other and receptive to alternative views. How­ever, the open-mindedness may not exist in the daily practice of health policymaking. The model does not work in a polarized atmosphere. The deliberative model also assumes enough time for decision-making which is not available in times of an acute crisis.

The pragmatist turn in policymaking

In their plea for a revision of the science-policy relationship in times of crisis and the need for a pragmatist turn in policymaking, Greenhalgh and Engebretsen (2022) also reason from the premises of the deliberative model. They argue that the following tendencies characterized the management of COVID-19 by the UK government at several occasions:

• Scientism: excessive reliance on science to produce solutions.

• Reductionism: Conversion of complex problems into simple ones.

• Abstraction: neglect of context and a strong focus on generalizability.

• Linearity: knowledge should precede action.

• Scientific elitism: policymakers rely on an ‘inside track’ of trusted advisers.

• Exclusionary epistemology: only a limited range of scientific methods and moral views are acceptable for policymaking.

• Polarization: the tendency for scientists to separate in ‘camps’ rather than engage in dialogue.

The pragmatist turn they argue for rejects each of these tendencies and calls for less exclusive reliance on science, for embracing complexity (anti-reductionism), for attent­ion to the contextual factors (anti-abstraction), for acting judiciously under uncertainty instead of waiting for hard evidence (anti-linearity), for multiple advocacy (anti-scientific elitism), and for epistemological pluralism and dialogue instead of polarization. Further­more, the pragmatist turn emphasizes the need for social interactionism in policymaking. Policymakers must understand what facts and interventions mean for people and factor these meanings into their policy decisions and communication on these decisions. 

Role of citizen forums in deliberative decision-making

The citizen forum is a relatively new instrument for organizing deliberative policy­making. A forum (alternative terms are council, assembly, and panel) consists of a limited number of individuals forming together a cross-section of the population. Selection of the forum members takes place through a (stratified) lottery. The forum discusses complex problems in a limited number of sessions and formulates policy recommend­ations to the policymakers in charge. Members are fed with all information they need. A precondition is that each member has an open mind for information, is prepared to listen and change opinion based on good arguments. Thus, deliberation of argu­ments instead of the mere exchange of arguments.

Citizen forums are complementary to representative democracy. They are intended as an instrument to resolve some structural deficiencies in the representative democracy model, such as an overrepresentation of persons with high education in representative bodies, the impact of lobbyists on public policymaking, power-driven party politics, short-term horizon decision-making, and ‘phantom’ citizen participation. Citizen forums give ordinary citizens a role in public policy­making which should help to restore public confidence in public policymaking. A critical aspect of forums is how policymakers deal with their recommendations. They have no feature if policymakers put their recommendations aside.

There are several examples of citizen forums in health policymaking (Box 8.2). The United Kingdom Citizens Council consisting of a representative group of 30 people regularly provides the National Institute for Health and Care Excellence with a public perspective on overarching moral and ethical issues that the Institute needs to consider. A citizen forum in the Netherlands discussed acceptable criteria for decision-making on the composition of the benefits catalog of statutory health insurance. The forum identified sixteen acceptable criteria for making ‘package decisions’ two of which related to the disease (e.g. medical neces­sity), eleven to the characteristics of the treatment (e.g. effectiveness, availability of alternative treatments, and costs), and three to the person (e.g. age and lifestyle). It did not reach a consensus on the operationalization and the relative weight of these criteria in concrete situations (Bijlmakers et al., 2020). 

8.4 The observation-information-interpretation relationship

Central in the rational model is the emphasis on the role of information in policymaking. Policy decisions should rest on the best information available to disentangle the complex structure of public policy problems and investigate the effects of alternative policy interventions, including the (administrative) costs and the feasibility of these inter­ventions. Information is a precondition for rational decision-making and avoiding mistakes. But what is information, and how is information made meaningful for policymaking? To answer these questions, an analytic distinction must be made between observations, informa­tion, and inter­pretation (Figure 8.1).

The first step in the model is the conversion of observations (data) into information. The second step involves the conversion of information into policy-relevant information. The interpretation of information is the third step. Each step assumes a conceptual model that directs the collection of the observations, the inference of information from observations, and the interpretation of information. Interpretation requires a normative framework to judge information. The relationship between observations, information, and inter­pretation is reciprocal: observations are the raw material for information and information asks for an interpretation. At the same time, however, the need for information directs the collection of observations. Likewise, the interpretative framework directs the collection of observations and the conversion of observations into information.

The observation-information relationship

The inference of information from observations assumes a conceptual model or concept­ual filter to ‘steer’ the collection of observations. An illustration is the measurement of a nation’s level of healthcare expenditures. How much a country spends on health care is contingent on the definition of healthcare expenditures as well as the reliability and completeness of the observations. Which expenditures are counted as healthcare expenditures, and which are not taken into account? A single answer to this question does not exist. There is much variation in how countries calculate their healthcare expenditures. The definition of health­care expenditures influences information on healthcare expenditures. To make reliable international comparisons of healthcare expenditures possible, the Organization of Economic Coordination and Development (OECD) has developed the System of Health Accounts to determine which expenditures must be recorded as healthcare expenditures and which expenditures must be left out of consideration. The OECD definition produces a different picture of healthcare expenditures than national accounts (Box 8.3).

There are countless examples of how the underlying conceptual model influences information. A simple answer to at first sight simple questions such as how many hospital beds or IC units a country has or how long patients must wait for medical treatment does not exist. The path from observations to information is paved with methodological obstacles, even more so if the information is gathered on abstract concepts such as quality of care, primary care, long-term care, accessibility of health care, quality of life, or health and sickness. Information is critically contingent on the operationalization of these concepts, the completeness and reliability of the observations, the validity of the underlying causal model to estimate future trends or policy effects, the selected time span, the research methods, and the baseline period. The important lesson is that information or ‘facts’ is actually manufactured or con­structed know­led­ge: information is not discovered but inferred from observations based on an explicit or implicit conceptual model. An alternative model may produce other information. A great of policy discussions and political contests concentrates on the validity of the conceptual model. 

The information-interpretation relationship

Suppose a country spends 10 percent of its Gross Domestic Product (GDP) on health care. What does this percentage mean? Is 10 percent a problem? The answer to this question depends on the interpretative framework for giving meaning to this percentage. Whether 10 percent is considered a problem depends upon the normative framework used. The country’s level of healthcare expenditure is only meaningful information for policymaking if it is considered problematic. Policy problems are social or political constructs (chapter 3). Information does not derive its meaning from its intrinsic qualities but from the meaning given to it. Where optimists speak about a glass half-full, pessimists talk about a glass half-empty. Interpretation also involves the contextualization of information. Finally, interpretations are not cast in concrete; they can be revised later. 

Interpretation is also indispensable with regard to uncertainty. Uncertainty is inherent to all policymaking and policymakers must somehow deal with it. They can follow various strategies to reduce uncertainty but the problem cannot be completely resolved by collecting extra information. As a consequence, policymakers must fall back on interpretation to fill ‘information holes’. Finally, interpretation is critical in filtering information. Which information is considered relevant for policymaking? Who is believed and what is taken as true and relevant?

The process of inferring information from data and imbuing this information with meaning is called sense-making. Cognitively, sense-making takes place in the brains of the individual but it is foremost a collective process influenced by numerous factors, including institutionalized belief systems, political considerations, power relations, the group individuals belong to, standard operating procedures, domain conflicts, and the structure of the governance system (Douglas, 1986; Boin et al., 2021). Sense-making is closely related to political communication and public imaging (McNair, 2003). What sense does the population make of policy decisions? Facts only are not enough. Policy­makers must convince the people by a credible interpretation of what is going on and what they want to achieve. The critical role of public imaging cannot be underestimated.

The concept of sense-making highlights an important difference between the synoptic and deliberative model of health policymaking. The synoptic model ignores the critical role of sense-making in policymaking. Information has an instrumental function in policymaking that is used to select the optimal mix of policy instruments to achieve the stated policy goals. For its part, the deliberative model of policymaking makes sense-making a central part of rational policymaking. Rational policymaking requires a critical stance on information and interpretation.

The distinction between observations, information, and interpretation has implications for health policy analysts. A crucial aspect of their task is critically investi­gating the inference of information from observations and the conversion of infor­mation into policy problems (interpretation). Such an investigation requires detailed policy-issue knowledge. 

8.5 Sources and utilization of information

Policymakers can tap into multiple sources of information for policymakers. Which information resources are available to them and how they use it? 

Sources of information 

A distinction can be made between the following sources of information:

• Policy-oriented research 

Policy-oriented research aims to collect information about policy problems, future developments, the effectiveness of policy instruments, potential policy risks, public opinion and public confidence, and so forth. Sector policy analysts have an extensive toolbox of instruments at their disposal for quantitative and qualitative policy-oriented research.

• Evidence-based information (science-based information)

There exists no sharp dividing line between evidence-based information and infor­mation collected by policy-oriented research. Evidence-based information has to meet stricter methodological standards than policy-oriented research and is based upon theoretical hypotheses subjected to (rigorous) empirical testing. Most policy-oriented research is descriptive and case-oriented.

• Expert information

Sector-bound specialists and advisory bodies are an important source of information for policymakers to gather expert information on judicial, economic, organizational, international, social, technical, and other relevant aspects of alternative policies. Nowadays, health policymaking is unthinkable without a well-developed intelligence system or knowledge infrastructure to inform policymakers (box 4.3).

• Statistical information 

Statistical information has become an indispensable instrument for policymaking. Poli­cy­makers need statistical data to substantiate their plans. Plans based on quantitative data are considered superior to plans based on qualitative information only.

• Experience-based information

Experience is another important source of information for policymakers. Past expe­rience contains valuable lessons for what works or will fail.

• Colleague information

Contacts with domestic or foreign colleagues about their experiences in health policy­making may open information that otherwise may not be accessible. Particularly, infor­mation about crucial details easily is of great value in this respect. 

• Political information

Political information concerns the political context of policymaking. Important issues are the level of political support and public confidence, the identification of potential partners and adversaries, an estimation of their strategies, and information about how partners and adversaries could be involved in the policymaking process.

• Information provided by advocacy organizations

Advocacy organizations can provide policymakers with valuable information about their policy preferences, policy alternatives, policy effects, and policy risks and give insight into the level of support for policy plans.

• Media information

Policymakers read newspapers, watch TV and strip social media to find out what is happening in society and learn about public opinion and emotions. Occasionally, media information immediately influences the political agenda. 

• Information based upon trial and error (policy learning)

Finally, policymakers learn by trial and error. For this reason, policymaking should not be designed as a ‘one-shot’ operation but rather as a process of adaptation to changing conditions and new information.

Utilization of information

If policymakers can tap into multiple sources of information, the question arises which sources of information they use in practice. The synoptic version of the rational model of policymaking puts the utilization of analytical information derived from policy-oriented research, scientific insights, expert information, and statistical sources central. Information from these sources is assumed to be superior to ‘subjective’ information from other sources. Rational policymaking rests on ‘objective’ information. The task of policy analysts is to feed policymakers with this kind of information. The rational approach radiates great confidence in the problem-solving power of what is assumed to be objective information.

The deliberative version of rational policymaking follows a different approach. All information considered relevant for policymaking should be given attention, irrespective of the source where it comes from. Creating room for counter-argu­mentation can protect policymakers from making errors. Detailed information of concerned citizens on how policy measures will play out in practice is as important as information derived from modeling. Ignoring political information is asking for difficulties.

What about the policymakers’ use of information? Which information are they most interested in? We confine ourselves to a few general observations. The first observation is self-evident: the use of information is contingent upon the type of information needed. If policymakers need legal advice, they will consult legal experts; if they need epidemiological advice, they will consult epidemiological experts, and so on. The pro­blem formulation plays a directive role in this respect. It is no coincidence that policymakers based their policy decisions on COVID-19 almost exclusively on epidemiological and biomedical information provided by a select group of experts (Lohse & Canali, 2021). Second, policymakers do, in most situations, not rely upon a single source of information. Instead, they tap information from multiple sources. In this respect, it is noteworthy that they use a broad definition of evidence. Evidence is for policymakers every piece of information they hold for true and relevant. Science-based evidence competes with other kinds of information and is, in many situations, not the most important source of information to them (Lomas & Brown, 2009). Third, it should be noted that the use of information is always selective. Institu­tionalized beliefs, political considerations, obstructed communication channels, time pressure, power relations and experience, personal preferences, professional back­ground, and lobbying influence information filtering. Sometimes, policymakers even seclude them­selves from information to preserve internal unity. This social-psychological process is known as group­think (Box 8.4). The fourth observation concerns the prominent role of quantitative information: unless they need specific qualitative information, policymakers tend to prefer quantitative rather to qualitative information. A few insightful statistics often count more than qualitative analyses. The fifth critical factor is the credibility and source of information. Influence requires that information-givers (experts) have acquired a trust position and developed good contacts with relevant policy­makers. 

The implicit assumption is that policymakers are interested in information. Understanding the importance of being well-informed, they will do their best to acquire information. It is an open question whether this is always the case. An abundance of information can be confusing. Collection of information may take much time, cost a lot of money, and not lead to better insights. Policymakers may also feel it necessary to act at short notice. Writing about public policymaking, Keynes once observed in a frank mood that ‘there is nothing a government hates more than to be well-informed; for it makes the process of arriving at decisions much more complicated and difficult’ (Skidelsky, 1992).

Sometimes, policymakers even show disrespect for information. An example is President Trump’s way of acting during the COVID-19 pandemic. On several occasions, he blatantly disregarded the information of respected American public health experts on the magnitude and risks of the coronavirus for public health. He even deliberately misled the population by telling his audience that hydroxycholoquine was a simple medicine to cure or prevent an infection of the coronavirus without any conclusive evidence for its effectiveness. Even worse, scientists warned of cardiac toxicity and other harmful effects (Christakis, 2021).

8.6 Evidence-based health policymaking

Nowadays, there is a call for evidence-based policymaking in public policymaking. Eviden­ce-based policymaking ‘helps people make well-informed decisions about policies, programs and projects by putting the best available evidence at the heart of policy development and implementation’ (Davies et al 2000; p. 3). Its advocates claim that policymaking based upon validated evidence will yield better policy results than policy­making without science input. The ‘scientification’ of policy­making is viewed as a precondition for rational policymaking and an effective antidote to policy pitfalls.

Confusion exists on what the term evidence-based means. Most advocates of evidence-based policymaking hold the opinion that the term evidence should not be restricted to science-based or research-based information. They choose a broader interpretation of evidence. For instance, Davies and his colleagues reserve an explicit place for other kinds of evidence than science-based evidence by defining evidence-based policymaking as ‘the integration of experience, judgment and expertise with the best available external evidence from systematic research’ (Davies et al., 2000; p. 13). This view on the role of evidence in decision-making corresponds with Sackett’s definition of evidence-based medicine as ‘the integration of the best research evidence available with clinical expertise and patients’ values’ (Sackett et al., 2000: p.1). Evidence based on randomized-controlled trials (RCT) is not the only type of accepted evidence. Some authors prefer a broad definition of evidence-based policymaking. For instance, evidence-based policymaking is described as the process of integrating evidence-based interventions with community preferences to improve the health of the population (Kohatsu et al., 2004; Brownson et al., 2009).

Though the broad interpretation of the concept of evidence-based policymaking makes sense, it also obscures the distinction between evidence-based and not-evidence-based. Where to draw the line? This is not a purely theoretical issue because, as discussed in the previous section, policymakers use a very broad definition of evidence. They interpret all information they consider valid and relevant as evidence that must be factored into decision-making. Whether this information is evidence-based in the meaning of science-based or research-based is irrelevant in this respect.

The concept of evidence-based policymaking should not be misunderstood. It does not mean that policymakers must do what experts advise them to do. Advocates of evidence-based policymaking underscore that policymaking can never be completely evidence-based. Policymakers remain in charge of making policy decisions and always carry responsibility for their decisions. However, they should base these decisions as much as possible on evidence-based information.

In some areas of public health, evidence-based policymaking has a long history. For instance, public vaccination programs to protect the population against various diseases, including measles, mumps, and rubella (MMR), had a scientific basis. There is evidence that these programs have saved many lives. In his analysis of the effectiveness of the national vaccination program in the Netherlands, Van Wijhe (2018) estimated that mass vaccination campaigns had averted between six and twelve thousand deaths among those born between 1953 and 1992 and had reduced the number of reported disease cases, ranging from 50% for rubella to 90% for polio. The containment of COVID-19 by mass vaccination programs would not have been possible without the input from science.

The advance of health technology assessment in ‘package decisions’ is another manifestation of the role of evidence-based information in health policy. Rigorous research meeting the highest scientific standards is required to test the safety and (cost-)effectiveness of medical interventions. The need for evidence-based policymaking is also voiced in other areas of health policymaking. For instance, systematic empirical research into the effects of cost control policies should give insight into what works or does not work. Furthermore, evidence-based information on health risks has gained importance. Failing risk assessments can lead to expensive claims for compensation. Risk aversion and the ongoing juridification of relationships in modern society require evidence-based regulation to minimize health risks.

The concept of evidence-based health policy resonates with an optimistic belief in the power of science to contribute to health policymaking. However, this optimism is not undisputed because science mostly gives fewer answers than expected or hoped for. For this reason, some authors find it more appropriate to speak about ‘opinion-based’ policymaking (Segone & Prone, 2004) or ‘evidence-informed policymaking (Bowen & Zwi, 2006). There are also outright critical voices about the relevance of evidence in policy­making. Klein (2003) considers the concept of evidence-based health policymaking ‘a Delphic oracle difficult to decipher and apt to be mis­interpreted’. He considers health policymaking a process of trial and error and holds the assumption of a linear road from evidence to policymaking for ‘woefully inadequate’ (p. 429). Klein does not deny that science can contribute to policymaking, but the scien­tific community should give up the ‘delusional vanity’ of evidence-based policymaking. Rigorous and fast evaluations to learn from previous policies work better. History itself constitutes an important source of valuable information for policymakers.

Meanwhile, the call for evidence-based information in policymaking is not without risks. For instance, in her study of the role of evidence in the formulation of the European regulation on the provision of food information to consumers, Passarani (2019) observed that it had been easier to quantify the costs of food information for the industry and the retail sector than the benefits of food information to the public. Because policymakers were inclined to take quantitative evidence as more convincing than qualitative information, the public health community was disadvantaged. Likewise, Ter Meulen has warned of some potential ethical risks of evidence-based medicine. Declaring randomized-clinical trials (RCT) the ‘golden standard’ for evidence may exclude other kinds of research, such as observational research and qualitative studies, for building evidence with the result that treatments that are not suitable for a RCT have fewer chances to become reimbursable in health insurance. Patients are the ultimate victims of an exaggerated reliance on RCT in health policy (Ter Meulen et al., 2005). Finally, it should be kept in mind that scientific consensus can consign collegial critics to the margins and ultimately even result in ex-communication. Did not all scientists before Copernicus believe that the Earth was the center of the universe?

8.7 Contributions of evidence-based information to health policymaking

According to Weiss (1979), evidence-based information (Weiss uses the term research-based information) can have an instrumental, enlightenment function and political function in policymaking.

The instrumental function refers to the practical application of evidence-based information in policymaking. Information has an instrumental role if policy­makers collect information for problem-solving. For instance, they need informa­tion on the size and structure of policy problems or information on the potential effects and risks of alternative inter­ventions. The instrumental function of evidence-based information also captures the application of basic scientific research in policymaking. New know­ledge derived from basic research finds its way into practice. Weiss emphasizes that the findings of basic research in natural science are usually more compelling and authoritative than the findings of social research.

Secondly, evidence-based information can have an enlightenment function. Here, science is a source of new concepts and theoretical perspectives permeating the policymaking process. Science connects information in a causation model that gives insight into the relationships between observations. Nothing is as pract­ical as a good theory. Several paradigmatic shifts in health policymaking root in research. For instance, the emphasis upon health protection and health promotion draws upon research into the impact of external factors on health and disease. Research on the origins and spreading of cholera shed new light on how cholera outbreaks could be prevented.

Finally, evidence-based information can serve political goals. For instance, policymakers ask for more research to delay action or justify inaction, arguing that prudent policymaking requires more information. Another example is to create confusion. The tobacco industry has followed this strategy by spending large amounts of financial resources on research projects, the only purpose of which was to cast doubt on the relationship between smoking and lung cancer (Oreskes & Conway, 2011). Sometimes, policymakers refer to evidence-based information to legitimize their policy decisions. When the Dutch Prime Minister said that the government heavily relied upon the policy recommendations of the Out­break Management Team on how to respond to the COVID-19 pandemic, he used the latest available epidemiological information as legitimation for the government’s radical decisions to fight the pandemic.

8.8 Limits to the scientification of policymaking

There are several reasons for not overestimating the role of evidence-based evidence in health policymaking. First, all public policy­making is essentially a value-based activity to achieve something desirable. Science can support policymakers in accomplishing this task but cannot bridge the gap between ‘what is’ or ‘what works’ on the one hand and ‘what ought to be’ on the other hand. There is no definite scientific evi­dence for arguing that healthcare financing should rest upon the principles of solidarity: the choice for or against income solidarity is a political choice! Policymaking is a ‘trans-scientific’ activity (Majone, 1989). This is even true for the resolution of seemingly technical problems. For instance, scientists can inform policymakers on the toxicity of chemical products but cannot determine which level of toxicity is tolerable from a public health perspective. Setting standards is ultimately a value-bound activity requiring a political decision.

A second reason is that health policymaking cannot be reduced to an information-based process. According to Cairney (2016), the call for science-based policymaking ignores the dynamics of the policymaking process. Ideological convictions, material and immaterial interests, and power considerations always influence the course and outcome of policymaking. Policymakers always cope with uncertainties. By its focus on information, the call for science-based information perfectly fits in the instrumental approach to policymaking and neglects its political dimension. Actually, the call for science-based policymaking is tantamount to a call for the depoliticization of health policymaking.

Third, there is a fundamental gap between the ‘logic’ of science and the ‘logic’ of policymaking. Policymakers and researchers seemingly live in two different communi­ties (Caplan 1979). While science is directed at building know­ledge, policymaking is pragmatic, action-oriented, and often focused on short-term issues. While scientific know­ledge is propelled by systematic doubt, policymakers hate doubt and want to radiate confidence in the rightness of their decisions. Policymakers also feel uncomfortable with the abstractness and sometimes esoteric nature of scientific theories which they consider at odds with the complexity of the real world they act in. Error terms and confidence intervals which are common in econometric analysis are not helpful for policymakers pretending certainty. What further complicates the science of science-based information is that scientists frequently speak with many voices, confusing policymakers about who is right and wrong. It should be noticed, however, that the problem of many voices also creates opportunities for select­ive shopping or cherry-picking. Both policymakers and opponents use the scient­ific input that best suits their preferences. The risk of a confirmation bias (‘myside bias’) is always lurking. 

There are more reasons for a skeptical attitude towards the ‘scientification’ of policymaking. For instance, it is a matter of fact that the complexity of moderately structured and particularly unstructured problems is (largely) beyond the problem-solving capacity of scientific research (Hoppe, 2011). Research can help to unravel these problems or explore the potential effects of alternative policy interventions but cannot fully grasp their complexity. Doing science is being selective and making simplifying assumptions. Relevant contextual factors are often left out of consideration (decontextualization). Other reasons are that evidence-based knowledge is not available, incomplete, or too late. The out­worn phrase that ‘more research is needed’ is not helpful for policymakers being under political pressure to take action. 

Advocates of a scientific approach to policymaking proclaim that information should precede action to avert mistakes. There are two problems with this ‘know­ledge-then-action’ approach. First, much relevant information is only gained by doing. In other words, policymaking means policy learning. Second, opponents to the ‘knowledge-then-action’ approach warn of the risks inherent to this approach. Abstaining from action because of the quest for certainty or more information can do much harm. Learning by doing is an alternative and pragmatic approach (Greenhalgh & Engebretsen, 2022).

Finally, policymakers may demonstrate disinterest in scientific evidence. Sometimes, commissioning research to give policy decisions a scientific base is little more than an obligatory ritual dance. Research is not commissioned to buttress policy decisions with evidence-based information, but to legitimize these decisions that have already been made at an earlier stage (Box 8.5).

Finally, it should be kept in mind that the predilection for science-based policymaking is not without risks. Policymaking dominated by scientific experts may become a tech­no­cratic activity (Weingart, 1999). Through hiding themselves behind these experts, real power passes from policymakers to experts and fundamental policy choices may remain concealed. At the same time, a dominant role of scientific experts in public policymaking can put them in a vulnerable position. They run the risk of becoming involved in political disputes. 

8.9 Uncertainty and risk 

The rational model accords information a central place in policymaking. Policy decisions should rest on the best information available. But what if information is not or only partly available and policymakers see themselves confronted with uncer­tainty? Actually, this is the default situation. Uncertainty is inherent to all policymaking. Policymakers never possess complete information about what is going on, what the effects and costs of their policy measures will be, how opponents will react, what the next day may bring, and so on. Policymakers claiming the ‘truth’ fool themselves. Overconfidence has proven to be a source of avoidable policy failures. Policymaking during the outbreak of COVID-19 resembled in many respects sailing in the fog (Box 8.6).

Uncertainty is linked to risks. While some risks are known, other risks are unknown. There are even unknown unknowns. Risk can be defined as the probability of an occurrence multiplied by the extent of damage, injury, or loss. The problem with this definition is that it fails to understand risk as a social construction. Risk has not only an ‘objective’ but also a ‘subjective’ or man-made dimension. What one individual perceives as a big risk, another may perceive as a small risk. Objective risks can even be completely overlooked, and small risks be dramatically overestimated. Risk perception is a matter of sense-making influenced by historical, social, and cultural factors (Douglas, 1986). Furthermore, it can be influenced by political and bureaucratic skirmishes with­in the state ‘machinery’ in which participants ventilate their own version of the risk that must be encountered (Christensen & Painter, 2004). If policy­makers or stakeholders have an interest in emphasizing, amplifying, or mitigating the magnitude of risks, risk perception can easily become politicized. Risk perception is also critical in policy narratives (Versluis et al., 2019).

Of great interest in health policymaking are uncertain risks which Van Asselt and Vos (2006) define as ‘uncertainties that may inhibit danger’. Uncertain risks frequently arise for food safety, occupational health, and environmental hazards. Vaccination programs have always raised questions about potential adverse reactions and long-term effects. Uncertain risks are the product of technological innovation and are central to what Beck has called the ‘risk society’ (Beck, 1992). Policymakers may perceive these risks differently. See, for instance, how the World Health Organization, the European Union and its member states dealt with the Swine flu (Box 8.7).

8.10 Strategies to deal with uncertain risks

Policymakers follow various strategies in coping with uncertain risks. A distinction can be made between the following strategies: (a) doing policy research; (b) consultation; (c) reduction of complexity; (d) learning by doing; (e) application of the precautionary principle; (f) building a resilient health system; (g) covering up; (h) risk denial. Notice that the strategies of cover up and risk denial do not fit in the rational model of policymaking but in the conflict model (chapter 10). They are mentioned here for the reason of comple­teness. 

Doing policy research

Doing or commissioning policy research to gather information on policy problems and strategies to resolve these problems is a straightforward strategy to deal with uncertain risks. However, policy research is no guarantee for success because of information problems, simplistic assumptions, false inferences, or ignorance of relevant data. Conclu­sions and recommendations may be biased for political reasons and contain service­able truths. 

Consultation 

A second strategy is consultation. Many policy failures could have been avoided, had policymakers better listened to well-informed experts or well-informed stakeholders. Consultation can also bring uncertainty information to light (Van Asselt & Vos, 2006). However, expert or stakeholder information can be wrong, selective, or biased for political reasons. Furthermore, more information does not necessarily mean less uncertainty. If experts or stakeholders disagree with each other and feed policy­makers with contradictory information, consultation may even result in (more) confusion, for instance, on the safety of food additives, growth promotors in animal food, or the effectiveness and safety of new vaccines.

Reduction of complexity

Reduction of complexity by only considering small or incremental policy changes is a third strategy for coping with uncertain risks. The rationale of the strategy is to reduce the need for information. The probability of unanticipated risks is lowest if policy­makers abstain from ambitious policy change. Reduction of complexity is cen­tral in the strategy of ‘incrementalism’ (Braybrooke & Lindblom, 1963). They describe this strategy as ‘moving away from social ills rather than moving toward a known and relatively stable goal’ (p. 71). Policymaking is a process of serial and largescale incremental policy chan­ges. If a change appears unsuccessful, it can be repaired by remedial action. Bray­brooke and Lindblom claim that incrementalism is a reasonable strategy in the context of multiple uncertain risks. Radical policy plans like reforms run the risk of doing more bad than good. On the other hand, however, incrementalism has been criticized for being a risk in itself. Piecemeal engineering or ‘muddling through’ will fail in the context of major threats and (creeping) crises (Boin et al., 2020).

Learning by doing

A common element of the above strategies is that information precedes decision-making. An alternative strategy is learning by doing. This strategy is inspired by the experience that much relevant information on the effects of policymaking can only be collected in practice. Pragmatic decision-making enables policymakers to adapt their policies to changing or unforeseen circumstances. 

Application of the precautionary principle

An alternative strategy to cope with uncertain risks is to make use of the precautionary principle. According to this principle, policymakers are legitimized to make protective decisions in the absence of conclusive evidence for the occurrence of an uncertain risk. Rationality calls for caution. The application of the principle is closely associated with technological change. Technological change is heralded as a manifestation of progress but often surrounded by concerns about uncertain risks. If these risks cannot be excluded, what then is an acceptable risk? Which risk standards must a product meet for market authorization? Questions like these play an important role in market regulations within the European Union. Governments also referred to the precautionary principle in fighting the COVID-19 pandemic. Absence of evidence is not evidence of absence.

The precautionary principle is closely associated with the uncertainty paradox (Van Asselt & Vos, 2006). This paradox holds that science cannot provide the conclusive evidence policymakers are hoping for to substantiate and legitimize their policy decisions. The dilemma of policymakers is that they nevertheless must make a decision. The precautionary principle offers them a way out. It legitimizes them to take action without hard evidence.

The precautionary principle is an open principle. When is it opportune to resort to it? Is any scientific dispute reason for resorting to it? Can the principle do more harm than good? The principle is also silent on the question of which policy measures should be taken and how it relates to other principles.

The precautionary principle plays an important role in setting risk standards. In the aftermath of various food-safety scandals (BSE, dioxin, salmonella, and others) health authorities have imposed ever stricter standards in an attempt to restore public confiden­ce in food safety (Vos, 2004). Strict procedures for testing the safety of vaccines and post-market surveillance are in place to avoid public health disasters that have taken place in the past. 

Building a resilient health system

It is a no-brainer that public health crises cannot be well predicted. Public health experts have frequently warned policymakers of the potential outbreak of new pandemics, but they could not inform them about the when, where, and how of these pandemics. How, then, should policymakers prepare themselves for the outbreak of a pandemic? A rational strategy is to build a resilient health system which can be described as a system that is able ‘to prepare for, manage and learn from a sudden and extreme disturbance. Resilience is about maintaining the core health system functions’ (Sagan et al., 2021: ix). A study of the European Observatory on Health Systems and Policies on how countries had dealt with the COVID-19 pandemic identified twenty lessons for how to strengthen the resilience of health systems, including, among others, effective political leadership, the development of a clear and timely policy response, strengthening monitor­ing, surveillance and early warning systems, transferring the best available evidence to policy, effective coordination within (horizontal) and across levels of government (vertical), and ensuring transparency, legitimacy and account­ability in policymaking (Sagan et al., 2021). Other requirements are the need for buffer capacity that can be rapidly mobilized, the organization of crisis simulations, and the reflection of normative dilemmas that may occur during a public health crisis. 

 Covering up

Sometimes, policymakers pursue a strategy of cover-up. In their analysis of the politics of SARS which caused fear and panic in 2002, Christensen and Painter (2004) concluded that China in the initial stage of the crisis had deliberately chosen this strategy. Important information about the event was kept from the public for ‘security reasons’. Many of SARS statistics were not just state secrets but even ‘military secrets’. Fear of economic damage also played a role. A delegation of experts from the World Health Organization that had planned to investigate the outbreak of SARS did not get immediate access to the Guang­dong province. To divert attention, the then-Chinese government blamed Hong Kong for the outbreak in Beijing. Christensen and Painter speculate that the strategy of cover-up cannot be separated from the political climate at that time. The crisis coincided with a period of leadership trans­ition. Political leaders wanted to avoid any trouble and maintain calm and stability. After new leadership had come into power, China made a U-turn by promising more transparency and greater international cooperation. Restoring confidence in China became a priority.

Risk denial

Risk denial is a purposive strategy to soften or ignore risks. Risk denial occurs when policymakers underestimate or overestimate risks against their better judgment. Former President Trump repeatedly used this strategy to downplay the impact of COVID-19. The Atlantic published a long list of what it called ‘An unfinished compendium of Trump’s overwhelming dishonesty during a national emergency’ (Paz, 2020). On several occasions the president publicly contradicted his main public health advisors including the director of the US Center for Disease Control and his chief medical advisor. In February 2020 he told the nation that ‘the outbreak would be tempora­ry: ‘It’s going to disappear. One day, it’s like a miracle—it will disappear.’ He also boasted that ‘Coronavirus numbers are looking MUCH better, going down almost everywhere,’ and cases are ‘coming way down.’ He said this when coronavirus cases were increasing or plateauing in most American states (Christakis, 2021). 

8.11 Conclusion and suggestions for health policy analysis

The rational model postulates that policymaking should not be the outcome of political struggle, ideological convictions or power relations but rest upon the best available information. The synoptic model describes how policymaking should ideally be organized to achieve the best results. The alternative deliberative model underscores the role of argumentation, interpretation, multiple advocacy, and justification in policy analysis. Policymaking requires the use of various sources of information. 

The rational model has important implications for health policy analysts. As researchers, they must study the role of information in the policymaking process. Suggestions for research questions are:

• Which information from which sources do policymakers refer to in justifying their policy choices, the organization of the policymaking process, the structure of the governance system, and the lesson they draw from policy evaluation? 

• Which information and information sources are undervalued or not taken into account? How do policymakers convert observations into information? What is the conceptual model that underlies the selection of observations, the conversion of observations into information, and the interpretation of the information? 

• Is health policy organized as a technocratic process dominated by field experts or is there ample room for deliberation, argumentation, and multiple advocacy? 

• How much importance do policymakers attach to evidence-based or research-based information warranting their policy assumptions and choices? Which factors influence and restrict the use of this type of information? 

• Does essential information reach the inner circle of policymaking (decision-center), and which factors filter the influx of this information to this center?

• Which uncertainties must policymakers deal with? Are they sufficiently aware of these risks and which strategies do they follow to cope with them?

In their role of policy advisor, the task of health policy analysts is to feed policymakers with information and, in the awareness that information is always manufactured know­ledge, to scrutinize its validity and credibility. ‘Speaking truth to power’ (Wildavsky, 1979) to preserve policymakers from avoidable mistakes means that they must build up policy-issue expertise to sift the wheat from the chaff. The role of ‘producer of arguments’ (Majone, 1989) requires personal credibility. However, policy-issue know­ledge only is not sufficient. Health policy analysts must also acquire policymaking knowledge to be effective. They must know when and how to feed policymakers with information to ensure they are well-informed in their decision-making. In addition, well-informed means that policymakers are fed with information on uncertainties and risks. Last but not least, well-informed means that policymakers are aware of political obstacles and the role of information in political conflicts. This is the topic of chapter 10. 

References Chapter 8

Beck U (1992). Risk Society: Towards a New Modernity. Sage Publications.

Boin A, Ekengren M, Rhinard M (2020). Hidden in Plain Sight: Conceptualizing the Creeping Crisis. Risk, Hazard & Crisis in Public Policy, 11(2): https://doi.org/ 10.1002/ rhc3. 12193

Boin A, McConnel A, ‘t Hart P (2021). Governing the Pandemic. The Politics of Navigating a Mega-crisis. Palgrave MacMillan.

Bowen S, Zwi A (2005). Pathways to ‘Evidence-Informed Policy and Practice: a Frame­work for Action. Plos Medicine, 2(7): e166, https://doi.org/10,1371/journ. pmed.00201 66

Braybrooke D, Lindblom Ch (1963). A Strategy of Decision. Policy Evaluation as a Social Process. The Free Press.

Brownson R, Baker E, Fielding J, Maylahn Chr (2009). Evidence-based Public Health: A Fundamental Concept For Public Practice DOI: 10.1146/annurev.publhealth.031308.100134

Cairney P (2016). The Politics of Evidence-based Policymaking. Palgrave Pivot.

Caplan N (1979). The Two-communities Theory and Knowledge Utilization. American Beha­vioral Scientist, 22(3): 459-470. doi.org/10.1177/000276427902200308

Christakis N (2021). Apollo’s Arrow: The Profound and Enduring Impact of Coronavirus on the Way We Live. Little Brown & Company.

Christensen T, Painter M (2004). The Politics of SARS – Rational Responses or Ambi­guity, Symbols and Chaos? Policy and Society, 23(2): 18-48. doi: 10.1016/S1449-4035(04)70031-4.

Colebatch H (2009). Policy. Open University Press (3^rd edition).

Davies H, Nutley S, Smith P (eds) (2000). What Works? Evidence-Based Policy and Prac­tice. Policy Press.

Douglas M (1986). How Institutions Think. Syracuse University Press.

Greenhalgh T, Engebretsen E (2022). The Science-policy Relationship In Times of Crisis: An Urgent Call for a Pragmatic Turn. Social Science & Medicine, 306:115140. doi: 10.1016/j.socsci med .2022.115140.

Hajer M, Wagenaar H (eds) (2003). Deliberative Policy Analysis. Understanding Governance in the Network Society. Cambridge University Press.

House of Commons, Health and Social Care, and Science and Technology Committees (2021). Coronavirus: Lessons Learned To Date (report).

Klein R (2003). Evidence and Policy: Interpreting the Delphic Oracle. Journal of the Royal Society of Medicine 96(9): 429-431. doi: 10.1177/014107680309600903.

Kohatsu N, Robinson J, Torner J (2004). Evidence-based Public Health: An Evolving Concept. American Journal of Preventive Medicine 27(5): 417-421. doi: 10.1016/j.amepre.2004. 07.019.

Lindblom Ch (1950). The Science of Muddling Through. Public Administration Review, 19(2): 79-88.

Lohse S, Canali S (2021). Following ‘the’ Science? On the Marginal Role of the Social Sciences in the COVID-19 Pandemic. European Journal of the Philosophy of Science 11:99 doi: s13194-021-00416-y

Lomas J, Brown A (2009). Research and Advice Giving: a Functional View of Evidence-informed Policy Advice in a Canadian Ministry of Health. Milbank Quarterly, 87(4): 903-926. doi.org/ 10.1111/j.1468-0009.2009.00583.x

Majone G (1989). Evidence, Argument & Persuasion in the Policy Process. Yale University Press.

McNair B (2003). An Introduction to Political Communication (3rd ed.). Routledge.

Oreskes N, Conway E (2011) Merchants of Doubt. Bloomsbury.

Passarani I (2019). Role of Evidence in the Formulation of European Public Health Policies. A Comparative Case Study Analysis. Maastricht University (PhD-study).

Paz Chr (2020). An Unfinished Compendium of Trump’s Overwhelming Dishonesty During a National Emergency. The Atlantic.

Reddy L, Nolan (2018). Bill Digest: Health (Regulation of Termination of Pregnancy) Bill 2018 (report).

Rittel H, Webber M (1973). Dilemmas in the General Theory of Planning. Policy Scien­ces, 4: 155-169.

Rovers E (2022). Nu is het aan ons: oproep tot echte democratie. De Correspondent.

Sackett D, Strauss S, Richardson W, Rosenberg W, Haynes R (2000). Evidence-based Medicine: How to Teach and Practice EBM. Churchill Livingstone (2^nd edition).

Sagan A, Webb E, Azzopardi-Muscat N, De la Mata I, McKee M, Figueras J (eds) (2021). Health System Resilience During COVID-19: Lessons for Building Back Better. European Observatory on Health Systems and Policies.

Segone M, Prone (2008). The Role of Statistics in Evidence-based Decision Making. UNICE Work Session on Statistical Dissemination and Communication.

Slovic P (1999). Trust, Emotion, Sex politics and Science – Surveying the Risk Battlefield. Risk Analysis, 19(4): 689-701. DOI: 10.1023/a:1007041821623

Simon H (1991). Bounded Rationality and Organizational Learning. Organization Science, 2(1): 125-134. doi.org/10.1287/orsc.2.1.125

Skidelsky R (1992). John Maynard Keynes: a Biography. Volume 2: the Economist as Saviour, 1920-1937. MacMillan.

‘t Hart P (1991). Irving L Janis ’ Victims of Groupthink. Political Psychology, 12(2): 247-278. doi.org/10.2307/3791464

Ter Meulen R, Biller-Adorno N, Lenk Chr, Lie R (eds) (2005). Evidence-based Practice in Medicine & Health Care: a Discussion of the Ethical Issues. Springer.

Van Asselt M, Vos E (2006). The Precautionary Principle and the Uncertainty Paradox. Journal of Risk Research, 9(4): 313-336. doi.org/10.1080/13669870500175063

Van Wijhe M (2018). The Public Health Impact of Vaccination Programmes in the Netherlands. University of Groningen (dissertation).

Versluis E, van Asselt M, Kim J (2019). The Multilevel Regulation of Complex Problems: Uncertainty and the Swine Flu Pandemic. European Policy Analysis, 5(1): 80-98. doi: 10. 1002/ epa2.1064.

Vos E (2004). Overcoming the Crisis of Confidence: Risk Regulation in an Enlarged European Union. Maastricht University.

Weingart P (1999). Scientific Expertise and Political Accountability: Paradoxes of Science in Politics. Science and Public Policy, 26(3): 151-161. doi.org/10.3152/147154399781782437

Weiss C (1979). The Many Meanings of Research Utilization. Public Administration Review, 39(5): 426-431. doi.org/10.2307/3109916

Wildavsky A (1987). Speaking Truth to Power. The Art and Craft of Policy Analysis. Rout­ledge.