Making Unknowns Known: When is that necessary and unnecessary in science and policy?
Science is the quest for knowledge. That knowledge is accumulated by making predictions (developing hypotheses) and then making observations (collecting data) to accumulate evidence that supports or rejects the predictions. In this paradigm the goal is always more observations, so that more is known and what is predicted is better supported by the known evidence. Put another way, when the goal is knowledge there isn’t a way to have too much of it.
This is not true of policy. The primary goal of policy making, and decision making is not an accumulation of knowledge, it is an accumulation of better outcomes through quality decision making.
This is not to say that policy making does not benefit from knowledge. Ron Howard notes that decision making relies on the answers to three questions: What do you want? What can you do? and What do you know? Howard states, “If you do not have any information linking what you do to what will happen in the future, then all alternatives serve equally well because you do not see how your actions will have any effect”.
This quote provides the key difference in the paradigm of a scientist and a decision maker, in that one’s view of uncertainty caused by our gaps in knowledge can differ distinctly depending on our perspective. As David R. Smith says, “To a scientist, the key uncertainties are those that suggest interesting alternative hypotheses. To a decision maker, the key uncertainties are those that affect decision choice; in some cases it is possible to make a smart choice despite unresolved uncertainty.”
In addition, in some cases uncertainty can not be reduced and decisions must be made with the knowledge at hand. For example, there is some inherent randomness in life and some information is just too difficult or too costly to collect. There are tools policy makers can use in these cases, irreducible uncertainty may be an existential challenge to a scientist, but to a decision analyst it is just an indication that a risk analysis approach that incorporates the uncertainty is likely the best path to making a decision.
Where the challenge and sometimes conflict can arise between scientists and policy makers is when there is a gap in knowledge, and that gap could be closed or at least reduced. What then?
This, to my decision analyst eye, is when scientists need to recognize that their role when they are asked to support policy making for a specific policy is primarily to share knowledge.
For example, say a policy needed to be selected and implemented right now. The role for science can not be to list and evaluate additional hypotheses, but only to state what is known. While often there is some time between the state of a decision process and when an action will be implemented, there often is not so much time, or available resources, to make everything that we might like to know known.
In these cases the best approach is to first determine the value of additional knowledge. That is, to treat the decision to gain additional knowledge before acting as one of the many available actions for the decision makers to choose between. If the delay and costs of accumulating the knowledge are worth it because this is the best option, great! But if not, then policy is better made without that knowledge.
Whether collecting additional is a good choice in policy making will change from situation to situation. Perhaps as some sort of saying goes it is best to treat anyone who fails to recognize that the value of knowledge and the importance of knowledge gaps depend on the situation as trying to sell you something.
Supporting material:
Howard, RA. 2007. The foundations of decision analysis revisited. Pages 32-56 in Edwards W, Miles RFJ, von Winterfeldt D, eds. Advances in Decision analysis: From foundations to Applications. Cambridge UK: Cambridge University Press.
Smith, DR. 2020. Introduction to Prediction and the Value of Information. Pages 189- 195 in Runge MC, Converse SJ, Lyons JE, Smith, DR, eds. Structured Decision Making: Case Studies in Natural Resource Management. Baltimore USA: Johns Hopkins University Press.
