Mathematical models are used to simulate complex real-world phenomena in many areas of science and technology. Large complex models typically require inputs whose values are not known with certainty. Uncertainty analysis aims to quantify the overall uncertainty within a model, in order to support problem owners in model-based decision-making. In recent years there has been an explosion of interest in uncertainty analysis. Uncertainty and dependence elicitation, dependence modelling, model inference, efficient sampling, screening and sensitivity analysis, and probabilistic inversion are among the active research areas. This text provides both the mathematical foundations and practical applications in this rapidly expanding area, including:

• An up-to-date, comprehensive overview of the foundations and applications of uncertainty analysis.
• All the key topics, including uncertainty elicitation, dependence modelling, sensitivity analysis and probabilistic inversion.
• Numerous worked examples and applications.
• Workbook problems, enabling use for teaching.
• Software support for the examples, using UNICORN – a Windows-based uncertainty modelling package developed by the authors.
• A website featuring a version of the UNICORN software tailored specifically for the book, as well as computer programs and data sets to support the examples.

Uncertainty Analysis with High Dimensional Dependence Modelling offers a comprehensive exploration of a new emerging field. It will prove an invaluable text for researches, practitioners and graduate students in areas ranging from statistics and engineering to reliability and environmetrics.

Dorota Kurowicka, Delft University of Technology, The Netherlands.

Roger M. Cooke, Delft University of Technology, The Netherlands.

Uncertainty Analysis with High Dimensional Dependence Modelling

By Dorota Kurowicka Roger Cooke

John Wiley & Sons

Copyright © 2006 John Wiley & Sons, Ltd.
All right reserved.
ISBN: 978-0-470-86306-0

Chapter One

Introduction: Uncertainty Analysis and Dependence Modelling

1.1 Wags and Bogsats

‘… whether truef or not [it] is at least probable; and he who tells nothing exceeding the bounds of probability has a right to demand that they should believe him who cannot contradict him’. Samuel Johnson, author of the first English dictionary, wrote this in 1735. He is referring to the Jesuit priest Jeronimo Lobo’s account of the unicorns he saw during his visit to Abyssinia in the 17th century (Shepard (1930) p. 200).

Johnson could have been the apologist for much of what passed as decision support in the period after World War II, when think tanks, forecasters and expert judgment burst upon the scientific stage. Most salient in this genre is the book The Year 2000 (Kahn and Wiener (1967)) in which the authors published 25 ‘even money bets’ predicting features of the year 2000, including interplanetary engineering and conversion of humans to fluid breathers. Essentially, these are statements without pedigree or warrant, whose credibility rests on shifting the burden of proof. Their cavalier attitude toward uncertainty in quantitative decision support is representative of the period. Readers interested in how many of these even money bets the authors have won, and in other examples from this period, are referred to (Cooke (1991), Chapter 1).

Quantitative models pervade all aspects of decision making, from failure probabilities of unlaunched rockets, risks of nuclear reactors and effects of pollutants on health and the environment to consequences of economic policies. Such quantitative models generally require values for parameters that cannot be measured or assessed with certainty. Engineers and scientists sometimes cover their modesty with churlish acronyms designating the source of ungrounded assessments. ‘Wags’ (wild-ass guesses) and ‘bogsats’ (bunch of guys sitting around a table) are two examples found in published documentation.

Decision makers, especially those in the public arena, increasingly recognize that input to quantitative models is uncertain and demand that this uncertainty be quantified and propagated through the models.

Initially, it was the modellers themselves who provided assessments of uncertainty and did the propagating. Not surprisingly, this activity was considered secondary to the main activity of computing ‘nominal values’ or ‘best estimates’ to be used for forecasting and planning and received cursory attention.

Figure 1.1 shows the result of such an in-house uncertainty analysis performed by the National Radiological Protection Board (NRPB) and The Kernforschungszentrum Karlsruhe (KFK) in the late 1980s (Crick et al. (1988); Fischer et al. (1990)). The models in question predict the dispersion of radioactive material in the atmosphere following an accident in a nuclear reactor. The figure shows predicted lateral dispersion under stable conditions, and also shows wider and narrower plumes, which the modellers are 90% certain will enclose an actual plume under the stated conditions.

It soon became evident that if things were uncertain, then experts might disagree, and using one expert-modeller’s estimates of uncertainty might not be sufficient. Structured expert judgment has since become an accepted method for quantifying models with uncertain input. ‘Structured’ means that the experts are identifiable, the assessments are traceable and the computations are transparent. To appreciate the difference between structured and unstructured expert judgment, Figure 1.2 shows the results of a structured expert judgment quantification of the same uncertainty pictured in Figure 1.1 (Cooke (1997b)). Evidently, the picture of uncertainty emerging from these two figures is quite different.

One of the reasons for the difference between these figures is the following: The lateral spread of a plume as a function of down wind distance x is modelled, per stability class, as

[sigma](x) = A[x.sup.B].

Both the constants A and B are uncertain as attested by spreads in published values of these coefficients. However, these uncertainties cannot be independent. Obviously if A takes a large value, then B will tend to take smaller values. Recognizing the implausibility of assigning A and B as independent uncertainty distributions, and the difficulty of assessing a joint distribution on A and B, the modellers elected to consider B as a constant; that is, as known with certainty.

The differences between these two figures reflect a change in perception regarding the goal of quantitative modelling. With the first picture, the main effort has gone into constructing a quantitative deterministic model to which uncertainty quantification and propagation are added on. In the second picture, the model is essentially about capturing uncertainty. Quantitative models are useful insofar as they help us resolve and reduce uncertainty. Three major differences in the practice of quantitative decision support follow from this shift of perception.

First of all, the representation of uncertainty via expert judgment, or some other method is seen as a scientific activity subject to methodological rules every bit as rigorous as those governing the use of measurement or experimental data.

Second, it is recognized that an essential part of uncertainty analysis is the analysis of dependence. Indeed, if all uncertainties are independent, then their propagation is mathematically trivial (though perhaps computationally challenging). Sampling and propagating independent uncertainties can easily be trusted to the modellers themselves. However, when uncertainties are dependent, things become much more subtle, and we enter a domain for which the modellers’ training has not prepared them.

Finally, the domains of communication with the problem owner, model evaluation, and so on, undergo significant transformations once we recognize that the main purpose of models is to capture uncertainty.

1.2 Uncertainty analysis and decision support: a recent example

A recent example serves to illustrate many of the issues that arise in quantifying uncertainty for decision support. The example concerns transport of Campylobacter infection in chicken processing lines. The intention here is not to understand Campylobacter infection, but to introduce topics covered in the following chapters. For details on Campylobacter, see Cooke et al. (Appearing); Van der Fels-Klerx et al. (2005); Nauta et al. (2004).

Campylobacter contamination of chicken meat may be responsible for up to 40% of Campylobacter-associated gastroenteritis and for a similar proportion of deaths. A recent effort to rank various control options for Campylobacter contamination has led to the development of a mathematical model of a processing line for chicken meat (these chickens are termed ‘broilers’).

A typical broiler processing line involves a number of phases as shown in Figure 1.3. Each phase is characterized by transfers of Campylobacter colony forming units from the chicken surface to the environment, from the environment back to the surface and from the faeces to the surface (until evisceration), and the destruction of the colonies. The general model, applicable with variations in each processing phase, is shown in Figure 1.4.

Given the number of Campylobacter on and in the chickens at the inception of processing, and given the number initially in the environment, one can run the model with values for the transfer coefficients and compute the number of Campylobacter colonies on the skin of a broiler and in the environment at the end of each phase. Ideally, we would like to have field measurements or experiments to determine values for the coefficients in Figure 1.4. Unfortunately, these are not feasible. Failing that, we must quantify the uncertainty in the transfer coefficients, and propagate this uncertainty through the model to obtain uncertainty distributions on the model output.

This model has been quantified in an expert judgment study involving 12 experts (Van der Fels-Klerx et al. (2005)). Methods for applying expert judgments are reviewed in Chapter 2. We may note here that expert uncertainty assessments are regarded as statistical hypotheses, which may be tested against data and combined with a view to optimizing performance of the resulting ‘decision maker’.

The experts have detailed knowledge of processing lines, but owing to the scarcity of measurements, they have no direct knowledge of the transfer mechanisms defined by the model. Indeed, use of environmental transport models is rather new in this area, and unfamiliar. Uncertainty about the transfer mechanisms can be large, and, as in the dispersion example discussed in the preceding text, it is unlikely that these uncertainties could be independent. Combining possible values for transfer and removal mechanism independently would not generally yield a plausible picture. Hence, uncertainty in one transfer mechanism cannot be addressed independently of the rest of the model.

Our quantification problem has the following features:

There are no experiments or measurements for determining values.

There is relevant expert knowledge, but it is not directly applicable.

The uncertainties may be large and may not be presumed to be independent, and hence dependence must be quantified.

These obstacles will be readily recognized by anyone engaged in mathematical modelling for decision support beyond the perimeter of direct experimentation and measurement. As the need for quantitative decision support rapidly outstrips the resources of experimentation, these obstacles must be confronted and overcome. The alternative is regression to wags and bogsats.

Although experts cannot provide useful quantification for the transfer coefficients, they are able to quantify their uncertainty regarding the number of Campylobacter colonies on a broiler in the situation described below taken from the elicitation protocol:

At the beginning of a new slaughtering day, a thinned flock is slaughtered in a ‘typical large broiler chicken slaughterhouse’…. We suppose every chicken to be externally infected with [10.sup.5] Campylobacters per carcass and internally with 108 Campylobacters per gram of caecal content at the beginning of each slaughtering stage….

Question A1: All chickens of the particular flock are passing successively through each slaughtering stage. How many Campylobacters (per carcass) will be found after each of the mentioned stages of the slaughtering process each time on the first chicken of the flock?

Experts respond to questions of this form, for different infection levels, by stating the 5%, 50% and 95% quantiles, or percentiles, of their uncertainty distributions. If distributions on the transfer coefficients in Figure 1.4 are given, then distributions per processing phase for the number of Campylobacter per carcass (the quantity assessed by the experts) can be computed by Monte Carlo simulation: We sample a vector of values for the transfer coefficients, compute a vector of Campylobacter per carcass and repeat this until suitable distributions are constructed. We would like the distributions over the assessed quantities computed in this way to agree with the quantiles given by the combined expert assessments. Of course we could guess an initial distribution over the transfer coefficients, perform this Monte Carlo computation and see if the resulting distributions over the assessed quantities happen to agree with the experts’ assessments. In general they will not, and this trial-and-error method is quite unlikely to produce agreement. Instead, we start with a diffuse distribution over the transfer coefficients, and adapt this distribution to fit the requirements in a procedure called ‘probabilistic inversion’.

More precisely, let X and Y be n– and m-dimensional random vectors, respectively, and let G be a function from [[??].sup.n] to [[??].sup.m]. We call x [member of] [[??].sup.n] an inverse of y [member of] [[??].sup.m] under G if G(x) = y. Similarly, we call X a probabilistic inverse of Y under G if G(X) ~ Y, where ~ means ‘has the same distribution as’. If {Y|Y [member of] ITLITL} is the set of random vectors satisfying constraints ITLITL, then we say that X is an element of the probabilistic inverse of {Y|Y [member of] ITLITL} under G if G(X) [member of] ITLITL. Equivalently, and more conveniently, if the distribution of Y is partially specified, then we say that X is a probabilistic inverse of Y under G if G(X) satisfies the partial specification of Y. In the current context, the transfer coefficients in Figure 1.4 play the role of X, and the assessed quantities play the role of Y.

In our Campylobacter example, the probabilistic inversion problem may now be expressed as follows: Find a joint distribution over the transfer coefficients such that the quantiles of the assessed quantities agree with the experts’ quantiles. If more than one such joint distribution exists, pick the least informative of these. If no such joint distribution exists, pick a ‘best-fitting’ distribution, and assess its goodness of fit.

Probabilistic inversion techniques are the subject of Chapter 9.

In fact, the best fit produced with the model in Figure 1.4 was not very good. It was not possible to find a distribution over the transfer coefficients, which, when pushed through the model, yielded distributions matching those of the experts. On reviewing the experts’ reasoning, it was found that the ‘best’ expert (see Chapter 2) in fact recognized two types of transfer from the chicken skin to the environment. A rapid transfer applied to Campylobacter on the feathers, and a slow transfer applied to Campylobacter in the pores of the skin. When the model was extended to accommodate this feature, a satisfactory fit was found. The second model, developed after the first probabilistic inversion, is shown in Figure 1.5.

Distributions resulting from probabilistic inversion typically have dependencies. In fact, this is one of the ways in which dependence arises in uncertainty analysis. We require tools for studying such dependencies. One simple method is to simply compute rank correlations. Notions of correlation and their properties are discussed in Chapter 3. For now it will suffice simply to display in Table 1.1 the rank correlation matrix for the transfer coefficients in Figure 1.5, for the scalding phase.

Table 1.1 shows a pronounced negative correlation between the rapid transfer from the skin ([a.sub.extA]) and evacuation from the chicken (ca), but other correlations are rather small. Correlations of course do not tell the whole story. Chapter 7 discusses visual tools for studying dependence in high-dimensional distributions. One such tool is the cobweb plot. In a cobweb plot, variables are represented as vertical lines. Each sample realizes one value of each variable. Connecting these values by line segments, one sample is represented as a jagged line intersecting all the vertical lines. Plate 1 shows 2000 such jagged lines and gives a picture of the joint distribution. In this case, we have plotted the quantiles, or percentiles, or ranks of the variables rather than the values themselves. The negative rank correlation between [a.sub.extA] and ca is readily visible if the picture is viewed in colour: The lines hitting low values of [a.sub.extA] are red, and the lines hitting values of ca are also red.

(Continues…)

Excerpted from Uncertainty Analysis with High Dimensional Dependence Modellingby Dorota Kurowicka Roger Cooke Copyright © 2006 by John Wiley & Sons, Ltd.. Excerpted by permission.
All rights reserved. No part of this excerpt may be reproduced or reprinted without permission in writing from the publisher.
Excerpts are provided by Dial-A-Book Inc. solely for the personal use of visitors to this web site.