RESEARCH READING

Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences

Naomi Oreskes, Kristin Shrader-Frechette and Kenneth Belitz (1994, Science)

*My musings in italics

Verification, or establishing the truth, of numerical models is the first target of this paper. Beginning with the observation that public policy places high demands on relevant models, the authors begin with a description of open and closed systems. They then proceed to demonstrate that models, as open systems, cannot be verified.

Open systems have multiple influences and extenuating circumstances, leading to the possibility of a true claim failing verification - a false negative. However, a closed system's claims are derivable and thus verifiable mathematically (putting aside the issue of imperfect human & mechanical computation, not discussed here). The authors discuss some contributors to the incompleteness of open systems: imprecise inputs; continuum mechanics and a loss of fine structure; uncertain relations between small measured inputs and large generalised model inputs; and a priori assumptions embedded in the 'real world' dataset used for verification.

The last contributor is especially poignant, since a model's poor data fit may falsify one of these "auxiliary hypotheses" rather than the model. If the model is altered instead, it could lead to overfitting.

The authors note that modelling is under-determined, as multiple models can match the verification data. This leads to selection along “extraevidential considerations like symmetry… or metaphysical preferences.” Further, since components cannot usually be tested individually, multiple unseen errors may cancel out.

Moving onto validation, defined as a test of legitimacy or logical soundness, the authors write that it is independent of model results. Often, validation as a term is misused to mean verification. Examples are cited from the United States' Dept. of Energy and the International Atomic Energy Agency. Validation, they argue, is not the same as consistency between systems, nor can it be data-derived. A valid model can still be unreliable.


In discussing how numerical systems are 'verified' in practice, the authors state that solutions might be "bench-marked" against an analytical range. Extensions beyond this range, however, cannot be verified by definition, since the analytical result would be used.

Two-step calibration might be used, where only a portion of data is used to train the model and the rest to compare it with. Yet the authors claim that both steps are in fact calibration, since the second step may lead to parameter adjustment. This could lead to “fine-tuning,” especially in the competitive environment science is produced in with strong positive results preferentially published. See “Why most published research findings are false”, John P.A. Ioannidis. Yet it should be noted that techniques exist to circumnavigate this, e.g. using bootstrapping numerical re-sampling as per. "Diatoms as indicators of climatic change", Bigler & Hall 2002.

The authors discuss the difference between models having forced empirical adequacy, rather than verification, before moving to problems in extrapolation. Culprits include complex variables and magnified time-dependent errors.

Consequentially, the authors surmise that the veracity of models cannot be confirmed or demonstrated, only increased in probability, and that a statement of confirmation commits the fallacy of affirming the consequent. This is illustrated with a hypothetical claim that, if it rains, a person will be inside the house. The person is subsequently observed inside the house, leading to the fallacious deduction that it is raining. The fallacy arises here, as in modelling, because multiple models and theories explain the observations.

Finally, the authors discuss how 'verify' and 'validate' are used as affirmative terms, rather than terms of degree. They propose a neutral "precision and accuracy" evaluation language. Describing models as a heuristic, as a guide to further study but not susceptible to proof, the authors note that the modeller has a responsibility to “delineate the limits of the correspondence between the model and the natural world”. Since models can be affected by bias, their best use is in challenging hypotheses rather than verification.