RESEARCH READING - OCCAM'S RAZOR

 
The Role of Occam’s Razor in Knowledge Discovery

Pedro Domingos, Data Mining and Knowledge Discovery 3, 409–425 (1999)



There are two interpretations of Occam's razor according to Domingos. They are that models should be:

1) Comprehensible
2) Accurate

In this paper, Domingos draws on a range of research to argue that applying Occam's razor on a choice of models can result in selecting a comprehensible model, but not necessarily an accurate one.

Observing that simplicity has "no satisfactory computable definition", Domingos states that a heuristic definition of Occam's razor (for example, reducing the number of parameters) leads to two formulations:

1st Razor: From two models with equal predictive error, choose the simpler one for simplicities' sake.

2nd Razor: From two models with equal calibration error, choose the simpler one for lower predictive error.

The 1st is generally true, whilst the 2nd is generally theoretically and empirically false.

Starting with some theoretical arguments for the 2nd razor, Domingos looks at the case for the Bayesian information criterion (amongst others). The BIC assumes that model parameters are distributed normally across candidate models. This gives the logarithmic probability of a certain model structure as equal to the likelihood of the structure given some calibration procedure, reduced by a complexity term dependent on the number of parameters. The weaknesses of this approach range from the long chain of assumptions required to use the BIC to the calculated probability being that of the structure, not the model. An example of a quadratic candidate is presented. As it has more model space than a linear model, the linear structure has a higher probability since the larger number of inaccurate quadratic candidates dilute the quadratic structure probability, even for cases where a quadratic structure is the more accurate.

Another theoretical argument for the 2nd razor comes from the idea of the minimum description length (MDL). The argument states that the best model uses the smallest number of bits to code for it and the data, given the model. Domingos argues that the belief that a trade-off between error and complexity results from Bayes' theorem is circular reasoning. This is because it is equivalent to stating that models with higher priors have shorter codes, but also that shorter coded models have higher priors. Sometimes, a complex model with the highest prior can be coded with shorter code compared to a simpler, lower prior model. In other words, after giving each model a prior, the models can be recoded such that those with the highest priors have the shortest code. A short code length does not imply better predictions or a more comprehensible model.

Moving on to theoretical arguments against the 2nd razor, the "No free lunch" mathematical theorem is briefly mentioned. It results from the underdeterminancy in model selection (arising from the fact that model fitting is an open system, with several candidate models always able to match calibration data - see the Oreskes post). The generalization of a model is discussed as resulting from it's Vapnik–Chervonenkis dimension, not the number of parameters. It is possible to have a model with an infinite VC dimension yet just one parameter.

Overfitting is often erroneously thought to be due to complex models. However, it is actually due to multiple comparisons. The probability of a model fitting the calibration data purely through chance rises if more candidates are selected from. Models with many parameters tightly constrained may thus be less susceptible to overfitting than broader, simpler models.

A final set of problems discussed by Domingos involve projections of systematic and random error. Heuristics that minimise complexity often assume rising complexity produces a faster increase in systematic than random error. This is demonstrably not the case for some systems.

Turning his attention to empirical tests of the 2nd razor, Domingos notes that corrections for multiple testing often produce better models than those based on MDL. Further, although the accuracy gain of complex versus simple models may be small it is not necessarily negligible. After providing examples of complex decision-tree models, with the right constraints, with more accuracy than simple ones, some physics examples are presented. For example, both the Copernican and Ptolemaic orbital theories had the same predictive error, so preferring the former was selection using the 1st, not the 2nd, razor (something which was of special interest to me given my background! A theme alluded to was the idea of simplicity resulting from the use of the model, not the final result. A Kuhnian paradigm shift could occur if an unwieldy model with too many patches was replaced by a leaner one. Perhaps the 1st razor is more central to scientific understanding than otherwise recognised?).

Geocentric model. Ptolemaic systems introduced epicycles to solar system bodies, reconciling it with observations. Image from redorbit.



Outright tests of simplicity versus accuracy in e.g. decision-tree modelling generally show that complex models are more accurate than the simple. Complex MDL systems with redundancy, or multi-model ensembles, are more accurate. Domingos states, from this evidence, that the 2nd razor is "typically false". The issue of errors in computation is unmentioned. Domingos does mention cognitive research having implications for comprehension.

Returning to the 1st razor, it is noted that since simplicity is not the same as comprehensibility, it can be rephrased in terms of domain-dependent comprehensibility.

The 2nd razor is then finally discussed as being trivially true after model coding, but of no help in calibration or selection. It is better to prevent overfitting by using domain knowledge, with the added bonus of increased comprehensibility to those with such knowledge. Domingos mentions ecosystem modelling using LAGRAMGE, a 'declerative bias' equation discovery program. The authors of the relevant paper found a model for phytoplankton growth in Lake Glumsoe with appropriate terms and accurate predictions. Pre-existing knowledge of algal growth was used in its construction. See Todorovski and Dzeroski, Declarative bias in equation discovery. Proceedings of the Fourteenth International Conference on Machine Learning, Nashville, TN: Morgan Kaufmann, pp. 376–384 (1997).

Complex ensemble models can be made comprehensible by choosing representative models, with lower, but better, accuracy than similarly structured single models. Explaining a model's results after calculation is also often more comprehensible than coding a fully comprehensible model.

In conclusion, it is recommended to use domain knowledge and the 1st razor when modelling, and to treat model accuracy separately from comprehensibility.


*My musings in italics

MODELLING APPROACHES - SOIL-MOISTURE BALANCE MODELLING

During yesterday's lecture on environmental modelling, Pf. Richard Taylor demonstrated an accounting model of groundwater recharge. His talk was very informative and entertaining, building up from the basics of groundwater systems to reach some profound conclusions, such as the tendency for global climate models (GCM) to overlook subsurface water flow and where that philosophy might originate. 

I'd like to expand on many of the modelling themes that he touched on. The accounting model was based on a 55yr dataset from Uganda, the details of which can be found here: Taylor, R.G., Todd, M., Kongola, L., Nahozya, E., Maurice, L., Sanga, H. and MacDonald, A., Evidence of the dependence of groundwater resources on extreme rainfall in East Africa. Nature Climate Change, Vol. 3, pp.374-378 (2013).

The accounting model looks at the water level of the saturated zone. A groundwater recharge event raises the water table. From the UK Groundwater Forum.


Taylor first  mentioned the WaterGAP model, as used in Portmann, F. et al., Impact of climate change on renewable groundwater resources: assessing the benefits of avoided greenhouse gas emissions using selected CMIP5 climate projections. Environmental Research Letters, Vol.8 (2013). Broadly speaking, the model was used by Portmann et al. to project future groundwater recharge (GWR). The relevant point here is that the GWR estimates were not calibrated.

In my post on the Oreskes paper it was seen that it is often difficult to calibrate components of complex models, due to small inputs generalized into large model inputs, feedback with other components, and selective or non-existent data on model inputs. In this case, no global observational data of GWR exists, making it impossible to calibrate WaterGAP's predictions. 

There is another serious shortcoming in the WaterGAP's GWR forecast - it determines a priori that greater precipitation leads to greater surface runoff, so it cannot be used to test this hypothesis. This arises because the model uses a simple cutoff threshold, whereby excess precipitation over this threshold goes straight to runoff. This precludes the possibility of alternate phenomena such as increased GWR.

As the Taylor et al. paper shows, the evidence from Uganda presents a different view. GWR only significantly occurs during extreme precipitation events. For example, the Makutapora record shows the 4-month 1997-98 ENSO contributed 25% of the total GWR in the 55 year record. Drawing two conclusions about modelling in the talk, Taylor (paraphrased) stated that models should "utilise historical data to [disprove] hypotheses" and "properly structured GWR models give better predictions", echoing the general sentiment from Oreskes et al.

Recalling an unfortunately futile effort by some hydrologists for representation in the IPCC's 5th assessment report, Taylor noted that GCMs tended to use a landsurface with no subsurface component. This reduced familiarity may lead to GCM modellers being less confident linking the subsurface component with climate change, misrepresenting GWR changes. 

Returning to the Taylor et al. model, Taylor outlined how a less complex accounting model based on local data was developed. It was built "from the ground up", using the empirical data. The model's simplicity came in part from the absence of physical knowledge of the flow of groundwater in that part of Uganda. The observed GWR was much more rapid than the most appropriate physical law, Richard's equation, would predict. It thus did not model process, but the water content in different parts of the ground over time (hence the "accounting" moniker).

In outlaying how the model extrapolated from the data into 2070, Taylor noted that most GCMs were trained by looking at past daily rainfall and distributing it in the most statistically appropriate way in the future (a "delta approach"). However, since a warmer world has less frequent and more intense precipitation (e.g. Pall et al., Testing the Clausius–Clapeyron constraint on changes in extreme precipitation under CO₂ warming. Climate Dynamics, Volume 28, Issue 4, pp 351-363 (March 2007)), the delta approach underestimates GWR from intense precipitation in a warmer world. This is especially true if focus recharge, such as from floods, become more common as a result of higher precipitation - another aspect that GCMs neglect.

Taylor made two other points that resonated with my physics background. In justifying Excel for modelling, he pointed out that "few colleagues have Matlab licenses or R experience, whereas everyone has Excel" and "simple models with calibrated variables are better than complex models with broad, uncalibrated components". A models' end-users, the modelling community, as well as Occam's Razor, will be considered further on this blog in the future.

DISCUSS

Tom Murphy's insightful post on the energy trap.

RESEARCH READING

Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences

Naomi Oreskes, Kristin Shrader-Frechette and Kenneth Belitz (1994, Science)

*My musings in italics

Verification, or establishing the truth, of numerical models is the first target of this paper. Beginning with the observation that public policy places high demands on relevant models, the authors begin with a description of open and closed systems. They then proceed to demonstrate that models, as open systems, cannot be verified.

Open systems have multiple influences and extenuating circumstances, leading to the possibility of a true claim failing verification - a false negative. However, a closed system's claims are derivable and thus verifiable mathematically (putting aside the issue of imperfect human & mechanical computation, not discussed here). The authors discuss some contributors to the incompleteness of open systems: imprecise inputs; continuum mechanics and a loss of fine structure; uncertain relations between small measured inputs and large generalised model inputs; and a priori assumptions embedded in the 'real world' dataset used for verification.

The last contributor is especially poignant, since a model's poor data fit may falsify one of these "auxiliary hypotheses" rather than the model. If the model is altered instead, it could lead to overfitting.

The authors note that modelling is under-determined, as multiple models can match the verification data. This leads to selection along “extraevidential considerations like symmetry… or metaphysical preferences.” Further, since components cannot usually be tested individually, multiple unseen errors may cancel out.

Moving onto validation, defined as a test of legitimacy or logical soundness, the authors write that it is independent of model results. Often, validation as a term is misused to mean verification. Examples are cited from the United States' Dept. of Energy and the International Atomic Energy Agency. Validation, they argue, is not the same as consistency between systems, nor can it be data-derived. A valid model can still be unreliable.


In discussing how numerical systems are 'verified' in practice, the authors state that solutions might be "bench-marked" against an analytical range. Extensions beyond this range, however, cannot be verified by definition, since the analytical result would be used.

Two-step calibration might be used, where only a portion of data is used to train the model and the rest to compare it with. Yet the authors claim that both steps are in fact calibration, since the second step may lead to parameter adjustment. This could lead to “fine-tuning,” especially in the competitive environment science is produced in with strong positive results preferentially published. See “Why most published research findings are false”, John P.A. Ioannidis. Yet it should be noted that techniques exist to circumnavigate this, e.g. using bootstrapping numerical re-sampling as per. "Diatoms as indicators of climatic change", Bigler & Hall 2002.

The authors discuss the difference between models having forced empirical adequacy, rather than verification, before moving to problems in extrapolation. Culprits include complex variables and magnified time-dependent errors.

Consequentially, the authors surmise that the veracity of models cannot be confirmed or demonstrated, only increased in probability, and that a statement of confirmation commits the fallacy of affirming the consequent. This is illustrated with a hypothetical claim that, if it rains, a person will be inside the house. The person is subsequently observed inside the house, leading to the fallacious deduction that it is raining. The fallacy arises here, as in modelling, because multiple models and theories explain the observations.

Finally, the authors discuss how 'verify' and 'validate' are used as affirmative terms, rather than terms of degree. They propose a neutral "precision and accuracy" evaluation language. Describing models as a heuristic, as a guide to further study but not susceptible to proof, the authors note that the modeller has a responsibility to “delineate the limits of the correspondence between the model and the natural world”. Since models can be affected by bias, their best use is in challenging hypotheses rather than verification.

FILED AWAY

I'm been thinking recently about the limits of scientific knowledge, both real and perceived. A post about Naomi Oreskes et al. 1994 paper on model verification will be up soon; in the meantime, here's a talk Ben Goldacre gave about the file-drawer effect in medical research.

Of course, not all file-drawer and other biases in science are ill-willed. Some may be the result of well-meaning but misplaced trust in a supervisor's hypothesis. More on this in the future.

As an aside, I'm not a fan of TED talks in general. They tend to elevate whizz-bang intellectualism rather than rational thought. They're an opportunity to profess one's knowledge and to associate with said people. Forget your skeptical hat at your peril.