DROUGHT AND WATER SUPPLY IN SOUTHERN ENGLAND

Following on from previous posts with a water management bent, let's look at the situation in England.

The popular image of England is drenched and sodden. As the saying goes, if you're on top of the Dover cliffs and can't see France it's going to rain; if you can, it's already raining.

However, this constant, uniform perception of rain over the UK is misleading. Parts of the UK, especially London and the south east (my home area), regularly experience drought. The most recent was in February 2012, leading the Environment Secretary to announce drought control measures. The Environment Agency maintains a drought management guide outlining publicity campaigns, hosepipe bans, restrictions on agricultural spray irrigation and further measures.

Droughts can be induced by low levels of rain, and/or by low levels of groundwater, itself caused by irregular or low precipitation. A drought in 2003 was driven by low rainfall (Marsh, The UK drought of 2003, Dec 2006). From 1961 to 1995, rainfall has increased in the winter but decreased in the summer (Osborn et al., Observed trends in the daily intensity of UK precipitation, Mar 2000). This has a large bearing on the UK's groundwater levels, since most groundwater recharge occurs during the winter when evaporation is lower. In summer, by contrast, higher temperatures lead to depleting groundwater levels. This suggests that groundwater may become more important to the UK's water supply.

Chalk reservoirs are found in East Anglia and Southern England. A 2011 model of the effects of climate change on UK groundwater in the reservoirs predicted no significant change, but with lots of sources of uncertainty (Jackson et al., Modelling the effects of climate change and its uncertainty on UK Chalk groundwater resources from an ensemble of global climate model projections, Mar 2011). Christopher Jackson was also the primary author of a recent review of ten studies, each looking at future climate change impacts on groundwater, which demonstrated the large uncertainties in groundwater level predictions. There were also "significant differences in current projections" between each study (Jackson et al., Changes in groundwater levels in the UK over the 21st century, 2013).

It would seem that there is good reason to investigate the extent to which groundwater should become a larger source of water for southern England.

MODELLING APPROACHES – STELLA & HYDROLOGICAL MODELS

STELLA is a modelling platform with a wide range of applications. In A formal approach to hydrological model conceptualization (1993), Jin Lee applies the use of STELLA to hydrological models. By dividing the process of modelling into 1) conceptualization and 2) programming and testing, Lee describes how hydrological model conceptualization can be integrated into STELLA.

Representing a model as a diagram of states and rates, or equivalently as stores and flows, Lee gives an example of such a diagram as reproduced below.

States are represented as rectangles; rates as opposing triangle pairs; and dependencies as arrows. For example, the amount of daily evaporation (upper right) depends on the evaporation rate, which is itself a function of climatic variables, soil conductivity and the amount of soil moisture. Representing a model conceptualization in such a way communicates which real-world phenomena are incorporated, which are idealized, and which are neglected. For example, the daily evaporation depends on the weather, which is neglected. Which phenomena are incorporated often depends on the desired complexity of the model, as well as the researcher’s background knowledge on the relative importance of different factors. Both of these were discussed in the Occam’s Razor post centred around Domingos (1999), and the post on model validity centred around Oreskes et al. (1994).

Lee points out how such a representation links easily to a mathematical formulation for each of the states and rates. Indeed, STELLA allows equations to be defined for each state and rate, incorporating all the relevant dependencies. For example, the equation for daily evaporation would take the evaporation rate as an input.

In Singh et al. (2010), the authors use a water balance model to assess the impact of climate change on the level of Loktak Lake in Northeast India. Water supply is important in the region, given the large population and dwindling groundwater reserves (Tiwari et al. (2009), also source of image with caption below). 

Dividing climate change projections into two groups, defined as A) 2 deg. C global warming from 7 different global climate models (GCMs) and B) 1 to 6 deg. C global warming from the HadCM3 GCM, the authors find that nearly all of the projections from group A, and all from group B, predict increasing lake levels. The authors point out that water management of the lake has already taken place, with two sub-catchments being isolated to reduce lake level rise. The results indicate that more work is needed, or communities and wetlands around the lake may be flooded.

The authors summarise their modelling approach in 3 stages: a calibrated model of the hydrological system dependent on climate data, which is then perturbed by altering the original climate data along the lines of the GCM projection, and then comparing the output with a baseline model defined by projecting current climatic conditions.

Finally, a stochastic hydrological model developed by Dincer et al. (1987) uses a conception similar to Lee’s, by representing a swamp as a series of stores along a line of flow. Each store, or cell, has an outflow dependent on the amount of water in the store. The water balance of each cell in turn depends on the sum of inflows minus outflows. By perturbing the inflow, the changes in swamp capacity and outflow can be measured, and the model was applied to changes to the Okavango swamp in Botswana. 

An interesting aspect of the model is its calibration procedure. The authors note that the long term independent variables (precipitation, inflow and evapotranspiration) are not stationary but probabilistic, meaning that different periods cannot be compared with each other. This leads to a systematic error whereby changes in flow distribution inside the swamp may be missed. The model is more reliably calibrated, however, over the discharge, water level and area variables. The authors conclude on the power of such network models, and the importance of groundwater to the Okavango Swamp.

BELATED BLOG INTRODUCTION

Who cares about models?

Michael Mann does. "Any conclusion about [global warming] causality required the use of climate models to estimate the relative contributions of the various factors, including human increases in greenhouse gas concentrations..." In the same interview about his book The Hockey Stick and the Climate Wars, he mentions models 8 times.

Deniers care too. The Heartland Institute's NIPCC website devotes whole sections discussing the limitations and misapplications of climate models (no link provided; I'm not advertising them).

Since so much depends on, and is misunderstood about, models in environmental science, from global and regional climate change; to hydrology, coastal management and energy infrastructure, this blog takes a critical look at the underpinnings of modelling and its applications to environmental phenomena.

My "research reading" posts look in-depth at some landmark, and some current, papers on the general process of modelling. This includes topics such as model calibration and model simplification. Within each post I also look at some papers applying relevant methods in an environmental science context.

My "modelling approaches" posts take the knowledge from the "research reading" papers and apply them to an environmental model; for example, a groundwater recharge model. I also mention other papers with supporting material or comparative approaches.

I'm hoping to synthesize these topics as the blog progresses. For example, I'm currently reading up on some models created in STELLA and comparing the different approaches taken, bearing in mind the critiques from the "research reading" papers.

Finally, anything without those headers is usually offbeat, humorously intended, provocative or Minecraft worshipping.

HUH? MULTIPLE COMPARISONS?

RESEARCH READING - OCCAM'S RAZOR

 
The Role of Occam’s Razor in Knowledge Discovery

Pedro Domingos, Data Mining and Knowledge Discovery 3, 409–425 (1999)



There are two interpretations of Occam's razor according to Domingos. They are that models should be:

1) Comprehensible
2) Accurate

In this paper, Domingos draws on a range of research to argue that applying Occam's razor on a choice of models can result in selecting a comprehensible model, but not necessarily an accurate one.

Observing that simplicity has "no satisfactory computable definition", Domingos states that a heuristic definition of Occam's razor (for example, reducing the number of parameters) leads to two formulations:

1st Razor: From two models with equal predictive error, choose the simpler one for simplicities' sake.

2nd Razor: From two models with equal calibration error, choose the simpler one for lower predictive error.

The 1st is generally true, whilst the 2nd is generally theoretically and empirically false.

Starting with some theoretical arguments for the 2nd razor, Domingos looks at the case for the Bayesian information criterion (amongst others). The BIC assumes that model parameters are distributed normally across candidate models. This gives the logarithmic probability of a certain model structure as equal to the likelihood of the structure given some calibration procedure, reduced by a complexity term dependent on the number of parameters. The weaknesses of this approach range from the long chain of assumptions required to use the BIC to the calculated probability being that of the structure, not the model. An example of a quadratic candidate is presented. As it has more model space than a linear model, the linear structure has a higher probability since the larger number of inaccurate quadratic candidates dilute the quadratic structure probability, even for cases where a quadratic structure is the more accurate.

Another theoretical argument for the 2nd razor comes from the idea of the minimum description length (MDL). The argument states that the best model uses the smallest number of bits to code for it and the data, given the model. Domingos argues that the belief that a trade-off between error and complexity results from Bayes' theorem is circular reasoning. This is because it is equivalent to stating that models with higher priors have shorter codes, but also that shorter coded models have higher priors. Sometimes, a complex model with the highest prior can be coded with shorter code compared to a simpler, lower prior model. In other words, after giving each model a prior, the models can be recoded such that those with the highest priors have the shortest code. A short code length does not imply better predictions or a more comprehensible model.

Moving on to theoretical arguments against the 2nd razor, the "No free lunch" mathematical theorem is briefly mentioned. It results from the underdeterminancy in model selection (arising from the fact that model fitting is an open system, with several candidate models always able to match calibration data - see the Oreskes post). The generalization of a model is discussed as resulting from it's Vapnik–Chervonenkis dimension, not the number of parameters. It is possible to have a model with an infinite VC dimension yet just one parameter.

Overfitting is often erroneously thought to be due to complex models. However, it is actually due to multiple comparisons. The probability of a model fitting the calibration data purely through chance rises if more candidates are selected from. Models with many parameters tightly constrained may thus be less susceptible to overfitting than broader, simpler models.

A final set of problems discussed by Domingos involve projections of systematic and random error. Heuristics that minimise complexity often assume rising complexity produces a faster increase in systematic than random error. This is demonstrably not the case for some systems.

Turning his attention to empirical tests of the 2nd razor, Domingos notes that corrections for multiple testing often produce better models than those based on MDL. Further, although the accuracy gain of complex versus simple models may be small it is not necessarily negligible. After providing examples of complex decision-tree models, with the right constraints, with more accuracy than simple ones, some physics examples are presented. For example, both the Copernican and Ptolemaic orbital theories had the same predictive error, so preferring the former was selection using the 1st, not the 2nd, razor (something which was of special interest to me given my background! A theme alluded to was the idea of simplicity resulting from the use of the model, not the final result. A Kuhnian paradigm shift could occur if an unwieldy model with too many patches was replaced by a leaner one. Perhaps the 1st razor is more central to scientific understanding than otherwise recognised?).

Geocentric model. Ptolemaic systems introduced epicycles to solar system bodies, reconciling it with observations. Image from redorbit.



Outright tests of simplicity versus accuracy in e.g. decision-tree modelling generally show that complex models are more accurate than the simple. Complex MDL systems with redundancy, or multi-model ensembles, are more accurate. Domingos states, from this evidence, that the 2nd razor is "typically false". The issue of errors in computation is unmentioned. Domingos does mention cognitive research having implications for comprehension.

Returning to the 1st razor, it is noted that since simplicity is not the same as comprehensibility, it can be rephrased in terms of domain-dependent comprehensibility.

The 2nd razor is then finally discussed as being trivially true after model coding, but of no help in calibration or selection. It is better to prevent overfitting by using domain knowledge, with the added bonus of increased comprehensibility to those with such knowledge. Domingos mentions ecosystem modelling using LAGRAMGE, a 'declerative bias' equation discovery program. The authors of the relevant paper found a model for phytoplankton growth in Lake Glumsoe with appropriate terms and accurate predictions. Pre-existing knowledge of algal growth was used in its construction. See Todorovski and Dzeroski, Declarative bias in equation discovery. Proceedings of the Fourteenth International Conference on Machine Learning, Nashville, TN: Morgan Kaufmann, pp. 376–384 (1997).

Complex ensemble models can be made comprehensible by choosing representative models, with lower, but better, accuracy than similarly structured single models. Explaining a model's results after calculation is also often more comprehensible than coding a fully comprehensible model.

In conclusion, it is recommended to use domain knowledge and the 1st razor when modelling, and to treat model accuracy separately from comprehensibility.


*My musings in italics

MODELLING APPROACHES - SOIL-MOISTURE BALANCE MODELLING

During yesterday's lecture on environmental modelling, Pf. Richard Taylor demonstrated an accounting model of groundwater recharge. His talk was very informative and entertaining, building up from the basics of groundwater systems to reach some profound conclusions, such as the tendency for global climate models (GCM) to overlook subsurface water flow and where that philosophy might originate. 

I'd like to expand on many of the modelling themes that he touched on. The accounting model was based on a 55yr dataset from Uganda, the details of which can be found here: Taylor, R.G., Todd, M., Kongola, L., Nahozya, E., Maurice, L., Sanga, H. and MacDonald, A., Evidence of the dependence of groundwater resources on extreme rainfall in East Africa. Nature Climate Change, Vol. 3, pp.374-378 (2013).

The accounting model looks at the water level of the saturated zone. A groundwater recharge event raises the water table. From the UK Groundwater Forum.


Taylor first  mentioned the WaterGAP model, as used in Portmann, F. et al., Impact of climate change on renewable groundwater resources: assessing the benefits of avoided greenhouse gas emissions using selected CMIP5 climate projections. Environmental Research Letters, Vol.8 (2013). Broadly speaking, the model was used by Portmann et al. to project future groundwater recharge (GWR). The relevant point here is that the GWR estimates were not calibrated.

In my post on the Oreskes paper it was seen that it is often difficult to calibrate components of complex models, due to small inputs generalized into large model inputs, feedback with other components, and selective or non-existent data on model inputs. In this case, no global observational data of GWR exists, making it impossible to calibrate WaterGAP's predictions. 

There is another serious shortcoming in the WaterGAP's GWR forecast - it determines a priori that greater precipitation leads to greater surface runoff, so it cannot be used to test this hypothesis. This arises because the model uses a simple cutoff threshold, whereby excess precipitation over this threshold goes straight to runoff. This precludes the possibility of alternate phenomena such as increased GWR.

As the Taylor et al. paper shows, the evidence from Uganda presents a different view. GWR only significantly occurs during extreme precipitation events. For example, the Makutapora record shows the 4-month 1997-98 ENSO contributed 25% of the total GWR in the 55 year record. Drawing two conclusions about modelling in the talk, Taylor (paraphrased) stated that models should "utilise historical data to [disprove] hypotheses" and "properly structured GWR models give better predictions", echoing the general sentiment from Oreskes et al.

Recalling an unfortunately futile effort by some hydrologists for representation in the IPCC's 5th assessment report, Taylor noted that GCMs tended to use a landsurface with no subsurface component. This reduced familiarity may lead to GCM modellers being less confident linking the subsurface component with climate change, misrepresenting GWR changes. 

Returning to the Taylor et al. model, Taylor outlined how a less complex accounting model based on local data was developed. It was built "from the ground up", using the empirical data. The model's simplicity came in part from the absence of physical knowledge of the flow of groundwater in that part of Uganda. The observed GWR was much more rapid than the most appropriate physical law, Richard's equation, would predict. It thus did not model process, but the water content in different parts of the ground over time (hence the "accounting" moniker).

In outlaying how the model extrapolated from the data into 2070, Taylor noted that most GCMs were trained by looking at past daily rainfall and distributing it in the most statistically appropriate way in the future (a "delta approach"). However, since a warmer world has less frequent and more intense precipitation (e.g. Pall et al., Testing the Clausius–Clapeyron constraint on changes in extreme precipitation under CO₂ warming. Climate Dynamics, Volume 28, Issue 4, pp 351-363 (March 2007)), the delta approach underestimates GWR from intense precipitation in a warmer world. This is especially true if focus recharge, such as from floods, become more common as a result of higher precipitation - another aspect that GCMs neglect.

Taylor made two other points that resonated with my physics background. In justifying Excel for modelling, he pointed out that "few colleagues have Matlab licenses or R experience, whereas everyone has Excel" and "simple models with calibrated variables are better than complex models with broad, uncalibrated components". A models' end-users, the modelling community, as well as Occam's Razor, will be considered further on this blog in the future.

DISCUSS

Tom Murphy's insightful post on the energy trap.

RESEARCH READING

Verification, Validation, and Confirmation of Numerical Models in the Earth Sciences

Naomi Oreskes, Kristin Shrader-Frechette and Kenneth Belitz (1994, Science)

*My musings in italics

Verification, or establishing the truth, of numerical models is the first target of this paper. Beginning with the observation that public policy places high demands on relevant models, the authors begin with a description of open and closed systems. They then proceed to demonstrate that models, as open systems, cannot be verified.

Open systems have multiple influences and extenuating circumstances, leading to the possibility of a true claim failing verification - a false negative. However, a closed system's claims are derivable and thus verifiable mathematically (putting aside the issue of imperfect human & mechanical computation, not discussed here). The authors discuss some contributors to the incompleteness of open systems: imprecise inputs; continuum mechanics and a loss of fine structure; uncertain relations between small measured inputs and large generalised model inputs; and a priori assumptions embedded in the 'real world' dataset used for verification.

The last contributor is especially poignant, since a model's poor data fit may falsify one of these "auxiliary hypotheses" rather than the model. If the model is altered instead, it could lead to overfitting.

The authors note that modelling is under-determined, as multiple models can match the verification data. This leads to selection along “extraevidential considerations like symmetry… or metaphysical preferences.” Further, since components cannot usually be tested individually, multiple unseen errors may cancel out.

Moving onto validation, defined as a test of legitimacy or logical soundness, the authors write that it is independent of model results. Often, validation as a term is misused to mean verification. Examples are cited from the United States' Dept. of Energy and the International Atomic Energy Agency. Validation, they argue, is not the same as consistency between systems, nor can it be data-derived. A valid model can still be unreliable.


In discussing how numerical systems are 'verified' in practice, the authors state that solutions might be "bench-marked" against an analytical range. Extensions beyond this range, however, cannot be verified by definition, since the analytical result would be used.

Two-step calibration might be used, where only a portion of data is used to train the model and the rest to compare it with. Yet the authors claim that both steps are in fact calibration, since the second step may lead to parameter adjustment. This could lead to “fine-tuning,” especially in the competitive environment science is produced in with strong positive results preferentially published. See “Why most published research findings are false”, John P.A. Ioannidis. Yet it should be noted that techniques exist to circumnavigate this, e.g. using bootstrapping numerical re-sampling as per. "Diatoms as indicators of climatic change", Bigler & Hall 2002.

The authors discuss the difference between models having forced empirical adequacy, rather than verification, before moving to problems in extrapolation. Culprits include complex variables and magnified time-dependent errors.

Consequentially, the authors surmise that the veracity of models cannot be confirmed or demonstrated, only increased in probability, and that a statement of confirmation commits the fallacy of affirming the consequent. This is illustrated with a hypothetical claim that, if it rains, a person will be inside the house. The person is subsequently observed inside the house, leading to the fallacious deduction that it is raining. The fallacy arises here, as in modelling, because multiple models and theories explain the observations.

Finally, the authors discuss how 'verify' and 'validate' are used as affirmative terms, rather than terms of degree. They propose a neutral "precision and accuracy" evaluation language. Describing models as a heuristic, as a guide to further study but not susceptible to proof, the authors note that the modeller has a responsibility to “delineate the limits of the correspondence between the model and the natural world”. Since models can be affected by bias, their best use is in challenging hypotheses rather than verification.