31 October 2019

The dangers of a “partial” impact analysis: the example of a study on the impact of a 100% conversion to organic farming in England and Wales

Over the last few days, a relatively high publicity was given by the media to the publication by a team of researchers, mostly from the School of Water, Energy & Environment, Cranfield University, UK, to the results of a study of the impact of a 100% conversion of agriculture into organic production in England and Wales.

The critical review of this study illustrates in many respects how difficult it is to analyse all dimensions of the food transition and how limiting the scope of research impairs the message sent to decision-makers and the public at large.

One justification put forward by the team of researchers for undertaking the study is the mixed results obtained in earlier studies on the impact of replacing conventional agriculture by organic production, despite claims that organic farming should produce less greenhouse gases (GHG) per tonne of crop product on the ground of a reduced use of farm inputs and an increased sequestration of carbon in the soil. However, some studies showed higher while others showed lower GHG emissions by tonne of produce after conversion [see two examples here and here among the many studies conducted on the subject].

One major originality of this study is the level at which it is conducted, not at crop plot or farm level, but at the level of an administrative area, in this case the unit constituted by England and Wales.

Main study findings

Based on the use of the Agri-LCA model [read] and of the OLUM linear programming model [read] the research team “predicted” (!) that after the 100% conversion to organic farming:

1. •Total food production in England and Wales expressed as metabolisable energy and protein produced would fall by around 40% compared to the conventional farming baseline, because of reduced yield and the need to introduce within crop rotation fertility-building grass leys with nitrogen-fixing legumes;
   

2. •Total GHG emissions directly associated to organic crops would be lower (less 20% for crops, less 4% for livestock and less 6% overall);
   

3. •At unit of production level, results would be diverse: for some crops or livestock products, GHG emissions would be reduced, for others they would increase;
   

4. •There would be higher carbon fixation in the soil but only during “the first decade or two following conversion, because any given soil has a finite capacity to accumulate [carbon]”;
   

5. •“The land area needed to make up for shortfalls in domestic production would be nearly five times the current overseas land area used for food for England and Wales” thus inducing higher GHG emissions overseas and bringing total land use required to feed the population to 1.5 times the land required under conventional agriculture.
   

Critical analysis of the study : a poorly defined question and an inappropriate method of analysis

The study therefore clearly provides apparently strong arguments against conversion into organic farming. These findings were presented spectacularly in the media through titles like “UK move to organic farming would 'increase emissions’” (FarmingUK) or “A totally organic agriculture would increase CO2 emissions” [Express Business, in French], titles sufficient to influence negatively public opinion towards the perspective of developing organic agriculture, particularly as these conclusions are presented as an outcome of a rigorous scientific study conducted by known researchers.

In the discussion of the results, the authors refuted criticisms on the lack of consideration of the impact on biodiversity on the ground that agricultural land expansion required under the organic scenario “might” impact more biodiversity overall than what could be gained locally.

In the same way, they discard the possibility that with more research on organic farming, yields could be increased, claiming that “these improvements are probably marginal, given the fundamental requirement for more leys in rotations under organic management”. The possibility that this “fundamental requirement” could be offset by improved crop management (e.g. introduction of new crop rotations, of new complementary crop associations, and other means of improved fertility management.) is not even considered, authors deliberately placing themselves in a static environment where neither technology, nor the population’s diet may change.

In response to criticisms, one of the team members justifies the limits of the study by explaining that: “Whether a different national diet could be provided by the same land area under all organic production is a different study. This was aimed at understanding limits to production. The study was based on rigorous modelling that had its foundations in establishing the biophysical limits of crop production without manufactured nitrogen” [read]. If one believed this clumsy claim, one can be surprised that the study conclusion are formulated in terms of GHG emissions and expansion of agricultural area, for example.

It appears clearly from this that there is a problem with the definition of the question the study was supposed to answer and with the inconsistency between this question and the findings produced.

The research question, the method used by the team of researchers and the results obtained illustrate consequences of compartmentalisation of our thinking [read]. While wanting to make a more comprehensive analysis (moving from a crop plot level to a country level), the team omitted to adapt the analytical method used to this new level and left out important aspects, in particular, the socio-economic dimension, of the change it was set to analyse. It assumed that comparing conventional and organic cultivation of say wheat plot, is the same thing as comparing England and Wales under conventional and organic farming. And it is wrong.

By introducing the “country” level dimension, the research team should have considered at least three additional and crucial dimensions (more could probably be added to this list): the policy dimension, the time dimension and the socio-economic dimension.

If the study is to produce results useful for decision making, then it has to be set in conditions that reflect reality. Expanding from crop plot level to country level means that it is not possible to simulate an experiment in which just one factor would be changing (in this case replacing the conventional production technique by the organic production technique) while “all other things remaining equal”, as in reality, other things never remain equal…

First, for the change whose impact is analysed to occur, certain conditions are required (at this level change cannot imposed by an experimenter as is the case at plot level).

Second, once change actually happens, it will trigger a chain of effects that may spread to several parts (and dimensions) of the food system, similar to when a drop hits a water surface and creates a ripple effect throughout the water body. These effects will need to be analysed and measured. Some effects found will be favourable compared to a desired outcome, while others will be negative, meaning that the desired outcome (or direction of change) should be defined beforehand. Also some effects may be expected, while others may be unexpected, and efforts should be made to try and identify them in advance.

The policy dimension refers to the fact that conversion will not take place automatically but will need to be supported by an appropriate policy mix that will promote organic farming.

The time dimension means, among other things, that the conversion does not take place overnight, that it requires time and that during this time, technologies will evolve as a result of more research being conducted on organic farming - encouraged by a supportive policy environment. This enhanced research effort on organic agriculture will certainly aim at increasing its productivity and at finding new ways of managing soil fertility will likely be found, making that the use of leys blocking part of the land proposed in the study just one among a variety of options available.

The socio-economic dimension means that, over time, consumption will change, as, while conversion to organic farming occurs, relative prices of food will change and awareness of consumers will evolve, influenced by communication campaigns. As consequence, there will be an evolution in the composition of diet. And when diet evolves, this will feed back on what is being grown, so the crop mix and the level of livestock production will also change [read] - it is sad to see that authors discard the possibility of a reduction of livestock products on the ground that “global trends are towards greater per capita and total meat consumption”, as if this past trend were cast in stone, when reduction in meat consumption is already being observed in several (rich) countries. Moreover, with appropriate policies and information, with time, food waste can be reduced. Finally, both the amount and the mix of food required will change considerably. 

To conclude, the research team made a commendable effort to expand the scope of the analysis of conversion to organic farming from a crop level to a country level. However, in doing this, they overlooked several dimensions that this change of level of analysis added to what they were analysing. Not considering these additional aspects makes that the validity of the conclusion drawn from the research can be challenged.

Unfortunately again, in the media, the presentation of the summary of the findings of the study are generally not qualified by the indispensable caveats (luckily there have however been some exceptions!). As a result, the message conveyed to citizens is that conversion to organic farming is not a solution as it would reduce production and increases GHG emissions (while the team themselves state in their paper that it is not the “single optimal approach to achieving environmentally sustainable food production”, a much more nuanced statement) [read in French].

This a clear example of how compartmentalisation of our thinking (and working) that makes us analyse only some aspects of the problem before us. The risk then is that the results of such a partial analysis can be taken as scientifically valid and that they will influence policy makers when deciding policies to be applied not just to a part of the food system, but to the food system as a whole and while being unaware of the multiple important consequences that have not been seriously considered by the researchers.

Team composition may be one of the reasons of the limited scope chosen for the study (one agroecology and agricultural systems modelling specialist, one soil scientist and modeller, one environment specialist and modeller and one agricultural economist). Another reason may be the central role played by the models used, whose architecture strongly determines the kind of results that can be obtained.

Finally, what does this study tell us, beyond its findings?

From a methodological point of view, it confirms the difficulty in trying to take into account the complexity of the food transition issue and the need to develop new better adapted tools making it possible to deal with the issue in a more holistic way. It also points to the need for mobilising multidisciplinary teams comprising varied skills from many fields (economics, sociology, environmental science, institutions, in particular).

It also shows us that measuring the impact of changes envisaged cannot be done simply by using one indicator (e.g. the variation of the amount of GHG emitted) but instead by using a set of indicators designed to account for all the “important” dimensions of the reality on which the impact should be measured. In the case of sustainable and climate-friendly agriculture, many aspects must be considered [read] for which valid indicators should be selected. It is by comparing results obtained for various alternative options that it will be possible to determine which one is preferable. For doing this, there is an array of more or less sophisticated multicriteria decision-making methods, mathematical or graphic, that can facilitate negotiation and decision making.

Evidently, some will say that this is far too complex a way of analysing reality. The fact is, however, that the world is complex and the more one tries to simplify it, the more risk there is to be ill informed and take flawed decisions whose consequences may include some rather unpleasant surprises...

—————————————

To know more :

1. •University of Reading website: 100% organic farming in the UK could significantly increase agriculture emissions, 2019.
   

2. •Smith, L.G., et al., The greenhouse gas impacts of converting food production in England and Wales to organic methods, Nature Communicationsvolume10, Article number: 4641, 2019.
   

3. •Harvey, F., Switching to organic farming could cut greenhouse gas emissions, study shows, The Guardian, 2018.
   

Selection of past articles on hungerexplained.org related to the topic:

1. •Obstacles to transition - Why is it so difficult to make our food system more sustainable and climate-friendly? 2019.
   

2. •Policies for a transition towards more sustainable and climate friendly food systems, 2018.
   

3. •Is there a new paradigm of agricultural research? 2017.
   

And many other articles available under our category “Research”.

1. ✴ Back to News