There is an extensive literature evaluating DSTs and systems. Examples are provided here to draw on previous experiences of the importance of DSTs and to elicit criteria for the evaluation made in this report.

Maynard et al. (2001) evaluated an extensive number of decision support systems used for different purposes. They attempted to develop generic critera for evaluation of DSTs, emphasising that different stakeholders and decison makers apply different sets of criteria. Four broad types of criteria were identified, all having effects on the costs and benefits of using DSTs:

• Effectiveness - refers to the level of the fulfillment of the goals of the DST. Flexibility to adjust to end-users requirement is part of this criterion, being important to save time using DSTs.
• Efficiency - refers to the degree of performance of the DST; one example of a sub-criterion that falls under this domain is whether the use of the DST reduces costs or increases the profitability for those who use the tool.
• Satisfaction - refers to end-users perception of the tool.
• Use - refers to the ease of use and the extent of DST use among relevant end-users.

Ease of use, which is important for the perceived costs of using the DSTs, was also of concern in an evaluation of DSTs by Inman et al. (2011). Based on a literature review, these authors defined “ease of use” as the ability of DSTs to present information to the user in a clear and familiar way, with rapid comprehension. They also identified that difficulties using a DST might negatively affect the overall satisfaction with the tool, referring to Sanders and Courtney (1985). A study by Srinivasan (1985), referenced in Inman et al. (2011), showed that the greater the time spent using a DST the lower the perceived effectiveness became. Inman et al. (2011) suggested that the actual time used for operating the system can be used as an indicator of the effectiveness as it is related to the ease of use of the DST. Technical suitability and transparency can also be indicators for effectiveness.

The summary from an EIP-AGRI workshop on “Tools for Environmental Farm Performance” (EIP-AGRI 2017) also points out that ease of use is important. It was concluded that reasons for poor uptake of DSTs among farmers are:

• the tool is not found to be useful by the farmer,
• the tool might be difficult to understand,
• the DST may require the farmer to spend a lot of time setting it up or learning how to use it,
• the costs outweigh its perceived benefits.

These findings are in line with our findings in »National and international use of decision support tools and barriers to their uptake.

The summary of EIP AGRI also mentioned lack of trust, for example the concern that using the DST may lead to new regulations being imposed on the farmers. Another conclusion from the wokshop was that the DST should be affordable in the context in which it is expected to be used. This means that for marginal producers the costs of using the DST should be very low, while “high-value agro-industrial production systems might be willing and also able to invest more, in order to gain more” (EIP AGRI 2017, page 18).

Related to the efficiency criterion, many studies have revealed that economic outcome and minimization of risks are important. In a study of Danish farmers, Pedersen et al. (2011) found that 92% of the farmers who responded to the survey preferred it when advice improved economic outcome and reduced risks (related to pesticide application to reduce diseases and weed). Farmers’ considerations about the economic outcome are a trade-off between the cost of pesticides or other treatment, and the marginal benefit from this use. The advice provided by DSTs and/or advisors should therefore be able to provide information about this trade-off. Rose et al. (2016) conducted a survey among farmers in the UK and pointed out the importance of the cost of using a DST and its influence on uptake by end-users. They concluded that DSTs that are free of costs or provided by a grant are more likely to be used, and they also highlighted that usability and relevance are important criteria for the success.

In »Survey and review of existing decision support tools we refer to a farmer survey made by Defra (2015) where economic gain from using tools was found to be important. Drawing on information from in-depth interviews and focus groups Defra (2015) found that, amongst other things, farmers wanted tools to be more user friendly and more flexible (ease of use was important) and also that the potential economic gain should be explicitly demonstrated (efficiency matters).

Axelsen et al. (2012) evaluated the Danish Plant Protection Online, and they also concluded that the large amount of time required and level of complexity have had led to a low uptake of this tool among farmers.

For catchment and national assessments of costs and benefits other methods and tools exists.

Ward (2007) reviewed studies on the use of economic concepts and tools for the analysis of management of water resources, and summarized economic analyses to support policy decisions. It was found that there are many methods and approaches, including valuation of water, valuation of water quality management, optimization models and integrated model approaches. Cost-benefit analysis is one approach that attempt to give advice on both benefits and costs, Similarly, there are also a number of tools which have been developed to advise on both how the costs can be minimized and the benefits maximized. The DSTs for advising how to minimize costs include cost-effectivess approaches. Balana et al. (2011) have reviewed a large number of assessments that have been made to estimate cost-effective combinations of measures to reduce nutrient losses to the aquatic environment. Balana et al. (2011) concluded that many studies performed before 2011 were based on models of ‘representative’ farms without capturing the variability among real-world farms. In addition, they concluded that many studies were based on a few examples of effects and did not include uncertainties in cost and effectiveness estimates. The review indicated that examples of DSTs that capture spatial modelling beyond farm level and which can be used to assess the effects of uncertainty and heterogeneity on the cost-effectiveness results should be favored.

For valuation of benefits for society, different approaches can be used for decision support. A number of authors have developed criteria for, and recommendations for, how valuation of benefits of water quality improvements as well as other environmental improvements can be measured and used for policy advice. Meta-analysis is such an approach, also called the “study of studies” (Bergstrom and Taylor, 2006). Meta analysis and regression analysis represent attempts to statistically measure systematic relationships between data from valuation studies, as well as data for human population, environmental characteristics of the place where the regression results should be used. Following Bergstrom and Taylor (2006), meta-analysis is a useful tool for advising on the benefits of water quality improvements that build on extensive experiences from many studies. The strength of meta regressions as a DST is the ability of this approach to combine and summarize large amounts of information from previous studies, and build on these experiences for policy advice.

The weakness of meta analysis might be that spatial differences are neglected when using data from a number of studies to create a generic function to measure the value of an improvement. The spatial differences between locations can be huge, and several studies highlights this by using ecosystem services assessment tools to address how ecosystems services and goods vary spatially. This type of information can be very important in order to enable social planners and decision makers to target the efforts to where the benefits are the largest and outweigh the costs.

Bateman et al. (2013) demonstrated the development of such a spatial assessment framework applied in the UK, using land-use as example to provide information on the benefits of ecosystem services from land use- and climate changes, on water quality services, biodiversity and other services. This type of assessment method has also been applied as DST in Denmark, and the DST’s include the creation of scenarios for how ecosystem services such as recreation, biodiversity, water quality regulation and climate regulation, are influenced by set aside of land at different locations. Spatial maps were used in both Bateman et al. (2013) and Termansen (2018) to illustrate the distribution of the value stemming from the ecosystem services and their spatial, locations.

Note: For full references to papers quoted in this article see

» References