Outcome Evaluation Designs

The following sections provide more technical detail for outcome evaluation designs. Many consider the experimental approach using randomised control trials as the ‘gold standard’ for evaluation evidence. In many TDM projects this may not be practical, and quasi-experimental or non-experimental methods will be sufficient.

Experimental approaches

You decide who is in each of the comparison groups in an experiment, usually by randomly allocating people into each group. The group that receives or participates in your project is the intervention or treatment group, and there may be more than one intervention or treatment group that receive different levels or types of project activities.

If a group does not participate in the project, it is a control group that provides the counterfactual case; giving us a sense of what might have happened to the treatment group without the intervention.

An experiment where the potential participants in the project are randomly allocated into each of these groups is a randomized control trial (RCT). If all of potential participants share common characteristics that might affect the success of the project, random allocation into groups controls for these factors so that they will be equally likely to be present in each group. In other words, because the groups are the same in terms of all characteristics that might affect the project, except for the fact that one group does get the project and the other does not, then we can say that differences in outcomes are due to the project.

Experimental approaches should be considered in the following circumstances:

• Scope of the project: The project has a small number of well-defined final outcomes and it can be delivered in isolation from other activities that might also affect these outcomes.
• The Intervention context: it must not be possible for the groups to affect each other. For example, members of the project group should not be able to communicate to the control group information that might alter the latter’s behaviour. This will be tricky if members of the treatment and control group live in the same geographic area. If the project is rolled out in different sites, there should not be major differences in the way it is implemented across these sites.
• Stability of the project: the project should not change in any significant way while the experiment is running, otherwise it will be hard to know exactly what form of the project affected the outcomes. Similarly, it must be possible to limit entry and exit from the project by participants so that the members of the treatment group remain comparable to the control group.
• It must be ethically appropriate: Even if an experimental design is technical possible, it might not be appropriate to use one. Some ethical questions you need to think about include:
• Will members of the control or experimental groups be exposed to known risks or known harmful outcomes?
• Will the experiment mean members of the control group are denied services which are known to be beneficial on the basis of existing evidence?
• Will members of the control group be denied access to services to which they have an historical entitlement?

For TDM projects, an experimental design can be useful to assess the impact of information to customers, as illustrated in Box 1. 

Box 1: Randomisation in the Walk in to Work Out project

While many people regard the classic RCT as the ‘gold standard’ for evaluation designs, RCTs are not always desirable or feasible. As an evaluator you may not always have the level of control over project implementation to apply an experimental design, and there may be ethical issues with varying treatment options among the groups. In other instances, RCTs may simply be unnecessary to answer the specific KEQs that have to be addressed.

Quasi-experimental design 

This design is also based on the principle of comparing outcomes from a defined group who receives the intervention with a group not receiving the intervention, however, you cannot fully control the composition of your comparison groups. Groups are constructed once data are gathered and can be divided up according to characteristics of the people (or things) from which data have been taken (hence the term ‘statistical control’).

For example, one local government area (LGA) may receive a TDM project that includes an app providing real-time information about travel-choices, whereas another LGA gets the same project except that the app is not provided to them. You might observe later differences in behaviour, but because you did not randomly allocate people to live in each of the LGAs, these differences may be due to factors such as the age distribution of the populations, proximity workplaces, other transport options, etc. To make a more meaningful comparison you will need to gather data about all these factors and ‘control’ for them using the appropriate statistical modelling technique, examples of which are in Table 1.

Table 1: Statistical techniques for quasi-experimental designs

Given the complexity and analytical skills required to understand the choices in Table 1, an alternative to prescribing a preferred analysis as part of your evaluation planning, you can specify how the decision-making about statistical model will take place, and what kind of checks will be put in place to ensure that the right approach is taken. For example, a technical advisory team may be set up to work through the detailed issues involved in choosing appropriate statistical analysis.

Non-experimental/theory-based approaches 

Experimental and quasi-experimental approaches rely on access to a comparison group that provides a counterfactual; an ‘image’ of what the group getting the project would have looked like had they not received the project.

In many instances, you might not be able to get data on a comparison group. All you have is information from participants in your project. In these situations, you can draw on several non-experimental approaches to make conclusions about effectiveness. One of the strengths of these approaches, over experimental and quasi-experimental approaches, is that they can generate evidence about process issues, so that process and outcome evaluations can be conducted simultaneously. 

These approaches assess effectiveness by either or both of the following methods: 

• Checking whether the results support causal attribution. An example of such an approach is actor attribution, whereby participants in the project are asked to explain results. Similarly, expert comparison compares actual outcomes to expert predictions. 
• Ruling out alternative explanations. An example is general elimination methodology which Identifies alternative explanations for your outcomes and then systematically investigates them to see if they can be ruled out. 

Table 2 provides a menu of options, most of which involve a detailed case-study of the project. If you find that some of these are relevant to evaluating your TDM project, you should explore these further (as well as experimental and quasi-experimental designs) by looking at the UK Government’s Magenta Book Annex A.

Table 2: Non-experimental methods

A theory-based approach is particularly suitable when the evaluation: 

• Seeks to test the theory behind an intervention. For instance, the success of an intervention to promote cycling might be based on the number and kinds of organisations that need to be involved, the level of investment required, the number and type of engineering and education measures to be implemented, the number of people and locations that need to be involved in order to achieve the anticipated outcome and impact targets. You could evaluate this by collecting data on each element, and then the observed outcomes are explained with (and attributed to) the extent to which inputs / activities and outputs were achieved as planned.  
• Investigates a complex project that has multiple interacting outcomes, and/or consist of different components or is implemented in different locations.  
• Needs to consider how contextual factors (e.g. people, organisations or socio-economic circumstances) influence the success of an project.
• Seeks to generate new knowledge that can be used to inform future TDM projects.
• Is interested in identifying both anticipated and unintended outcomes and how they have been achieved.   

When to take measurements 

A key part of outcome evaluation designs is the decision when to take measures for your indicators. At the very least some kind of before-and-after plan is necessary to make some assessment of the impact of your project. At either end of the process you might need to consider taking measures at a number of points: 

• Baselines measures provide a reference point from which changes can be gauged. The simplest approach is to take measures just before the TDM project is implemented. However, this may not always be sufficient to provide a counterpoint to post-project measures. Many TDM-relevant data series, such as customer journey numbers, traffic volumes, etc. have both random and seasonal ups and downs. Thus, a single baseline measure may capture the relevant series on one of these ups or downs and produce misleading impressions when compared to post-project measures. To mitigate against this possibility, baseline measures should be collected over time before implementation so that any inherent random fluctuations can be considered in the data analysis. Similarly, seasonal variations can be assessed and post-project measures can be taken at corresponding points in the seasonal cycle.
• During the project, measures can be taken to observe how enabling outcomes are emerging and whether they correspond to the pathway laid out in your logic model. 
• After the project more than one measure may need to be taken to allow for outcomes to fully mature.