Personalised marketing – avoiding the evaluation pitfalls

The purpose of this work was to provide guidance to local authorities and others who may wish to apply a personalised marketing programme in a given area and to measure the impact of such an intervention. Our experience, along with others such as Stopher (2003) and Richardson (2003), identified several evaluation pitfalls that could lead to overstating the results of the intervention.

'Personalised Marketing' is used to describe a programme aimed at changing people's travel behaviour by a combination of education, persuasion and provision of personalised information to either individual households or individual people. One of the best-known personalised marketing programmes in Australasia and Europe is "IndiMark" or "individualised marketing", also branded "TravelSmart".

Various personalised marketing demonstration programmes have claimed substantial success in decreasing car use and increasing trips by alternative modes, thereby convincing some local authorities that such programmes may be the "panacea" to congestion problems in urban areas. Our recent involvement in helping to plan the evaluation of a personalised marketing trial in Birkenhead (Auckland, New Zealand) caused us to review the international experience with various trials and their evaluation.

Our investigation specifically considered:

  • "pre-selection" of the area and participants / households for a personalised marketing initiative
  • pitfalls in the evaluation of the impacts of such programmes.

The evaluation ("after") and elicitation surveys in Birkenhead highlighted the inadequacies of the current public transport system, providing evidence to support the widely-used pre-selection criteria of a "good quality" public transport service being in place prior to the initiation of a personalised marketing programme.

Analysis of the "before" survey data revealed characteristics distinguishing "receptive" from "non-receptive" individuals and households. Not only did the analysis reveal significant differences in personal characteristics and attitudes towards "environmentally friendly modes", but also we found indications of pre-existing differences in mode use, which could confound potential evaluations of mode change after the intervention.

Furthermore, great care must be taken with respect to the sample used for the evaluation. The statistical power to detect significant differences between before- and after-measurements is determined not only by sample size but also by the variability of behaviour. If people vary greatly in the number of trips driven and/or distance driven on a day-to-day or week-to-week basis (even in the absence of an intervention), then larger sample sizes (and/or longer data collection periods than the usual one-day trip diary) are needed. We used the 1997/98 New Zealand Travel Survey to estimate day-to-day variability in transport behaviour (both distance and trips, for several different modes). Results suggest that the sample sizes required are distinctly larger than seen in some recent research locally. In particular, doubts are raised about whether or not the survey results from the South Perth IndiMark programmes can provide any statistically robust evidence of impact or, following on from this, if the high benefit-cost ratios claimed can be substantiated.

Report

The paper was presented at the Australasian Transport Research Forum 2003 in Wellington, New Zealand. A revised version of the paper was published in Transportation Engineering in Australia (2004). The paper is available as a pdf document:

Personalised marketing working paper (pdf)