What do effect sizes mean in real terms? Benchmarking effect sizes in behavioural research

Details

A fully-funded studentship is available to work alongside the team working on the BR-UK project (https://www.linkedin.com/company/behavioural-research-uk/) at the University of Sheffield. The aim of the project is to use a technique called ‘benchmarking’ to identify how effect sizes of varying magnitudes correspond to real world behaviour and outcomes. The studentship will be supervised by Dr Harriet Baird and Professor Thomas Webb and supported by Professor Paschal Sheeran (The University of North Carolina at Chapel Hill) and members of the BR-UK team, including Professor David Lagnado (University College London) who co-leads a workstream on methods and evidence synthesis.

BACKGROUND: Behavioural researchers typically evaluate the effect of interventions (e.g., a campaign to promote physical activity or encourage recycling) by evaluating whether behaviour or outcomes following the intervention are statistically different from before the intervention or among a comparison condition. Because statistical significance (or not) depends on the size of the sample and provides little information about the magnitude of the effect, researchers have started to move toward reporting the size of the difference using metrics like Cohen’s d, eta2, Pearson’s r, which is often interpreted qualitatively as, for example, small, medium, or large (Cohen, 1992).

However, while effect sizes are useful (e.g., for comparing interventions, estimating the likely effect of an intervention in a new context), they provide little information on the actual change in behaviour and / or outcomes that might be expected because of an intervention. In this sense, the real-world implications of behavioural interventions are hidden from those who might want to use them (e.g., policy makers). For example, what does an effect of d = 0.20 of an intervention designed to promote physical activity mean in terms of changes in physical activity and associated outcomes?

Benchmarking involves identifying how effect sizes of varying magnitudes correspond to real world behaviour and outcomes. For example, Wright et al. (2021) used the standard deviations reported in large-scale studies that used accelerometer data to chart the distribution of physical activity in a population (e.g., Clark et al., 2019) to convert effect sizes from reviews of various interventions designed to promote physical activity. The headline finding was that median effect size reported by reviews of interventions (d+ = .21) was equivalent to an increase of 1,320 steps per day, 15.6 additional minutes of daily moderate/vigorous physical activity, and a 4.3% increase in the proportion of participants meeting WHO guidelines for physical activity.

This studentship will extend benchmarking to new contexts, working with each of the four themes with BR-UK to benchmark effect sizes associated with interventions relevant to: (i) Environment and sustainability (e.g., interventions designed to reduce plastic waste or household energy consumption), (ii) health and well-being (e.g., interventions targeting smoking or alcohol consumption), (iii) resilient communities (e.g., interventions designed to reduce prejudice), and (iv) organisations, markets, and the economy (e.g., interventions designed to improve performance at work). We will also consider whether it is valuable to extend benchmarking to (i) draw analogies with commonly encountered / understood metrics and (ii) show how small effects accumulate to have important outcomes. Our hope is that benchmarking effect sizes will increase confidence in behavioural science.

REQUIREMENTS:

A master’s degree (or relevant research experience) in Psychology or a related discipline and at least a 2:1 at BSc level. It will also be an advantage if candidates have good knowledge of meta-analysis and quantitative statistics.

Overseas applicants are welcome to apply but would need to self-fund the difference between home and overseas tuition fees. Evidence of funding will be required.

Start date is 1 October 2024.

All applications to be received via Postgraduate Applications Online System by 12 noon UK time 24 April 2024 . Information and enquiries via email to [email protected]