By Charlie Gough, marine scientist, and Laura Robson, community health coordinator
What do you get if you put twelve evaluation specialists in a small room with four Population-Health-Environment (PHE) [1] practitioners? Five hours of mind-blowing conversations about capturing the complexity of Blue Ventures’ integrated work!
On a warm June morning in the quiet Devonshire city of Exeter, we (Charlie, Laura and Vik) found ourselves sitting outside the aptly named Exploding Bakery [2], sipping coffee with Kristen Stelljes, PHE Advisor to the Packard Foundation and PHE Ethiopia Consortium. Kristen has been living and working in Ethiopia for the past two years, with some fascinating insights to share about her time there. The reason that Kristen had joined us all the way from Addis Ababa was not simply to discuss the ins and outs of PHE, but also to meet with a group of eminent academics to discuss how we can evaluate such complex programmes. We talked about our expectations for the meeting as we walked around the city centre and strolled down the river, stretching our legs before heading over to the university campus to have our minds stretched by some very stimulating evaluation concepts.
This high-level panel had been brought together by Professor Paul Dieppe (Professor of Health & Wellbeing at the Exeter University) following a conversation with Blue Ventures’ Medical Director, Dr Vik Mohan. Sitting around the table in the conference room were some of the most forward-thinking minds in the field of the evaluation of complex systems, all waiting with eager anticipation to hear about PHE.
Women involved in Blue Ventures’ integrated PHE programme
After Paul’s opening and introductions from everyone, Kristen launched the meeting with an excellent overview of PHE. She described various evaluations conducted to date, which have largely focused on measuring outcomes. Studies from the Philippines and Ethiopia provide strong evidence that integrated PHE programmes result in better outcomes than single-sector approaches, but none of these evaluations manage to capture the full complexity of PHE by describing how it works; this was the main challenged to be tackled by the panel. Vik’s inspiring introduction to Blue Ventures’ integrated PHE programme in Velondriake, southwest Madagascar, grounded the ensuing discussions in the details of our work.
So how can we evaluate complex systems such as integrated PHE programmes? The academics were keen to delve into this question, with presentations showcasing various methods that could help us to describe how and why PHE works.
Dr Robin Durie started by introducing us to the concept of complexity; the idea being that the behaviour of a whole system is more than the sum of its parts. This means that a change in one part of a system can produce ripple effects throughout other parts of the system, with positive feedback loops or unintended consequences. This challenged us to think about our PHE programme not as an external intervention, but as part of a dynamic and unpredictable system, which can therefore be difficult to model.
Professor Laurence Moore, Dr Katrina Wyatt and Dr Oonagh Corrigan provided us with more food for thought, as they presented us with several tools that could be used to describe PHE processes. Laurence introduced us to social network analysis which involves mapping the relationships between different people, the strengths of those connections, and their coevolution with behaviours. This can enable us to identify “super facilitators” who are key change-makers within a system.
An indicator of transformational change from the Connecting Communities programme
Katrina’s presentation focused on capturing the complexity of transformational community change. She shared some inspiring stories about her involvement in the Connecting Communities programme, which engages some of the most deprived communities in the UK. Katrina is collecting their stories of change using a mixed methods approach including interviews, focus groups and longitudinal case studies. These case studies involve following individuals over time and tracking changes in their lives. Another interesting technique is participatory photography, which enables community members to represent transformations that are important to them by taking before and after photos, or creating photo stories that they can narrate. All of this information is used to develop a theory about how transformational community change is happening. Katrina’s presentation provided us with some exciting and pragmatic suggestions for how we could go about collecting stories that illuminate how PHE works in Velondriake. Oonaugh’s introduction to ethnographic approaches challenged us to consider how qualitative research methods could be used to capture people’s perceptions of change.
Last but by no means least, Professor Rob Anderson introduced us to realist evaluation. He started from the premise that all programmes are theories, because they are designed based on implicit theories about how social change is brought about. Realist evaluation seeks to make these programme theories explicit by looking at the mechanisms through which social change is produced. This approach recognises that programmes are embedded within social systems, so their outcomes are always mediated by the wider context. Realist evaluation therefore focuses on describing how interactions between mechanisms and contexts result in the outcomes that we observe.
Community meeting to discuss a locally managed marine area as part of Blue Ventures’ integrated PHE programme
These presentations set our minds racing with fresh possibilities! All of the PHE evaluations conducted to date have simply focused on assessing outcomes, and comparing them to single-sector programmes. What approaches like realist evaluation offer, however, is something completely and refreshingly different. They allow us to get inside the mysterious black box of PHE by mapping out the mechanisms through which outcomes are brought about, and by developing a theory of how PHE actually works. We can then compare our programme theory with other PHE programme theories in order to identify commonalities. This can help us to build up a picture of how these processes are leading to the transformational changes that we are observing across the board.
Weighing up an octopus catch in Velondriake, southwest Madagascar
These are game-changing insights for the PHE community! We left the meeting feeling completely inspired by these creative and enlightening ideas, liberated and empowered to think about tackling the issue of evaluation in new ways. Discussions continued over dinner and late into the evening, and even into the early hours of the following morning… So what are the next steps? We’re starting by sharing this learning within Blue Ventures, and with the wider PHE community.
We’re also deepening our understanding of these various methods by continuing our discussions with the academics. Then we’ll start to refine our evaluation questions and decide upon our approach. In the long-term we’re hoping to conduct a process evaluation of our PHE programme in order to develop a robust theory of how PHE works in our context. By documenting our chosen evaluation approach and supporting other organisations to conduct similar evaluations, we could build up a picture of how PHE works in various contexts. These descriptive case studies and programme theories would represent a huge contribution to the PHE community in terms of increasing and diversifying our evidence base.
Mind-bending stuff and not a p-value in sight!
For more information about the meeting please see this press release or contact Laura Robson.
[1] Population-Health-Environment is an integrated approach for tackling the interconnected challenges of poor health, population growth, unsustainable natural resource management, food insecurity, environmental degradation and climate change vulnerability. Blue Ventures’ integrated PHE programme in southwest Madagascar encompasses family planning, maternal and child health, water and sanitation, fisheries closures, locally managed marine areas, aquaculture enterprises and community education.
[2] A local café whose name conjures visions of dough oozing from windows out onto the street; a rather appropriate name considering the explosions that would be occurring within our grey matter later that day!
