Monitoring and evaluating programmes that take a market systems approach presents a distinct set of challenges. As the number and scope of market systems programmes grows, finding solutions for robust measurement of impact is an increasingly central task. We share here the views of two presenters at the M4P Hub conference that met a growing interest in measuring results from both practitioners and donors.
Goetz Ebbecke, SwissContact - Experiences with the DCED-Standard in Katalyst
Incorporating the DCED standard into Katalyst has been a challenging but vital process; the framework the standard provides has helped to guide and structure our monitoring and results measurement (MRM) system into a far more rationalised and thoughtful tool in our work. The inclusion has also very much been a natural evolution rather than an addition; there have been many similarities between the development of the standard and the issues we have faced over the years since Katalyst’s inception in 2003, and we believe it is in part this parallel learning experience that has made the standard such an appropriate tool for us.
Yet it was no light task. The main challenges faced were Katalyst’s sheer size with currently over 70 interventions in 16 sectors, aggregation, and building culture and ownership for the full integration of MRM in the design, management and implementation of our interventions. MRM in Katalyst is not simply an overhead or add-on; rather it is embedded at the heart of our management. Of course such a system needs significant resources, but this investment is vital and worth it when managing a project like Katalyst.
With the grounding in place, we are now focusing on exploring how we might better capture qualitative factors of systemic change, and we look forward to seeing how the standard may continue to assist us as we strive for continuous learning, refinement, and improvement.
You can access the conference presentation here: https://m4phub.org/userfiles/file/Brighton_9-11-2011.pdf
Julian Barr&Chris Barnett, ITAD - Value for money approaches for M4P projects
Donors, particularly bilateral donors accountable to their taxpayers, are very concerned that the development interventions they fund demonstrate value for money (VFM). This is true for M4P interventions, as it is interventions in sectors such as health, education and governance.
Building on our work for DFID’s Politics and the State Team on developing a VFM framework governance and conflict programming – also areas with difficult to measure results and indirect causality pathways – we explored a similar approach to developing a VFM framework for M4P. Our framework is based on a spectrum of ways to tackle VFM, from managing for VFM at one end (essentially results-based management), to hard-core measuring VFM at the other. Many people are worried that the latter drives programmes entirely towards a strict monetarisation approach to VFM, although given the nature of the high level results in the DCED standard, M4P is better placed than many types of intervention to respond.
Our paper presented a ‘middle-way’ approach to VFM, that is particularly useful to appraise VFM during both the design phase and mid-stream implementation of programmes. The approach draws on audit approaches and has some similarity to business process quality assurance, as manifest in ISO9001. We call it a ‘weightings and ratings’ approach. Its basis is to select key processes in the intervention, and develop specific criteria against which to rate the ‘quality’ of these processes. Different processes are weighted according to their importance in achieving outcomes and impact.
The main thrust of questions after the presentations were around the value for money of VFM approaches; i.e. in the context of interventions with limited management budget, isn’t assessing VFM an undue overhead? Our view is that VFM is good management, and is the basis of what good managers do anyway; it shouldn’t impose a significant extra overhead, and can be very useful for manager in ensuring resources are efficiently and effectively focused on the key areas for success. Thus, programmes need to see VFM as not only a one-off measurement of costs and benefits, but a tool to be used more proactively in the management of the programme.
You can access the conference presentation here: https://m4phub.org/userfiles/file/VFM%204%20M4P%20(ITAD).pdf