Completed projects

DPMG was selected to undertake a midterm review of the Open Government Partnership (OGP). OGP aims to secure concrete commitments from governments to promote transparency, empower citizens, fight corruption and harness new technologies to strengthen governance. This Mid-Term Review covered the five-year period from September 2011 through September 2016, and helped to inform OGP’s strategic refresh. The review focused on:

• Setting the context: macro-level outcomes on the state of the partnership
• Assessing progress in OGP’s theory of change: the role of OGP’s domestic policy mechanism and international platform in helping deliver outcomes
• Assessing effectiveness of OGP’s rules of the game: the role of OGP’s structure and policies in helping to deliver outcomes

The draft final report assessed four key areas, a) OGP’s theory of change, b) OGP’s chain of interactions and outcomes, c) rules of the game, and d) structure, organs, and finance. The final report can be found here. 

Client Feedback

"Thank you very much for working with us over the past six months, for your diligence and for your flexibility in adapting to our needs when they arose. It was a pleasure working with your team."  - Munyema Hasan, Program Manager, Learning and Impact, Open Government Partnership Support Unit 

DPMG was selected by Finland’s Ministry for Foreign Affairs to carry out an evaluation of inclusive education in Finland’s development cooperation between 2004 and 2013. The evaluation was designed to assess inclusiveness in education through country and regional case studies. It also assessed the Finnish development cooperation from the disability perspective through a desk study. This will contribute to the overall assessment of the Finnish application of the Human Rights-Based Approach (HRBA) in development cooperation. DPMG’s multi-disciplinary team consisted of sector-specialists, country-specific teams, and a qualitative methods expert. 

The evaluation had five components. The first component consists of a desk study on the enhancement of rights and equal opportunities of participation of people with disabilities and will provide overall context for the inclusiveness in the Finnish development cooperation. The second component consists of the final evaluation of Finnish cooperation in the education sector in Kosovo with focus on inclusive education. The third component is the final evaluation of Finnish cooperation in education sector in the Andean region with emphasis on bilingual education. The fourth component consists of case study on Finnish development cooperation in inclusive education in Ethiopia. The fifth component merges the findings of the other components and consists of a synthesis report. All components are closely interlinked and the evaluation is organized in such a way that cross-fertilization between the different components can take place. The reports and the webinar can be found here.

Client Feedback

"Thank you for all the hard work you have put into this assignment which reflects in the good quality of the final reports. It has been a great pleasure to work with you... Hopefully we will have the opportunity to work together again one day." - Sanna Takala, Evaluation Manager, Ministry for Foreign Affairs of Finland

DPMG was selected by Norwegian Agency for Development Cooperation (Norad) to evaluate Norwegian multilateral support to basic education, specifically through the United Nations Children’s Fund (UNICEF) and the Global Partnership for Education (GPE). The purpose of the evaluation was to facilitate more evidence-based policy and programming decisions both in Norway and in UNICEF and GPE, with a dual focus on accountability and learning. This was achieved through generating evaluation evidence on the relevance, efficiency and effectiveness of Norwegian aid to basic education through UNICEF and GPE, focusing particularly on the achievement of Norway’s policy objectives quality of learning, gender equality and equity, and through increasing the knowledge base of basic education. DPMG’s international team also conducted four country case studies in Nepal, Ethiopia, Malawi, and Madagascar. Click here for the final reports! 

Client Feedback

"Let me say that we are very pleased with the work that you have put into this report. We think you have responded in a very thorough manner to all the stakeholder comments – which we know was a big job -  as well as to the comments from our department, and we believe that the synthesis report is of high quality, and an interesting read. " - Eva Kløve, Senior Adviser, Norad

DPMG was selected by the Norwegian Agency of Development Cooperation (Norad) for a framework agreement governing “A series of evaluation assignments of the Norwegian Programme for Capacity Development in Higher Education (NORHED).” The purpose of this real-time evaluation program was to enable learning by relevant stakeholders within the field of higher education and development, so that future investments can become more effective in building capacity of higher education institutions. In particular, the evaluation program aimed to increase knowledge about the extent to which NORHED builds capacity in these institutions, and how capacity development can be conducted most effectively, and how higher education institutions affect development.

As part of the framework agreement, DPMG undertook a preparatory study to learn about what works and why in NORHED’s support to capacity development of higher education institutions in low- and medium income countries. The purpose of this particular study was to contribute to a discussion with the NORHED administration and other stakeholders about how the programme can make a difference, and to provide a basis for future evaluations of NORHED strategies. This was done through the development of a testable theoretical framework, including a proposal of methods and indicators, with input from, and verified through stakeholder consultation. The team developed a general theory of change of those factors required to develop the capacities needed to achieve NORHED’s objectives. The report is available on the Norad website here.

For its second assignment, DPMG undertook an Evaluation of NORHED’s award process. This evaluation described and assessed the mechanism that Norad uses to select grantees for the NORHED programme. The purpose of the evaluation is to improve the process of selecting grantees in the future. The report is available on the Norad website here.

For its third assignment, DPMG reviewed the evaluability of NORHED. The report examines the supervision processes in Norad and the results frameworks in the different NORHED projects. Based on this, it assesses whether it is possible to measure the results of the different projects. The report can be found on the Norad website here. 

Client Feedback

"We welcome the clear recommendations addressed to the NORHED programme staff [and…] we appreciate the team’s understanding of the NORHED programme as furthering capacity in education and research through the strengthening of academic units within specific disciplines/areas at target institutions rather than through overall institutional support. As the report correctly points out, the NORHED programme is not designed to directly change the system- and institutional level enabling conditions. The report’s account of these enabling conditions is therefore very valuable as a reminder of both fundamental risks to the programme and as opportunities for improving programme results. This point will be explored by the NORHED team in order to ensure that risks and opportunities are accounted for through ongoing monitoring and evaluation, to identify possible bottlenecks, to catch projects that are failing in time to get them back on track, and to guide potential new NORHED interventions.
We wish to thank the DPMG team for an instructive and useful report, and we look forward to further exchanges through the next phases of the real-time evaluation of the NORHED programme." - Norad’s Section for Research, Innovation and Higher Education

DPMG was selected by the World Bank for the First Independent Evaluation of the Partnership for Market Readiness (PMR). The Partnership for Market Readiness (PMR), which became operational in April 2011, is a grant-based, capacity building trust fund that provides funding and technical assistance for the collective innovation and piloting of market-based instruments that reduce greenhouse gas (GHG) emissions. DPMG’s three member team developed a detailed Evaluation Framework for periodic evaluations of PMR, assessed the effectiveness of the governance structure of PMR and the operational effectiveness of the grant allocation mechanism, and conducted an evaluation of PMR’s technical work, knowledge sharing and capacity building program at the country, regional and global levels. Click here for the final Evaluation Framework and click here for the final report. 

Client Feedback

“We would like to thank you and the team for your solid work on the PMR evaluation.  We very much appreciate your insight and recommendations, all of which will be very useful to help us take the PMR to the next phase.” – Xueman Wang, PMR Team Leader, The World Bank
“On behalf of the PMR Team, I would also like to thank you and your team for your work on the PMR evaluation. The PMR Participants and the Secretariat welcomed and appreciated your recommendations, which will be very useful to strengthen and enhance the PMR going forward.” – Adrien de Bassompierre, Senior Carbon Finance Specialist, The World Bank

DPMG completed a Program Completion Review of the Financial Sector Reform and Strengthening Initiative (FIRST) Phase II for the World Bank. FIRST’s mandate is to support the development of financial systems through technical assistance grants. This review covers the 349 FIRST-financed activities in Phase II. DPMG’s team reviewed FIRST’s portfolio, did an in-depth review of 29 projects for three case studies, interviewed 40 TTLs, clients, and stakeholders, and administered two surveys. The evaluation made an independent assessment and validation of FIRST’s i) achievement of project and program development objectives; ii) relevance and output quality; iii) additionality; iv) operational effectiveness and fund utilization; and v) lessons learnt and knowledge management. The review found that FIRST generally produces high quality, strategically relevant outputs, and that clients are highly satisfied with FIRST. The key issue FIRST faces is limited likely impact due to institutional limitations in the recipient countries and to a lesser extent, internal processes that can be made more efficient. For the full report, click here.

Client Feedback

“The Development Portfolio Management Group of the University of Southern California has presented an evaluation study of FIRST Phase II. It is a well thought, well written and rather insightful document that should be of great benefit to FIRST stakeholders’ greater understanding of FIRST and for the further improvement of its future operations. FIRST was fortunate to have a high caliber team working on this report, and we wish to express our appreciation for the openness and intelligence with which the report was prepared.” – Jorge Patino, FIRST Program Manager, The World Bank

DPMG completed a review of the use of evaluations provided by Norwegian Agency for Development Cooperation (Norad). This evaluation report was commissioned by the Norad Evaluation Department in order to respond to the need for understanding whether and how the evaluations it conducts are used, and how the systems for learning from and using those evaluations can be improved. In conducting this evaluation the team used a mixed-methods approach. This entailed document reviews, interviews with key informants, case studies of four specific evaluations, detailed review of the content and structure of related evaluation reports, a citation search for Norad evaluations, and a comparative organizational analysis of evaluation units in Norad peer organizations. The review found that Norad’s Evaluation Department has done a number of things to enhance its evaluation work’s usefulness and use by consulting with key stakeholders and establishing a system for disseminating reports, among other good practices. The limitations that the Evaluation Department faces in its effectiveness were two-pronged. First, in some cases evaluations were not targeted on high-priority issues or were not delivered in time to affect decisions. Secondly, some evaluation reports were found to not be user-friendly, albeit of good quality. The full report can be found online here.

Client Feedback

"Thank you for this final report. We appreciate the changes that have been made, in particular the sections added on cumulative learning and factors affecting decision making as well as questions added from the survey. There are many good observations, that will be important for EVAL in the remainder of our work.
There is huge interest in this study, and it has been nominated for a panel at the AEA annual conference, and will be discussed at other fora, notably in May, August and September, In Norway, Europe and elsewhere." – Beate Bull, Senior Adviser, Norad

DPMG completed a technical review and evaluation for the Arab Administrative Development Organization (ARADO). ARADO promotes and advances administrative development in the Arab region by increasing the level of knowledge and expertise on management and administrative development, informing government policy on administration, and education target beneficiaries. DPMG undertook a technical review and evaluation of ARADO activities from 2008 to 2012 in seven member-countries of the League of Arab States. The evaluation provided an assessment of the institutional framework, organizational performance, and internal control systems and processes. Additionally, it provided a strategic medium and long-term vision and formulated a strategic plan to improve ARADO’s current administration, operational performance, relevance, and impact. The review and evaluation concluded that ARADO provides valuable services in training, workshops, and certificate courses; however, there is a lack of linkages around ARADO’s different activities. The roots of these weaknesses are to be found both within the organization and in the constraints from Arab League regulations designed for publicly-funded entities and not for a self-supporting organization such as ARADO. It is the interaction between internal and external factors that has limited the effectiveness of the organization. The team suggested that ARADO exploit the potential synergy in its activities, in part by strengthening internal coordination and in part by streamlining contacts and consultations with each country. For the full report, click here.

Client Feedback

“Our thanks to…the DPMG team for the Report, which provides much strong analysis and many practical recommendations for the future.” – David Forrester, ARADO Advisor

DPMG undertook a review of the African Development Bank’s Annual Portfolio Performance Review (APPR) system. In the African Development Bank, the APPR has had two major purposes: first, it reviews the alignment of the evolution of the portfolio over the past year with ADB’s agreed corporate priorities; and second, it indicates the progress that has been made towards achieving the performance targets on portfolio management that form part of the corporate Results Management Framework. The report provided findings in a context where there are some concerns about portfolio performance that are not reflected in the existing system. The study identified three areas for consideration and improvement: i) the positioning of the APPR within ADB’s portfolio reporting framework; ii) its descriptive—rather than analytic—nature; and iii) the indicators used for the Results Measurement Framework.

For the Swiss Agency for Development and Cooperation (SDC), DPMG reviewed SDC’s results frameworks for country programs. As part of SDC’s reorganization process, result based management was strengthened at different levels and results frameworks for country programs were introduced. DPMG’s team was asked to assess the quality and pertinence of the model for country program results frameworks as well as selected applications of the model. The review found that the results framework could be improved by presenting more clearly the causal logical link between the issues and impediments facing the successful achievement of the country goals and SDC’s interventions.