Monitoring, evaluation and learning strategy (Phase I) and implementation (Phase II) for a healthcare-focused philanthropy

Introduction and Client Context

The Bristol Myers Squibb Foundation (BMSF), established in 1990 as a philanthropic arm of Bristol Myers Squibb, is a global leader in promoting health equity and improving health outcomes for populations disproportionately affected by serious diseases. Bristol Myers Squibb, the company, is a pharmaceutical company that produced ~$26 billion in revenue in 2019.

BMSF is a forward-thinking, mission-driven organization that seeks to promote health equity and improve health outcomes for populations around the world who are disproportionately affected by serious disease. BMSF drives change by funding innovative projects and solutions that improve medical care and community support for combatting disease.

Cicero’s work with BMSF was spread over two core phases. In the first phase, Cicero co-developed a rigorous monitoring and evaluation (M&E) strategy, outcome framework, and data system with the philanthropy team to help them determine, track, and aggregate the impact of the foundation’s investments on end-beneficiaries and institute a continuous learning process with its grantees. In tandem, the team defined a communication approach that would help the organization use insights emerging from M&E activities to bring greater visibility to the impact of its grantees work with beneficiaries around the world.

In the second phase, Cicero collected feedback from the client on their experience implementing the M&E strategy and refined the toolkit with a focus on improving its ease of use and adoption across the organization. This work was grounded in Cicero’s own philosophy of going on a learning journey with our clients. Where traditional advisory firms often put forth a strategy and close out an engagement, Cicero’s long-term relationships with our clients allow us to test whether our work helped yield value and fine-tune our findings until they deliver the intended impact.

Phase 1: MEL needs assessment and strategy

Initial Challenges​

In January 2020, the Foundation’s Board of Directors approved a new strategy that caused the Foundation to re-evaluate its global footprint and funding focal points. Foundation leadership decided that they needed a more streamlined and standardized monitoring and evaluation (M&E) process to tell a more cohesive impact narrative across the portfolio of grants and geographies. The Foundation also wanted to develop a system that focused more on demonstrating the tangible outcomes of their interventions as opposed to simply conveying outputs. The organization aimed to answer questions like the following:

1. Evaluation: Which measures should BMSF use for new programs and outcomes? How can BMSF improve the questions, data quality, and integration of its measurement and evaluation of programs and outcomes?
2. Sustainability: How can BMSF support its grantees to achieve the long-term sustainability of their impact and efforts beyond BMSF funding?
3. Storytelling: How can BMSF effectively communicate the significant impact it has achieved through its work and support?
4. Application: How can BMSF establish clear expectations for grantees to align on outcomes, data collection, and reporting from the start?

Cicero’s Approach & Results​

Cicero worked closely with the BMSF staff to develop a carefully thought-out impact framework—a clear map of how the organization intended to achieve health equity. This framework included the foundation’s primary objectives, target audiences, core values, and key activities and reflected both the input of the staff and external best practices from the field of health equity. Cicero gathered the latter by conducting extensive industry research and in-depth interviews with medical experts, grantees, and thought leaders. Once finalized, Cicero leveraged the impact framework to guide BMSF through developing a comprehensive, complementary list of indicators that would position BMSF to assess and convey its impact at the grantee, initiative, and foundation levels.

With an impact framework and list of indicators in hand, Cicero developed a series of tools for BMSF to operationalize the new M&E system. These tools included survey instruments through which to gather grantee impact data, impact report prototypes, discussion guides to support grantee oversight, a storytelling toolkit, and a comprehensive handbook outlining when and how to use each of the aforementioned items.

Phase 2: Fine-tuning for MEL user experience

Emergent challenges after MEL strategy design:

After taking time to trial BMSF’s new M&E collection and reporting system, grantees indicated strong preferences for a simplified set of metrics and the foundation sought to streamline the back-end reporting process. BMSF called upon Cicero to simplify the impact framework and M&E tools and improve the capacity of grantees and staff to be able to implement the M&E strategy. To ensure successful adoption the new framework, tools, processes, and systems across the Foundation, Cicero proposed a comprehensive implementation scope of work to help the Foundation which included: 

1. Tools & Technology: develop and deploy the necessary tools and technology to collect, manage, analyze, and report on M&E data.  
2. M&E Transition: design and manage a thoughtful transition to ensure the new M&E tools, systems, and processes are fully understood and effectively adopted by Foundation staff, key partner organizations, and the full portfolio of grantees. 
3. Reporting & Continuous Improvement: refine the Foundation’s approach to and processes for internal and external reporting and communications, program reviews, and strategic planning, and ongoing improvement efforts.  

Cicero’s Approach & Results​ in Phase II

Cicero’s approach was shaped by two end-goals: (1) improve the user experience (across all foundation stakeholders) with MEL tools and processes, and (2) ensure the MEL process yields compelling data and stories to inform the Foundation’s communications and future programming. To do so, the Cicero team built easy-to-use tools to help manage the MEL data lifecycle, from collection, to analysis, to insights, to communication and tested these with intended users, refining them along the way. This included building an impact and alignment survey in Qualtrics for use by grantees and program managers, and creating supporting literature to align with and onboard users onto these systems (e.g., admin manuals, onboarding decks, and a training program). Program managers were trained in the use of Tableau dashboard to evaluate and visualize monitoring data. The communications team was equipped with reporting templates to easily convert insights into stories for internal and external use.

To make sure that transition to new systems was smooth, this aforementioned process was kept as participatory as possible. Grantees provided valuable feedback on the use of survey tools based on which these tools were fine-tuned. Managers were trained on the use of Tableau in interactive training sessions and handheld during troubleshooting. To ensure capacity was institutionalized in the organization, Cicero roped in the business insights and analytics team from the BMSF corporate unit to help build tools and train their foundation colleagues on their use.

Conclusion and Key Takeaways​

The long-term structure of this engagement allowed Cicero to provide support across the MEL lifecycle: from designing, testing, refining, and operationalizing BMSF’s MEL strategy. In Phase I, Cicero equipped BMSF with the ability to evaluate the effectiveness of its grantee and foundation-led activities and, in turn, more readily identify and share evidence-based interventions with the field of health equity. In addition to ensuring a more efficient use of resources, this approach drives strategic alignment across the foundation and facilitates transparency with grantees from grant inception through close-out. Above all else, the new system positions BMSF to collect, aggregate, analyze, and report data insights in a way that shines a spotlight on individual grantee achievements while also conveying the full impact of the foundation’s rich portfolio of activities. 

By simplifying and fine-tuning the M&E framework and methodology in Phase II, Cicero improved the adoption and use of M&E tools by BMSF grantees and staff alike.  By piloting survey implementation and running training sessions, we helped build the technical competency of grantees and foundation staff with analytical and visualization tools (such as Tableau and Qualtrics) for M&E activities. This reduced turn-around time of converting data to insights and insights to impact narratives, a process that was previously cumbersome and time-intensive. Our team also helped foster a stronger working relationship between the philanthropy and corporate unit by creating a culture of sharing and collaboration between the communication teams of the two organizations. A tangible result of this was the inclusion of social impact narratives in the annual corporate report of Bristol Myers Squibb that was built using Cicero’s impact dashboard and storytelling templates.

Overall, Cicero’s engagement enabled the foundation to more efficiently and effectively monitor the impact of its work, empowered grantees to communicate their value more clearly, and helped the corporate unit to realize the effects of its philanthropic allocations more transparently.

Figure 1: Templates for impact narratives emerging out of the MEL process