The MULTI-ACT project is in the final stages of developing a key component of the research and innovation documentation infrastructure to ensure effective and outcome-orientated cooperation between all stakeholders through a new Collective Research Impact Framework. In this blog post we’ll reflect on its use for health care managers.   

What is the problem MULTI-ACT is trying to address?

We all know that researchers, patients and health care managers speak different languages, talk about different things and anticipate different outcomes when they consider research and innovation. Bringing all stakeholders together and teaching them to speak with each other rather than to each other is an important part of creating lasting and effective innovation in health care. This is particularly critical when it comes to providing health care solutions to patients with brain disease. Patients with brain pathologies present particular challenges to doctors, care workers as well as researchers trying to develop innovations to improve their care outcomes.

What is the solution MULTI-ACT offers?

MULTI-ACT appears to have stepped into this important field with the ambition to create guidance for all stakeholders to facilitate effective cooperation between patients and those professionally engaged with their care and researchers. MULTI-ACT aspires to produce tools based on a new model of effective cooperation including detailed guidance facilitating a 4-step care pathway. Scorecards are developed in parallel to this, helping all stakeholders to base their actions on shared values, identified patient needs and expectations. This aligns with the key requirements of all modern and progressive health care: to be patient centred, supportive and guided by the values of compassion, care and deliver this in a collaborative spirit to achieve holistic care outcomes.

The issue of patient engagement in decision-making

I would like to focus my comments below on the intricate issue of patient engagement which present particular challenges to health care staff in this field. Shared decision making is now a key criterion of high-quality health care delivery. We know that is results in better decisions, improved health care outcomes and increased medication compliance and lower care pathway attrition rates.

However, the barriers to implementing shared decision making remain huge. Even today, medical students are still inculcated with a deep belief in the superiority of medical evidence-based knowledge. Patient perspectives are still routinely neglected or delegitimised in patient doctor conversations. Yet, we know that holistic and effective health care can only be delivered when patients feel part of the solution. Their commitment to the health care packages depends on them feeling that their concerns, reservations and ideas about what constitutes good health care are being taken seriously. This often jars with what doctors think about their health care based on scientific evidence and detailed professional medical knowledge. Patient doctor encounters are too often still a clash between different perspectives, different cultures and different visions of what constitutes good care. MULTI-ACT provides the building blocks to build the bridge between all stakeholders when constructing efficacious care packages for patients with brain disease. It pays particular attention to the issues around patient engagement in research and provides a useful framework structuring patient engagement activity for researchers and clinicians. Its roadmap including four steps for patient engagement is instructive for everyone involved (see below).


It ensures meaningful engagement of patients in the process of innovation and locates MULTI-ACT’s facilitation work clearly in the upper rungs of Arnstein’s ladder of participation which is a widely accepted measure of patient involvement (Kaehne, Bray and Horowicz, 2020).

The project however also develops strategic management tools such as the MASTER Scorecard which allow stakeholders to demonstrate the potential impact of the innovations and initiatives for patient care. This builds the link between the innovation itself, grounded in shared decision making based on meaningful patient engagement, and the implementation phase where it is essential for managers to assess the impact on patients. Too often care innovations are well meaning and may prove to be effective but have little impact on patient’s life as their implementation is not assessed in the long term. MULTI-ACT provides a unique mechanism to support managers in the innovation cycle from inception, patient engagement all the way to post-implementation analysis.

It thus provides a unique tool to improve patient centred care for people with brain diseases that will be immensely useful for managers and health care providers in the future.  

This blog was written by Dr Axel Kaehne, President EHMA, Reader in Health Services Research, Medical School, Edge Hill University


Kaehne A, Bray L and Horowicz E. (2020). Co-producing health care – pragmatic principles and an illustration. Emerald Open Res 2020, 2:10

Does health research and innovation impact patients? How are we currently measuring this?

At present, the impact of research and innovation (R&I) is evaluated primarily through scientific publications, highlighting its excellence, but not sufficiently demonstrating the impact that R&I has on patients. Patient Reported Outcomes (PROs) – outcomes evaluated directly by the patient and based on the patient’s perception of their disease and its treatments – are used to evaluate the effectiveness of R&I in clinical trials. Could they also be used to assess the impact of R&I as a whole?  

What can we do to ensure that we are creating impact on outcomes that matter to all?

To increase the impact of research on patients and society, the evaluation of health R&I must move beyond conventional metrics measuring financial return or research publications. Only considering these measures is not enough because they focus only on part of the effects of health R&I. In contrast to this approach, health R&I need to be targeted, designed, implemented and evaluated taking into account the plurality of the interests and perspectives of multiple stakeholders involved in the research. Moreover, the patients’ voices and needs should matter throughout the entire research process. Specifically, health R&I should be evaluated through a collective accountability approach. This co-accountability approach requires the development of brand-new tools, procedures and metrics that shows the value generated by health research to society.

To achieve this purpose, MULTI-ACT has developed a model, known as the Collective Research Impact Framework (CRIF). It allows the effective cooperation of all the key actors involved in the health research field. It particularly relies on the direct collaboration and continuous dialogue among researchers, patients, patient organizations, pharma industry, and other relevant stakeholders to develop and implement a novel evaluation framework that suggests metrics to evaluate the impact of health R&I across five different but interrelated dimensions: conventional metrics related to the excellence dimension are integrated with new measures related to the economic and financial dimension (efficiency), to the social dimension, as well as to a dimension related to achieving mission success (mission dimension – efficacy; one explicit driver of the MULTI-ACT co-accountability approach). A fifth and transversal dimension is applied throughout the four dimensions of MULTI-ACT model as a tool for investigating the impact of R&I on outcomes that matter most to patients (Patient Reported Dimension (PRD)).

Why is the Patient-Reported Dimension (PRD) at the core of the CRIF?

MULTI-ACT advocates that patients must have a central role in the assessment of the whole health R&I process. The PRD considers patients as the key stakeholders, whose needs, knowledge, experiences and perspectives must be understood and incorporated into the health research evaluation process. Indeed, the PRD is not simply a dimension in itself, but rather it represents a key and transversal approach in which the other four dimensions (efficacy, excellence, social and economic) are rooted. In practice, this means that, although the efficacy (mission), excellence, social and economic dimensions should be evaluated through the context-specific indicators, the dimensions must also consider Patient-Reported Outcome (PRO) measures that incorporate the needs, interests and perspectives of patients.

Figure 1. CRIF dimensions

How does co-accountability make the Patient-Reported Dimension effective?

Co-accountability implies taking an approach that promotes the integration and involvement of relevant actors, and most importantly, patients in the design, implementation and evaluation of health R&I.

For this to be possible, MULTI-ACT is grounded on five co-accountability pillars that guide and support the engagement of patients and other stakeholders in the different steps of the health R&I process. All pillars require dialogue and receiving feedback during the research process. This collective impact assessment process is cyclical and iterative by nature and specifically promotes stakeholder engagement in (1) the definition of research priorities and objectives, (2) the development of an operative Framework, (3) the co-selection of indicators as well as (4) the collection and analysis of data to evaluate research, and (5) the reporting, monitoring and assessment of the health R&I process.

Within this co-accountability approach, MULTI-ACT seeks to guarantee that the needs and perspectives of patients are given the relevance they should have in health R&I. Therefore, the co-selection of indicators is one of the most relevant pillars. It ensures that the perspectives of patients and other stakeholders are considered when defining the evaluation system to be used to assess the impact of health R&I on the five dimensions of the CRIF.

Figure 2. Co-accontability Master Scorecard

This blog was written by: UNITN – Ericka Costa, Michele Andreaus; UBU – Carlos Larrinaga, Nicolás Garcia Torea, Juliette Senn; TAU – Anna Kork; FISM – Deborah Bertorello, Giampaolo Brichetto, Paola Zaratin

This blog entry is a direct continuation of the previous blog, “#6Patient Reported Outcomes: let’s keep patients engaged as key stakeholder in R&I”. Read it here and #2What is co-accountability and why is it important for multi-stakeholder initiatives?. Read it here.


Have you ever heard about Patient Reported Outcomes; do you know what they are?

Patient Reported Outcomes (PROs) are:

 “Any outcome evaluated directly by the patient him/herself and based on patient’s perception of a disease and its treatment(s)” (European Medicines Agency, 2014).

PROs are of utmost importance to the research & healthcare community for many reasons, such as research outcomes evaluation (e.g. use of PROs in clinical trials or observational studies), Research & Innovation (R&I) impact assessment, but also in healthcare for monitoring of disease progression and evolution.

Moreover, PROs are a core example of patient engagement. Although there is still much room for improvement towards a truly participatory approach in the design of the measures themselves, which are right now settled mainly in a top-down approach by clinicians. So far, PROs have mainly been used in post-marketing observational studies, though now they are increasingly used as secondary or tertiary outcomes in clinical trials on disease-modifying therapies and symptomatic treatments, whereas in rehabilitation trials are used as primary or co-primary outcomes. In this context, the integration of PROs with e-Health will be crucial, in the next years electronic PROs (ePRO) will be available. This revolution will allow to collect discrete data on self-perception of a disease and improve the usability of PROs in clinical and research areas.

The importance that PRO has gained in the last years is also highlighted by the many emerging initiatives in the field, with examples in the field of Multiple Sclerosis (MS): The Patient Reported Outcomes Initiative for MS (PROMS)[1] or the iConquerMS. There are also several databases at national level, such us the PROMOPRO-MS database, promoted by the Italian MS Society.

Thus, can we use PROs to maximize the impact of R&I?

We did it in MULTI-ACT, creating the Patient Reported Dimension (PRD).

Let’s see what exactly is the PRD…

MULTI-ACT proposes a co-accountability approach[2] in which conventional metrics related to the excellence dimension are integrated with new measures related to the economic (efficiency), social, efficacy (achieving the mission), and patient reported dimensions, resulting in the MULTI-ACT Master Scorecard[3].

The Patient-Reported Dimension (PRD) is a transversal dimension applied throughout the four dimensions of the MULTI-ACT model for enabling the Science of Patient Input [4], where PROs are investigated as metrics able to measure the impact of R&I on outcomes that matter most to patients.

The PRD includes indicators that are reported by patients, family and caregivers addressing functional reported aspects (i.e. PRO) and psychosocial reported aspects (i.e. metrics to evaluate the Return on Engagement – RoE). The indicators can be a collection of answers to questionnaires and active and/or passive data collection without the intervention of clinicians (e.g. eHealth via App/ICT devices like wearables or electronic bracelets). The main feature of the PRD is that it reports the perspective of the patient (PROs, RoE) or provides continued objective data (eHealth), therefore it’s not influenced by the clinician.

MULTI-ACT co-accountability focuses on the PROs (functional reported aspects) as it foresees the development of PROs as key indicators of impact, instrumental to enable a multi-stakeholder approach and effective patient engagement. Also psychosocial aspects (RoE) are important, and there is indeed the need to develop effective indicators.

A concrete example of functional reported aspects?

Take for example Multiple Sclerosis (MS), the case study of MULTI-ACT. The functional domains that matter most to people affected by MS are: Quality of Life and Satisfaction, Anxiety and Depression, Fatigue, Upper-limb dexterity, Locomotion, Cognitive function, Bladder function. Thus, PROs suitable to evaluate the impact of R&I on the outcomes that matter to them are indicators able to assess: patients’ satisfaction with their quality of life(Life Satisfaction Index and Neuro-QoL – Quality of Life in Neurological Disorders), their level of upper-limb dexterity (Abilhand – Manual ability for adults with upper limb impairment), their level of Anxiety and Depression (HADS – Hospital Anxiety and Depression Scale), their level of bladder function (OAB-Q – Overactive Bladder Questionnaire), their level of motor, cognitive, psycho-social fatigue(M-FIS – Modified-Fatigue-Impact-Scale, their level of locomotion (Twelve Item MS Walking Scale – MSWS-12). Click here for more info.

Functional domains that matter to people with MS (PwMS)

In fact, there is consensus in the clinical and scientific community that the classical clinical scales that are used to classify the level of disability in Multiple Sclerosis, such as the long-established Expanded Disability Status Scale (EDSS: a clinical scale used to classify level of disability in Multiple Sclerosis from 0 (no disability) to 9.5 (restricted to bed)), but also the Multiple Sclerosis Functional Composite Score (MSFC: is a composite score based on a walking performance test; a cognitive test and an upper limb performance test) ) are inadequate to capture the change of the patients’ clinical condition. See here for more info.

And what about psychosocial reported aspects?

Following a Public Consultation performed under MULTI-ACT activities and its Patient Engagement Guidelines, a selection of indicators that can be used for assessing the Return on Patient Engagement (RoE) have been identified to assess both the performance of patient engagement (i.e. the success of the initiative in terms of participation) and its effectiveness (i.e. the success of the initiative in term of real impact of the participation on the research process). An example of indicators for psychosocial reported aspects is presented below:

But, why is PRD so important?

The PRD is of utmost importance to evaluate the impact of R&I on outcomes that matter most to patients and keep patients and stakeholders engaged along the R&I continuum.

MULTI-ACT co-accountability focuses on the PROs (functional reported aspects) as it foresees the development of PROs as key indicators of impact, instrumental to enable a multi-stakeholder approach and effective patient engagement. The fact that PROs are scientifically validated measures reported by the patient (final beneficiary of the health research) capture the interest of all the stakeholders. Indeed, PROs are able to focus on the needs of the final beneficiary of R&I and also to capture outcomes that correspond to the needs of all the other stakeholders.

We would love to hear your thoughts! If you wish, please email us a statement: We are curious to hear from your society. Comments will be used in the further steps of MULTI-ACT development.


This blog was written by: Deborah Bertorello, Giampaolo Brichetto, Paola Zaratin.

[1] To know more on the initiative read: Measuring outcomes that matter most to people with multiple sclerosis: the role of patient-reported outcomes

[2] This blog entry is a direct continuation of the previous blog, What is co-accountability and why is it important for multi-stakeholder initiatives?. Read it here

[3] The Master Scorecard (MSC) is an adaptive tool for the application of the co-accountability model and its five dimensions. The MSC consists of a detailed list of indicators evaluating aspects of measurement linked to the different dimensions that can be tailored into different contexts and missions.

[4] “Science OF patient inputs” occurs when data of people with a disease are used (active and passive contribution) to evaluate the impact of R&I.

What is Theory of Change and why is it a useful, yet imperfect, approach for making change real

India, West Bengali. It is a sunny morning and a brand-new project is about to start: it is a capacity building initiative that aims at training judges, social workers, paralegals, lawyers and NGOs on human trafficking rules, laws and regulations. Everything is ready: the classroom is equipped, there are pens, notebooks and background materials on the desk. There is coffee, tea and water, and a bunch of flowers on the welcoming desk. Some local authorities have been invited and should attend the training’s kick-off.

But when the time came, none of the trainees showed up… Wrong venue? Wrong date? Wrong timing?

No. Simply, lack of a strong Theory of Change (ToC): the zealous and dedicated project designers and managers developed an initiative, without carrying out a proper ToC, not considering all the risks and assumptions and not involving proper stakeholders in the project design phase. Working with an activity-based perspective, instead of a result-oriented vision in an open context, they overlooked a small, insignificant and crucial detail: India has a caste system still deeply enrooted and it would have been impossible for trainees, belonging to different castes, to sit in the same class. Promptly, thanks to a risk management strategy and a flexible way of responding, the training calendar was revisited, the classes were reorganized, and the project had a positive result and a “happy ending”.

But, wait, what is a Theory of Change?

The Theory of Change approach was popularized in the 90’s as a methodology applied to development interventions, but its application has been increasingly spread to other areas, becoming more structured, also thanks to several studies and to the work of experts.

A good definition is provided by the site:

A Theory of Change is essentially a comprehensive description and illustration of how and why a desired change is expected to happen in a particular context. It is focused in particular on mapping out or “filling in” what has been described as the “missing middle” between what a program or change initiative does – its activities or interventions – and how these lead to desired goals being achieved”1.

In other words, using the definition of USAID, a Theory of Change, might be expressed as, “If we do X, Y and Z, it will lead to W”2.

A ToC is a participative approach that helps to identify a causal pathway and to define the rationale behind an intervention.

It allows reasoning on the most important, yet often misunderstood, aspect of a social intervention: the difference between implementing an activity and bringing about a change.

The process to build a ToC starts by identifying the desired long-term goals and then works back from these to identify the outcomes, and how they are related to one other causally, for the goals to be achieved. Then, the process continues, mapping out the outputs that must be put in place to contribute to the outcomes, and the activities to be implemented, to allow the outputs to be realized. In building up the result chain, there is the need to conduct a proper assessment of the context, along with a detailed analysis of potential risks, that may occur and a strategy to face them, if needs be; taking into consideration also the assumptions, meaning the conditions that must be in place to allow the project to be implemented in a proper and timely manner. Exactly what the project designers skipped in the above mentioned case in West Bengali.

How could a ToC be useful for MULTI-ACT Model appliers?

According to MULTI-ACT Model, appliers should define a common agenda involving relevant stakeholders, identifying and tackling their intended issue with a unifying long-term approach and a clearly defined set of objectives and actions necessary to pursue their long-term goals.

The ToC methodology answer very well to this need and could represent a useful tool not only to clarify the objectives and the agenda of a given initiative, but also to monitor and manage the activities carried out, ensuring their coherence with the long-term goals. Therefore, a ToC could be seen as a dynamic tool that requires appliers to have a strong flexibility both in developing and revising their approach.

This is particularly true for the initiatives that will profit of MULTI-ACT Model, which will be required to develop a comprehensive description of how and why their desired change is expected to happen. A very difficult task to perform considering the nature of health research and the need of engaging stakeholders in the research process. For this reason, it is extremely important to trigger a deep reflection on the process that move from desired long-term goals to outcomes, outputs and activities, and on the control or influence that the initiative might have on the attended results.

One of the purpose of MULTI-ACT is to offer a comprehensive methodology to appliers of the Model, which could help them both in the definition of their objectives and throughout the development of their initiative.

This blog was written by: Sabrina Liberalato, Edoardo Sangiovanni, Corrado Paternò Castello, Luca Molinari, Andrea Gavazzi, Ilaria Giuseppina Neirotti



A central activity in MULTI-ACT is the establishment of a group of experts focused on patient engagement: the Patient Engagement Group (PEG).  This PEG has developed guidelines to engage patients in activities throughout the Research & Innovation (R&I) Continuum.

There are many opportunities for patient engagement but so far these have generally focused on medicine lifecycle driven initiatives in which patients have been engaged to improve study design to ensure a trial is acceptable to participants

In order to develop science with patient input MULTI-ACT has identified the sequence of activities in the R&I continuum where patients should be engaged in order to maximize the impact of R&I.

MULTI-ACT has identified 7 steps in the R&I path in which patients can contribute in different forms.

The MULTI-ACT PEG is also developing strategies to mediate, negotiate and build shared perspectives on the development of innovative metrics.

Research outcome measurement is complex, involving different dimensions. This complexity requires to consider the values and the perspectives of different stakeholders, in particular those who can input their immediate and personal insight into research assessment: the patients.

Nowadays reality is that “many PROs were developed with insufficient understanding about what matters to patients and inadequate partnership with patients (as highlighted in the “Patient-Reported Outcomes: Design with the End in Mind” (Milken Institute, Cynthia Grossman, January 2018)

A distinctive characteristic of MULTI-ACT is co-creation. MULTI-ACT has the objective to produce a collective impact framework (CRIF), a methodology to tackle deeply entrenched and complex social problems. Collective Impact is an innovative and structured approach to making collaboration work across government, business, philanthropy, non-profit organizations and citizens to achieve significant and lasting social change. The approach calls for multiple organizations or entities from different sectors to abandon their own agenda in favour of a common agenda, shared measurement and alignment of effort.

In a collective impact framework, the different stakeholders don’t have the goal to see their particular interest reflected in the model but sharing a common vision and agenda.

Unlike collaboration or partnership, Collective Impact initiatives have centralized infrastructure – known as a backbone organization – with dedicated staff whose role is to help participating organizations shift from acting alone to acting in concert.

The CRIF has 5 dimensions: excellence, efficiency, mission and social and the “patient reported dimension” that is considered as a core point of the model.

The central role of patient reported dimension does not signify that patients are more important than other stakeholders but that the whole research process, and the health research impact assessment, has to be done with an active participation of patients.

A key challenge to enable Patient Reported Outcomes (PROs) as key metric to measure impact of research on patients is for MULTI-ACT to help finding a happy medium between PROs that work at population level and PROs that can be individualized for use in care and treatment.

This blog entry is a direct continuation of the previous blog, “Responsible research & Innovation – reducing the distance between science and society”. Read it here.

Over the last decade, several attempts have been made to try and reduce the distance between science and society. These have led to the definition of a new approach, which has been called Responsible Research and Innovation (RRI).

Responsible Research & Innovation (or RRI) is a cross-cutting approach that aims to make research more inclusive, participatory and ethically responsible. 

RRI seeks to openly raise issues associated with research and innovation in order to anticipate the consequences and involve society in the discussion of how science and technology can help create a society we would want for future generations. (source:

RRI implies that societal actors (researchers, citizens, policymakers, business, third sector organisations, etc.) work together during the whole research and innovation process in order to better align both the process and its outcomes with the values, needs and expectations of society. In healthcare research particularly, this means to put the expectations of patients and society at the heart of research programs.

Below are the key themes of RRI:


MULTI-ACT responds to the following RRI key themes:

How will MULTI-ACT achieve it?