Part 2f - User involvement in research, monitoring and evaluation

User involvement in Research, Monitoring and Evaluation

Research, Monitoring and Evaluation use similar skills and approaches, but there are distinctions between them.

Definitions:

Research - an inquiry that seeks to reveal or extend knowledge through the systematic collection of information. While research in the natural sciences sets out to test a theory, social research aims to describe, interpret and better understand social processes, events or settings.

Monitoring - the root of the word is to remind or give warning.
Monitoring compares what is actually happening to an agreed standard of best practice. Monitoring needs to be ongoing to ensure that services reach and continue to meet that standard. The element of scrutiny /accountability means that a service cannot monitor itself.

Evaluation - collecting information that allows an assessment of how well a programme, service or organisation does what it has set out to do, in terms of effectiveness, efficiency and social acceptability.¹ For example, in meeting the needs of service users, as they themselves define them.

'Research' has been used in this section as the term to cover each form of inquiry.

Mental health and learning difficulty service users have led the way in actively participating in research and service monitoring and evaluation. Others can do so too, provided that they are:

Adequately trained (being a service user doesn't make you a good interviewer, such skills are learned and practised)
Supported (including access to advice on research method, debriefing after interviews, admin support)
Paid for their work and expertise

'User-led' does not mean amateur!

Why should such exercises be user-led?

"The path along which research progresses depends partly upon who decides what matters are examined, and how." ²

To ensure that they are relevant in that they include the concerns and interests of service users
To help avoid professional/researcher bias by including many perspectives
To aid understanding and interpretation
To identify gaps in inquiry
To increase accountability to the people who use services

A ladder for levels of user involvement in research, monitoring and evaluation³

See Arnstein's Ladder of Participation

User-led

Service users (or community members) have overall control of the process.

'Professional' and user-researchers work as colleagues, and all contribute skills or experience that are equally valued

Service users and researchers collaborate in conducting a process that has been instigated elsewhere

Service users act as advisors to the researchers

Service users are engaged to conduct part of the exercise, but take no part in planning and design. For example, users are employed as interviewers, but use an interview schedule designed by researchers

The views and experiences of the people who use a service are sought, but users are not otherwise involved at any stage in the process

Ways in which service users can be involved in research ⁴

Identifying topics for inquiry
Prioritising topics for inquiry
Commissioning research
Designing research
Managing research
Undertaking research
Analysing and interpreting information obtained
Writing up reports
Disseminating results
Evaluating the process

Recommendations to increase user involvement in research ⁵

Resources should be made available to help user groups to initiate and carry out research
Funding for projects needs to take into account costs of involving service users in the process
Sufficient resources need to be allowed to include the views of under-represented groups (ethnic minority groups, or people who do not communicate conventionally, for example)
All research which concerns service users should be disseminated in accessible ways

Research Ethics (standards of conduct)

Key issues are:

Respect for, and guarding of, the rights and wellbeing of participants
Obtaining informed consent - not just giving information, but ensuring that it is understood
The appropriate use and protection of data
(see www.dataprotection.gov.uk for details of the Data Protection Act 1998)
Access to information about the research and its findings
The benefit from the work to service users

Research that repeats work that has already been done, or does not contribute to existing knowledge in a way that is of benefit to service users, is never ethical.

It is good practice to seek advice on the ethics of your research. Other agencies can ask NHS Ethics Committees for advice.
In 2000 the Department of Health will be publishing 'A Research Governance Framework for Health and Social Care', setting out the standards to which all such research must be conducted. Standards relating to ethics, science, information, health and safety, and finance have been consulted on. (www.doh.gov.uk/research/index.htm)

Standards in the Draft Framework include:

All research which involves patients, users or carers, their material or data is referred to independent ethical review to safeguard the dignity, rights, safety and wellbeing of the participants
Research is pursued with the active involvement of service users and carers, including, where appropriate, those from groups such as black and ethnic communities

Participatory Research

" Research is part of the process of people being involved, gaining their own information, developing their own agendas, taking part in decision making and changing the structures they work in."6

Traditionally, research has been carried out by 'expert' researchers on passive research subjects. The researcher and their sponsors determine the research question, the methods by which information will be gathered, interpret the results, and make recommendations based upon that analysis. Since research cannot be value-free this gives them considerable power. Research subjects who cannot control the interpretation and uses of the information they provide can feel exploited and abused.⁷

Participatory research changes the power balance. 'Research subjects' become co- inquirers, taking part in defining priorities for research, formulating the research question, devising approaches to collecting information, gathering information and interpreting it. The professional researcher becomes a facilitator and consultant rather than director of the process. As with any research, the validity of the findings rests upon the assumptions made and approaches used being made explicit, so that the reader can decide whether the conclusions can be justified from the data.

This approach has a number of consequences:

The agenda for research and research question will be subject to negotiation, and may not follow the precise path envisaged
The pace at which the research proceeds will be dictated by participants
There may be a number of interpretations of findings. But if it is accepted that there is no single 'truth' then the resulting broad picture of meanings will be more consistent with the real life situation
Research partners may need supporters, people who will react, not initiate
User-researchers will need to be trained and paid for their work
There will be an expectation that findings will lead to action

Action research

Describes a dynamic approach where the understanding acquired is used to inform future action. Such an approach is appropriate when examining health and social care services because its aims are:

Change	to make improvements
Development	of policy, of a service, of professional practice, of the individuals involved
Involvement	of the subjects of the inquiry (service users and service providers) in the research process and in developing solutions to problems identified.

For example, the inquiry might identify areas where services are not performing optimally, gaps in services or needs that are not being met. Participants in the process plan change and then evaluate its effect. The process continues over time, and is usually represented as a spiral:

These are the steps:

group meets to decide focus for inquiry and how to gather information
information gathered
group analyse data and reconsider original focus in view of findings
group collect further information to test initial findings and to meet adjustments to focus
further analysis and reflection
plan for change
implement change
repeat process to evaluate change and establish new focus

Advisory Groups

A group of actual or potential service users to advise a project can:

Examine and challenge assumptions ⁸
Keep the priorities and concerns of the group to the forefront
Take part in preparing information about the project so that those who participate do so with informed consent
Suggest questions for interview schedules and questionnaires
Advise on interpretation of findings
Present findings verbally
Prepare an accessible summary
Ensure dissemination of findings

The research may be improved in this way, but control remains with the researchers. A good starting point for gaining experience, however.

In two research exercises conducted by mental health service users it was found that:⁹

User-researchers were able to undertake and complete research projects when adequately trained, and supported by a trusted person
Users were able to acquire necessary skills within a realistic time scale
Research was 'de-mystified' for the users involved
Time-scales need to allow for periods of personal difficulty (although user-researchers experienced periods of considerable mental distress during one project, research proceeded successfully)

Examples of research involving people who use services

The Impact Project

This is a local action research project to look at the impact of service user involvement on decision-making at clinical and policy level within mental health services. The topic was identified and prioritised by service users, who are working on the study in partnership with managers, clinicians and researchers. The team have received training in action research method.

The aims are to:

Develop a transferable, flexible model for the evaluation of user involvement in decision-making
Measure current impact on decision making, as a baseline
Develop an action plan to obtain improvements in user influence on decision making

Hull & East Riding Community Health Trust The RED Unit Bridlington Hospital YO16 4QP
01262 607144

Wiltshire Social Services funded the Wiltshire and Swindon Users Network to research users' views of the local Independent Living Fund (enabling people to purchase social care directly). A group of users obtained training, including ways of analysing qualitative information, to enable them to design, conduct and write up and disseminate the research. ¹⁰

The Mental Health Foundation facilitated user involvement in research into people's strategies for living with mental distress. ¹¹

The National Centre for Social Research were appointed as consultants to the study to ensure that standards of quality that would make the research widely acceptable were met, and to provide training and supervision for user-researchers:

A user - researcher visited user groups to consult them and take advice on the relevance of the research
An advisory committee of representatives from mental health user groups was set up to oversee the research
Interviewers with experience of mental distress were appointed

Monitoring and standard setting

Monitoring

The root of the word is to remind or give warning.

Monitoring compares what is actually happening to an agreed standard of best practice. Monitoring needs to be ongoing to ensure that services reach and continue to meet that standard. The element of scrutiny /accountability means that a service cannot monitor itself.

A Prerequisite for Monitoring - Standard Setting

Monitoring and evaluating services and measuring performance depend on first defining criteria against which 'success' could be judged, or setting standards. Standards should be evidence-based where evidence of effectiveness is available, but they are also rooted in values.

In the past standards often reflected how well the agency performed, not how well services met the needs of service users. It is now widely recognised that standards should reflect the effect services have on the quality of life of the people who use them. You should involve service users and potential users in setting standards for services, or defining what a quality service looks like.

The most appropriate way of doing this is to use approaches that allow people to envisage their ideal situation, and what services to support such a situation would look like. Group work has advantages:

Interaction allows ideas to be shared, clarified and refined
Themes, topics and principles that emerge can be 'tested' on other groups,

and priorities identified

A consensus on what standards should include can be developed
Standards agreed are owned by service users themselves
Standards can be stated in a clear and user-friendly way that is free of organisational jargon

Standards will change over time, so they need to be revisited regularly.

An example of user-led monitoring

Monitoring Our Services Ourselves ^{12, 13}

Standards developed by mental health service users were produced and circulated as the East Yorkshire Charter. People who were involved in developing the charter formed a Monitoring Team to make sure that standards remained 'live'. They received training from the Health Trust, were paid for their work, and received support from the full-time project worker employed by the Trust to promote involvement of mental health service users.

The monitoring exercise was incorporated into the purchasers' own schedule so that work was not duplicated, and the validity of the monitoring recognised. The team decided to implement a rolling programme, monitoring one of the four localities within the Trust each year.

Team members developed the questions they wished to ask, offering a choice of approach - questionnaire, one-to-one interview or group discussion. Piloting the questions with a user group identified the need to define terms to ensure a common understanding. The team provided information to staff and service users prior to the monitoring exercise.

Answers to closed questions and free comments were included in a report to service managers and purchasers. The Team ensured that there were mechanisms to:

Respond to the issues raised
Feedback to service users, particularly those who had contributed
Share positive comments with staff

The first exercise led to an action plan for service improvement. The team produced a practical guide for others drawing on their experience (see references below).

" ..the Team feel they have made a significant impact demonstrating that people using services can provide valuable and qualitative feedback which workers may find it impossible to discover themselves."

Evaluation and outcomes

Evaluation

Definition:
Collecting information that allows an assessment of how well a programme, service or organisation does what it has set out to do, in terms of effectiveness, efficiency and social acceptability.¹ For example, in meeting the needs of service users, as they themselves define them.

An evaluation gives you a picture of how a policy, service or organisation is meeting its aims from a range of viewpoints. Views of staff, service managers, commissioners and other stakeholders are included, and weight is given to how people who use the service perceive it.

Evaluation is not an end in itself, but provides a baseline from which to implement and evaluate change. There are advantages to using people with experience of using the service to obtain the views of other service users, as well as taking part in planning and managing the process.

See 'Peer interviewing'

A Prerequisite for Evaluation- Outcomes

Outcomes are "the impact, effect or consequence of a service or policy" ¹⁴

"evaluations should focus on user definitions of needs and outcomes, and how effectively services enable users to achieve their aspirations. Outcome measures must reflect the users' perceptions of quality of life, rather than 'improvements' in objective criteria defined by service providers or funders." ¹⁵

Effectiveness of a service has often been evaluated in terms of things that could be objectively measured - the number of people receiving a service, for example, rather than how well service users' needs are met. But subjective or value-based assessment is also crucial when assessing the efficiency, effectiveness and appropriateness of services.

When users judge the success of health or social care services, how a service is delivered (in a way that respects the person as an individual, for example), is as important as the effect of that service. Thus there are a number of considerations when evaluating a service, ranging from the aims of a service or intervention, to the impact of the service or intervention on the quality of life of the person receiving it, as illustrated below.

Overall Aim	Service input -what is done	Service output (quantitative measurements)	Outcome from process -the way services are delivered (qualitative measurement)	Outcome for service user -end result for service user as they perceive it
For Mr Smith to continue to live at home (supporting everyday functioning)	e.g. -Information/advice -Aids/adaptations -Care management -Home care	e.g. -X Aids provided -Y hours Home care provided -Mr Smith able to prepare simple meals	Balance of independence and support achieved	'I feel confident and safe to remain at home'

Derived from ^{15, 16}

Outcomes for Social Care (OSCA)

In 1997 the Department of Health commissioned a major new research initiative on outcomes of social care for adults. The purpose of the initiative is to deliver research evidence which can inform the planning, commissioning and delivery of services which are:

Valued by users and carers
Cost effective
Promote better outcomes for service users and carers.

The research programme consists of 13 studies, organised around different community care client groups, although the majority focus on services for older people. The key aims are to:

Develop practical tools and systems for collecting information based on user defined outcomes
Evaluate the effectiveness of social care services from a user perspective.
Develop an evidence base of systematic reviews of 'what works' similar to those developed in health services.
Consider the costs and benefits of different social care interventions from a user perspective.

Many of these studies are now approaching completion and the Department of Health plans to publish an overview of their methods and findings in December 2001. In the meantime, further information about the research programme including details on how some of the tools which have been developed can be obtained - See 'Other Resources'

An example of a user-led evaluation

A user-run information and support organisation asked Anglia Polytechnic University to evaluate its activities.⁸ The university suggested that the organisation carried out the evaluation themselves, gaining skills for the future. Funding was obtained to appoint a research advisor and pay the user-researchers for 10 months work. Members of the group took part in appointing the research advisor. The group formed a research committee to work on the project. They:

Identified what they wanted to evaluate
Decided on a method
Designed and piloted a questionnaire schedule
Used the schedule to obtain the views of half the members of the organisation, as well as past and potential members and professionals in contact with the organisation
Presented the findings to the organisation's management committee
Wrote up a summary report

See King's Fund study of service evaluations led by disabled people

Involving users in staff selection and training

Service Users as trainers and consultants

Service users have experience of services and expertise on their own situation. All service users who, for example, sit on committees, take part in writing service specifications or in inspection and monitoring, are therefore consultants, and should be treated and paid as such. Serious involvement in any case means drawing up an agreement so that everyone is clear about their role and what is expected of them.

There is evidence that involving service users in staff training improves the way people experience services. Service users can offer disability awareness and deaf awareness training, for example, as well as taking part in developing programmes for staff induction, training and development.

A number of groups and individuals with direct experience of services are providing a professional service to participate in training, as well as offering training to other users. For example, the United Kingdom Advocacy Network in the mental health field.

Bringing in a token, unsupported service user to a training session is not ideal, although practitioners do value hearing first-hand experience. One health service manager described listening to an audiotape of patients' experiences as being "slapped around the face for half-an-hour with the truth". ¹⁷

Involvement in Staff Selection

Again, most experience to date comes from the fields of mental health and learning difficulties. Few areas to date involve service users throughout the process, from drawing up the job description and post advertisement to equal weighting of interview scores. Many only involve service users in a preliminary informal interview or meeting and take their views into account in the formal process. While service users may take part in selecting basic grade care staff few are involved in managerial appointments!

Research into user involvement in staff recruitment in the field of learning disability found that¹⁸

Concern amongst service providers that people with learning difficulties would not understand confidentiality or equal opportunities practice were unfounded when adequate training, practice and support were given.

Factors found to promote effective involvement in the recruitment process were commitment of staff, trade unions and personnel departments, and involving people from the beginning of the process

Those involved in the recruitment process believed that involving service users meant that more suitable candidates were appointed. For example, information about attitudes to working with people with learning difficulties could be gained.

The Community Health Trust in East Yorkshire involve mental health service users in appointing mental health nursing staff, and have produced guidelines for interviewers and notes for candidates to explain the role of the user - interviewer.

User- interviewers receive the same training as Trust staff . They sign a contract that commits them to confidentiality and working within Trust policies, and are paid expenses. They are involved on an equal basis with Trust staff in planning the interview and scoring the candidates' responses to pre-determined questions.

Service Users can also be involved in selecting service providers given similar training and support.