Part 2c - Surveys, interviews and other ways of gaining information

Surveys, Interviews and other ways of gaining information

Practitioners and agencies need information

To learn from the ways service users experience services
To develop services that meet the needs and expectations of the people who use them
To know how they are performing

Surveys

Surveys are a way of gathering information using a questionnaire or an interview schedule.

"Questionnaires can be perceived as intrusive and interrogatory by users, and less in their interests than in that of the service provider"26

Surveys may be distributed by post (or increasingly, via a computer) or within a service setting. They may be completed by the individual, or by someone who records the participant's response on their behalf.

To provide reliable information, a survey needs to reach a representative sample of all people in the study group (the public, users of specific services, residents of a particular area) and obtain a high response rate.

How many is a 'representative sample'?

There are statistical formulae for working out how many people you need to survey, and how many forms have to be returned, to give confidence that the responses you get from your sample population (say a random sample of one in ten of the people who use day centres) can be safely generalised to the population you are interested in (the total number of people who use day centres). If you need a statistically representative sample it is worth taking expert advice about sample size for a particular purpose. (See, for example, www.doh.gov.uk/pssuser/index.htm for guidance on sampling for the Personal Social Services user satisfaction survey).

If your sample is small (as may be the case with some groups of health and social care service users) you will get information that is indicative. You just need to be careful about what you claim from the data.

The Office for National Statistics report that a written questionnaire survey will achieve a 30% - 40% response rate. This can be increased to between 60% - 75% by sending two reminders. The majority of forms are returned within the first 10 days.²⁷

Questionnaire surveys are good for:

Counting - how many use a service, hold a particular view, and so on
Obtaining factual information
Obtaining an overview from which to identify priorities for more detailed study

Advantages:

Can reach large numbers of people
Provides baseline data
Responses confidential

Consider:

You need to be able to identify the target group from which to sample
Dependent on literacy and shared cultural concepts
Difficult to design to get the appropriate questions answered.
Don't underestimate the difficulty of designing a questionnaire that provides good information. Consider using forms that have been tested elsewhere
Needs piloting with a sub-sample of the identified group
Can be unsatisfactory for respondents:
- Questions may not seem relevant to respondent
- There is little opportunity for explanation or interaction
- Results are not owned by participants - people have commented that they feel like objects of research, not a person
- It may be unclear to them how the exercise will lead to action
Needs considerable administration time - can be costly
Open questions can be more difficult to analyse
With low response rates, there are concerns about how representative results are, even with a carefully chosen sample
You may learn how many think or do particular things, but not why - not the most appropriate method for investigation or exploration of issues. You may need to follow up with interviews or group discussions. See Interviews and Focus group discussions

Making the most of surveys:

Improve by using discussion groups or semi-structured interviews with the target population to develop the survey instrument - to identify priorities, the preferred order of questions, check that appropriate language is used, establish shared understanding of meanings, for example
Use the language the target group would use, for example, use common rather than medical health terms or social work jargon
Providing information on what to expect from a service will give respondents a yardstick to judge the service by. See Monitoring & Standard setting
Increase inclusiveness by offering alternatives to a written self-completion survey form where possible (assistance with completion, telephone or face to face interview)
Use large print, leave space to write comments, ask for personal information at the end²⁸
Make sure confidentiality is understood (numbered forms make people think you can identify them)
People may need a polite reminder and a second form

A survey of people over 70 was redesigned after comments from user panels of older people. It achieved a 70% response rate. The initiator attributed the good response rate to suggestions made by the older people which made the survey more acceptable and straightforward.

Satisfaction Surveys

"Factors peculiar to users will weight the outcome. Their demoralisation, gratitude for support, low expectations, unclear expectations, vulnerability, absence of other choices, over-dependence on professional expertise....."29

Surveys are often used to measure 'satisfaction' - for which purpose they are least useful. Satisfaction surveys often report stunningly high levels of satisfaction - an average of 72% of respondents to Joint Review postal questionnaires report services as 'excellent' or 'good'. However, response rates are low, so validity is in question. '80% satisfied' may look good as a headline, but you have learned very little. Could you have spent that time and money to better purpose?

What is required of an instrument to collect user views is that it:

Produces an accurate reflection of experiences, views and opinions
Produces information that is useful, in that it addresses issues that are important to users
Can result in action for change or to improve quality³⁰

There are concerns about how satisfaction surveys meet this requirement:

The term 'satisfaction' is not defined, and there is little clarity about what is meant
A service consists of many component parts. What is being evaluated? The process? Outcome? Degree to which needs are met?
The concept of satisfaction relates to prior expectations, but service users may not have enough information to form expectations before they experience a service
Service users are being asked to evaluate or judge a service with no standards to measure against
Expectations will change over time, so measurement at different points is not measuring the same thing
Low expectation appears to produce high satisfaction ratings
The USA experience (where people are paying directly for services so can choose between providers, see Consumer or Citizen?)is that satisfaction is generally rated highly, but does not translate into behaviour. Very small differences in satisfaction rating, even at the top of a scale, related to large differences in the numbers of people changing their health care provider

Most people who require services want them to address quite specific needs. It is these that you need to identify and focus on, asking specific questions related to quality of life and achievable change, rather than attempting to address a general 'satisfaction with services'.

Ongoing quality monitoring, using standards developed with service users, are more productive for gauging perceptions and experience of services than are periodic surveys.

See 'Monitoring and Standard setting'

Hampshire Social Services Department set out to reconcile the limitations of user satisfaction surveys with government and management demands for systematic ways of collecting feedback from service users.

They worked with a local user group to establish the principles on which user surveys should be based. They should be:

integrated into normal departmental processes
developed in partnership with user organisations - methods and materials
distributed by front line staff
safeguarded for confidentiality
returned to and analysed at a central point
findings fed back to managers, front-line staff and service users

The routes chosen to obtain feedback from service users were through:

Reception and Assessment Teams at initial contact with the department
(a quarterly survey of the first 30 contacts on a given date provided a "small manageable
exercise at local level, but resulting in over 1500 people being surveyed over a year")

The care plan review system, for care management
(questionnaire handed out to everyone at review)

Provider services, for service delivery

Written surveys are used to ensure anonymity. Large print and tape versions are available.

The surveys have highlighted areas where change is needed to improve the service.
For example, although 79% respondents knew what action was being taken after assessment, less than 50% knew who would take action or when this would happen.
Staff hear about positive comments, and anyone reporting a negative experience is invited to provide contact details so their concerns can be followed up.

The department views this approach as 'one tool in their toolkit'. They can compensate for its limitations by supplementing it with in-depth interviews and other exploratory methods, but they are beginning to collect core information in a systematic way as part of existing processes.

Performance Management Unit, Hampshire Social Services

Panels

Panels range from large numbers of people contributing via postal surveys to smaller groups of people who meet to contribute their views. The key feature of a panel is that it continues to exist over a period of time. (Whereas focus groups meet on only one occasion).

Citizen panels are a random sample of the population large enough to be representative of the population they are drawn from (usually 1000 or over people).

Panels are recruited to provide a pool of people from whom to:

Obtain information and feedback over a period of time
Draw smaller samples with particular characteristics, for example, people within a certain age range
Recruit for in-depth interviews, meetings or group discussions

Most panels limit the length of time a panel member can serve, and replace them after that time. Others 'refresh' the panel by recruiting a certain percentage of new members each year, allowing new opinions to be introduced and new members to be absorbed. The aim is to avoid panel members becoming better informed than the average citizen by virtue of their participation.

The Somerset Health Panels

These were set up in 1993 to 'build a consensus on the values to be used to guide health resource allocation decisions'.

Eight panels of twelve people meet to discuss an issue posed by the Health Authority . This number was considered to be sufficient in number and coverage of the geographical area to give confidence that the wider public view would be represented. Members were chosen to mirror the social and demographic profile of the area. Panel members are given information on which to base their deliberation. Both participants and managers regard the process favourably.³¹

Panel members will be influenced by their own, direct experience, and by indirect experience such as that of family, friends and neighbours. They will also be influenced by seasonal factors (high winter demand for GP appointments) and media coverage of local and national issues and events.

Panel recruitment and response rates

Experience indicates that 10,000 postal recruitment questionnaires will result in 1,500-2,500 responses. In order to match the demographic profile additional efforts need to be made to recruit certain sectors of the population. For example, young people are otherwise under-represented. Media appeals are sometimes used to recruit panel members, and agencies are increasingly using their website to recruit and contact panel members.

See 'Electronic information'

Nugent suggests that a well-recruited and maintained panel should achieve a 65-85% response rate when surveyed.³²
Vittles found response rates of 70%+ from face interviews, 60%+ from telephone interviews and 20%+ from postal surveys.³³ North Wiltshire's 'The People's Voice' panel achieved a 92% return rate to its first three postal questionnaires. This was in the context of a high profile for community consultation activities generally, including a newsletter to feed back to panel members information gained from each questionnaire and its impact on Council actions.

Nugent suggests that citizen panels have been used with two main objectives in mind:

Consultation about future policies and services
To measure performance and satisfaction with existing services

He believes that the usual periodic sampling is poor for the latter. He suggests continuous sampling, of one twelfth of the panel each month, with results aggregated then averaged. This would give a better overview of cycles in satisfaction and the effects of media coverage of issues.

Smaller panels than those generally used by Councils and Health Authorities could provide useful, if not statistically representative feedback. Some might be derived from registers of people with a particular interest in common, for example, carers, people using a particular service on a long-term basis such as those with diabetes. The group could be provided with information and attend meetings or discussion groups so that informed views can be obtained. Smaller standing panels are cheaper to run and can provide more immediate feedback. Electronic panels are increasingly becoming a possibility.

Panels are good for:

Identifying particularly good or poor perceptions of a service for further research
Seeing how views change over time
Can be used to test the effect of external events e.g. media coverage

Advantages:

Can be recruited to represent the characteristics of the population or group so that findings (if response rates are adequate) can be generalised to the population as a whole
Means of collecting data from a fairly large sample of the population
Various means of collecting information can be employed, and initial data can be explored in greater depth
Performance monitoring requires that the same question be asked of the same people over an extended period
Organisation benefits from appearance of wanting to involve 'ordinary' people
Panel can be pre-informed to obtain more considered views

Consider:

May be difficult to achieve a reflection of the characteristics of the whole population or group
Needs considerable infrastructure to recruit and maintain a large panel and manage data
For ongoing performance monitoring, an increase in satisfaction may be an effect of being on the panel
Panel members need to be well motivated to continue involvement over time
Loss of panel members through natural wastage needs to be allowed for

Timely acknowledgement and feedback,to include publicising how the information gained has been used to improve services, can help maintain interest. You might do this through the local press, or through a newsletter to panel members.

The Government has commissioned the 'People's Panel', a representative sample of 5,000 people who are being asked their views on public services and how they can be improved. Any publicly funded body can have access to the data collected, or use the panel themselves.

www.cabinet-office.gov.uk/servicefirst

The Fife User Panels³⁴ were established to ensure that older people could contribute directly rather than through voluntary organisations, to influence the planning of health and social care services. The project was supported by two half-time workers with secretarial and consultancy support.

Eight panels were established around the region " to enable frail older people to discuss with each other their experiences of growing older and of using health and social care services, and to use the outcomes of such discussions to influence the provision of services to become more sensitive to older people's needs".
Seven of the panels consisted of frail older people while the eighth was matched in age, but consisted of people who were not dependent on services at that point.

Members were nominated through health and social care workers, churches and the local Age Concern group, but the 'snowball effect' (people recruiting others from their own networks) increased as the panels developed. Ongoing recruitment was needed to replace those who became ill or died, but the loss of members was not found to have a negative effect on the panels, and new members were successfully introduced.

The age range was 67 to 93 years with an average age of 82. Housebound older people and those with dementia were not included. The panels met monthly, with transport provided to accessible and comfortable venues. Reports were produced from their discussions.

Evaluation of the project was built in, to address three questions;

Is it possible to identify frail older people who are, or could be, users of health and social care services, and to involve them in regular panel meetings designed to encourage them to articulate their views about such services?
Do participants experience benefits from being involved in such panels?
Do the panels exert any influence over service plans and provision?

To address the second question peer interviewers helped design an interview schedule and conducted interviews with panel members.
To examine the impact of the panels on services three issues were selected as a focus for document analysis and key informant interviews. (These were domiciliary care, hospital discharge, and information and communication).

The panels were described as an "effective but resource intensive" way of enabling older people to express their views and welcomed by service providers as an innovative way of engaging with older people.

There was some concern about how representative the panel's views were. Individuals were recruited not as representatives, but to contribute their own views and experiences.
The report concludes that those who participated could be considered typical of older service users. (See 'Representativeness & accountability') Credibility was increased when agencies were more aware of the process that generated panel reports - how topics were generated and explored, and had a quantitative idea of members' contributions.

There was also some tension between agencies' desire to use the panels for their own purposes and the aim of the panels; to enable older people themselves to identify and prioritise issues, and introduce new perspectives. As panels have progressed, the variety of issues discussed, and panel members' skills in responding when agencies needed quick feedback, increased.

The distance from statutory services enabled a free expression of views and maintained a user-led rather than bureaucratic feel. Participants felt valued as the ability to contribute, that many felt had been lost with age, was restored.

Analysis of how panels functioned to explore user views and to influence services revealed:

Discussions were largely initiated by members recounting and reflecting on their own experiences and responding to service changes

Discussion was facilitated and supported by project workers in a variety of ways:
- Direct questions
- Feeding in information members might not be aware of
- Using exercises or structures that facilitated discussion
- Making suggestions about potential ways forward

As panels started to develop their own agenda on an issue, they sought direct contact with service providers. This dialogue prompted specific proposals to be developed. Having established this contact and made proposals, members expected feedback.

Interviews

Structured Interviews

Data collection might be a more accurate description of a structured interview. The purpose is to gain specific and pre-determined information. The same questions are asked in a set order by someone who records the responses, usually in a tick-box type format. The interviewer does not have the same scope for exploration as in a less structured interview.

A structured survey instrument does not have to be so 'user friendly' as a questionnaire, so can be set up in a way that makes it easier to analyse.

Good for:

Obtaining concrete information (numbers, yes/no, service use etc)

Advantages:

Quicker and cheaper than less formal interviews
Requires less skill to administer
Quantitative data can be obtained in a form that is relatively easy to analyse
Allows for explanation of question if necessary, and checking of understanding

Consider:

Values, feelings and reasons for views can't be explored
Limited opportunity for dialogue might make experience unsatisfactory for participant
The answers you get depend on the questions you ask - easy to influence responses with a prescriptive framework

Semi-structured interviews

General areas of enquiry are determined, but there is scope for the interviewer to explore the respondent's feelings and perceptions, and ask further questions in response to the information given.

(Unstructured interviews are described, but it could be argued that the interviewer always has at least a mental agenda).

Good for:

Obtaining rich descriptive information
Exploring views, and why people hold them

Advantages:

Can be used for more sensitive topics and with people who don't function well in groups
Can enrich data from more structured methods

Consider:

Demands more skill and sensitivity of the interviewer
Interviewer/interviewee relationship can influence responses
Interviews time-consuming to conduct, record and analyse
Needs skills in analysing information gained
May need translation or signing if a peer interviewer is not possible
Some articulacy in speech or signing required of the interviewee
Number of interviews likely to be possible may limit how far you can generalise the information obtained
Needs quality control where more than one interviewer taking part to ensure consistency
Can raise uncomfortable issues for the respondent, and Interviewers may need emotional debrief

Peer Interviewing

Using people to conduct interviews who share characteristics of those being interviewed.
For example, age, sex, experience of service use, class, language or ethnic origin.
Peer interviewing has been widely used in work with young people and in user-led service monitoring in mental health. There is potential for use in other service areas.

The Sainsbury Centre for Mental Health suggests a number of advantages of user interviewers, supported by their own and others' research.³⁵

User interviewers:

Have a rapport with interviewee through their own experience
Understand difficulties with jargon
Are not discomfited by people hearing voices or having other difficulties during the interview
Are unbiased and independent
Are more likely to be told what users really think about their situation than 'professional' researchers or staff
Help users bring out views that may be different to those of professionals
Bring out issues that providers may not have been dealing with or seen as a priority
Help interviewees to think about and explain their problems, use of services and treatments in their own way

Peer interviewing was used in an evaluation of mental health users' and carers' experiences of involvement in the planning of their care.³⁶

One third of the 78 people interviewed preferred a researcher to a service user, usually for reasons of confidentiality. The authors report that peer interviewers did not have difficulty in containing information gained during interviews. They were successful in developing a rapport with interviewees.

Another study ³⁷describes the development of a model of user-focused monitoring in mental health using peer interviewers.

12 volunteers from user groups and service settings trained as interviewers.

A semi-structured interview was developed through role play with user-interviewers as part of their training. This was used with a random sample of 58 mental health service users, contacted through their key worker. The authors note that the sample was small, but these were vulnerable users with severe and enduring mental health problems (in the top tier of the Care Pathways Approach) who would not have been contacted normally.

The interviews were used to find users' knowledge about services, views on, and satisfaction with, the service they received, and their perception of gaps in the service.

The study found that:

User-interviewers were able to elicit clear responses from users with severe psychiatric and social disability

Users were more likely to participate if their key worker was enthusiastic

More work on engaging key workers was needed before the project so that user-focused monitoring would not be seen as a threat

Researchers felt that criticism of the interview as a method that does not encourage critical reflection, (Elbeck and Fecteau 1990 suggest focus groups are more effective), was not supported by the thoughtful and varied responses obtained

'User interviewers elicit more open and honest responses than professionals do'. They created a relaxed atmosphere in which users felt able to be more open than they might have been with a professional who could influence the service they received

Seven of the final eight user-interviewers were felt to have conducted interviews in a consistent manner

See 'An example of user-led monitoring'

Other ways of obtaining information

Roadshows

The 'What Matters to You' roadshow toured the North Wiltshire Council area, providing daytime and evening opportunities for people to write what mattered to them on flipcharts.

In a fortnight over 8,000 people visited the roadshow and more than 5,000 comments were obtained. Staff and elected members involved in the roadshow could see and hear what local people were saying. The issues identified were further explored in 'Visioning Days' in each of the five community areas in the authority to establish future agendas.

Observation

Watching and recording what happens.

Advantages:

Provides feedback on actions or behaviours that may not be consciously recognised or articulated, for example, staff/service user relationships
Allows comparison between expressed knowledge or intent and actual practice
Where individuals do not communicate conventionally, allows preferences to be established through observation of behaviour and emotions

See 'Engaging with people who have a learning difficulty' and 'Engaging with people with dementia'

Consider:

Observer can influence behaviour
Observation not always ethically acceptable
Dependent on interpretation of observer where interpretation cannot be checked with people observed

Story telling, life histories, diaries

Ways of obtaining descriptive information of richness and depth, that is not dependent on literacy (you can use audio diaries) and can be suitable for use with people with learning difficulties or memory loss. Although these approaches have the benefit of informality, the same rigorous standards of information giving to obtain informed consent apply. Skill is needed in using these approaches and in analysing the information obtained.

Stories provide a hypothetical situation that can be used to prompt discussion and draw out values and priorities. Being able to decide how someone else might act in a situation can be less threatening than direct approaches.

Life stories (or biographical interviews in research jargon!) allow people to share and reflect on their experiences in their own words and to identify what is or has been important or critical to them. It can be an affirming experience for people who feel devalued ³⁸and may not have the confidence to participate in other ways. Visual means can be used to prompt reflection (time lines, critical incident mapping, body maps, photos, music, poems and other reminiscence material).

Diaries provide a means of collecting information about activities, events, (giving some information on frequency), experiences and people's reflections on them. They can be more private than speaking directly to an interviewer. Diaries can reveal differences in what people think or say they do, and what they actually do.

Using existing information

"There are many ways of using informal mechanisms to collect information about the needs and views of... people. Too little use is made of complaints, helplines, the observations of front line staff and development workers, and opportunities to collect information as well as give it, such as health information sessions. Such information is often not valued because it is seen as unrepresentative - but if recorded, collated and analysed for patterns it can supplement and enrich information collected by more formal means." ³⁹

Front-line staff hold large amounts of information, including local information and knowledge of unmet need, appropriateness of services and gaps in provision, but there is seldom any systematic means of capturing this
Councillors can be a conduit for information exchange. Taking part in consultations can help them to know what local people think and to represent them more effectively.⁴⁰ It is suggested councillors may have an enhanced role in working with Primary Care Groups
Community Health Councils are aware of issues causing concern to people using health services, and sometimes carry out specific surveys. For example, East Yorkshire Community Health Council surveyed 713 people using hospital outpatient departments to obtain views on healthcare provision in the area ⁴¹
After a patient on the Stroke Unit kept a diary of their hospital experience the Occupational Therapy department of Ulster Community and Hospital Trust introduced a "Communications Book" for patients, relatives and staff to make comments and ask questions. It is "proving a very useful way of sharing opinions and experiences as part of the continuous improvement programme."
Information leaflets could include a tear-off slip for comments

Using information collected by other people

Information is collected by central government, by other agencies and by other departments. Gaining information is costly, so make sure you know what is already available before collecting your own. A joint approach to information gathering is cost-effective, avoids the same individuals being approached by different people, and gives an impression of 'joined-up' services.

Reorientating complaints procedures

Lose that word! It has only negative connotations, and people think that something has to go seriously wrong before they use a 'complaints procedure'. More informal comment or feedback mechanisms allow anonymous positive comments, descriptions of experiences, and constructive criticism requested to benefit others. Much more information on how a service is performing could be obtained, and could act as an early warning system to identify problems rather than waiting for things to go wrong.

A study⁴² with people with learning disabilities and their families identified barriers to the use of complaints procedures

'Complaint' had negative connotations such as 'moaning'

people did not know the procedure to complain

concerns about the consequences of complaining - fear of retribution, not being able to control the process or predict the outcome

perceived messages that service providers discouraged responses, poor past experiences of other forms of participation, implicit threats of service withdrawal

concern about complaining about one aspect of an otherwise valued service

reluctance to be critical of front-line staff

difficulty in questioning services in the independent sector

worries about taking time when others seen to be in greater need

People may be getting mixed messages from staff. The study found that few front-line staff had received any specific training about complaints procedures, but were aware of dissatisfaction or conflict that was not followed up.

Staff:

Equated 'complaints' with 'disputes' and failed to distinguish between complaints and 'gripes'
Were reluctant to be seen to 'push' people into complaining
Believed it was only worth complaining about things that could (in their judgement) be changed