Research Digest

Issue Number 1

•  This study implemented and evaluated a system of goal attainment scaling (GAS) in The Manchester Learning Disability Partnership.

•  It was possible for staff to set adequate goals and to construct adequate scales to measure the outcome of work agreed.  However, the quality was variable, and many staff found it difficult to make their work fit the measurement method.

•  There was resistance to the use of the method by some staff, and there was little evidence of the negotiation of goals with users and their allies. These difficulties are sufficient to cast doubt on the viability of the method.

• Staff report that the structure helps them to improve the clarity and focus of their work and there is now a ‘discourse of outcomes’ in the service that was lacking before.

•  Substantial change was produced in the majority of cases.  However flaws in the measurement method means that comparisons made using GAS scales could be confounded.

 Measuring and monitoring outcomes in community learning disability teams

The Manchester Learning Disability Partnership has been researching ways of monitoring and measuring the outcomes of work done by staff in community learning disability teams.  This has been an area where little work has been done in the past, in comparison to the ‘hard services’ that provide a home, or day time care and support.  Routine measurement of outcomes in field settings presents a variety of challenges, including:-

1) The heterogeneity of service users, their needs, and therefore the desirable outcomes - even within specific client groups such as ‘people with learning disabilities’.
2) The historical emphasis on activity, or other process measures in such services.
3) The insensitivity of many global measures of quality of life or adaptive functioning to positive improvement in key domains.
4) Problems of validity and reliability, especially outside a research context.
5) Professional resistance and potential disagreement regarding the appropriate specification of outcomes.

We identified two types of measurement strategy:  a) the use of a broad brush measure of functioning or quality of life, and b) an approach based on individualised goals.  Rejecting the former in this context as too time consuming and insensitive to change in key areas, we used a version of the second: Goal Attainment Scaling (GAS).  GAS is based on goals set with individual clients.  It employs a method common to all goals to create a scale for possible outcomes.  This is done in advance, at goal setting.  This yields follow up scores that can, in principle, be aggregated and used to monitor and evaluate service performance in achieving outcomes with users of the service.
GAS was implemented across the four large community learning disability teams of the Manchester Partnership.

Findings

• The project encountered considerable implementation problems.  Only about 30 per cent of the expected number of goals were set and scaled over the course of the project, and this reflected widespread, although not universal resistance from staff.
• For all the disciplines in the service, there were GAS enthusiasts as well as critics.  Those that found it helpful valued its help in increasing the clarity of their work.
• A combination of questionnaires and interviews indicated that staff were satisfied with the briefing, training and guidance materials provided.  Their dissatisfaction reflected what were perceived as a cumbersome method, that increased paperwork, and which did not fit some types of work very well, particularly care management and psychological work.  Speech and language therapists and occupational therapists were broadly receptive to the approach, with nurses’ reaction somewhere in the middle.
• Goals set and the scales constructed were assessed for their quality by a panel of raters.  On most dimensions a majority of goals were rated as being of at least acceptable quality to at least two of the raters:  it was concluded that the majority of the goals and scales were at least usable for outcome measurement, if not always of a high standard.
• The reliance on the managerial system for the monitoring and progress chasing of the GAS implementation appears to have led to some backlash, although there was little dissent from the principle of measuring outcomes and improving the accountability to those that rely on the service.
• The use of GAS did appear to encourage, over time, the adoption of an ‘outcomes discourse’ in the service, at all levels.  Changing the service culture from a process/activity focus to a more outcome focussed one takes time:  despite the resistance encountered, this project has contributed to a change of emphasis that continues after the end of the original project.  When alternatives to GAS were considered, several staff groups were reluctant to abandon GAS!
• Because the GAS approach was chosen, then implemented and evaluated within a funded research project, there was little scope for involvement of staff in designing a method that best fitted this context. This may have reduced the potential ‘ownership’ of the approach by those who were expected to use it.  However, this would apply for any nationally implemented system of outcome measurement.
• The project also provided the service with a clearer overall picture of the kinds of goals the community team staff were working towards.
• GAS has been promoted as enabling user participation in deciding what the service will do and evaluating its effectiveness.  Despite encouragement, there was little evidence of any attempt to use the GAS materials in negotiation of goals with users and carers, and even less in the evaluation of outcomes.  Staff attributed this to levels of learning disability, but the lack of participation extended to families and support staff.
• Staff resistance meant that it was not possible to systematically research the reliability and validity of the goals set.  When the method was discussed with carers they were supportive of the general approach, but it was not possible to research their perceptions, or those of users, in a systematic way.

An analysis of goals that had been followed up indicated that the mean level of goal attainment was a little over the expected level.  Baseline levels were near the ‘floor’ or lowest possible score.  Average change was around two points on the five point scale.  This would appear to indicate that the community teams were, on average, achieving what they set out to attain with their clients.  More change was achieved on goals that involved changes external to the person than on those that were about individual changes (e.g. in behaviour or health status).
 

Will GAS do the job?
Despite the attractiveness of GAS as a common measure for all combinations of types of goal, worker, person, situation, the research team suggest caution in the adoption of GAS in the future.
 

• Because scales are anchored at the expected level of attainment (the mid point), comparisons of attainment (e.g. between workers, settings, or in terms of user characteristics) could be confounded by the different expectations in each case.
• The data sets produced are also complex, requiring types of statistical analyses that could be invalidated by the properties of the data.
• Even if the measurement method was adequate, a service would have to actually implement GAS to an appropriate standard, maintain this implementation, and manage the data arising from it to provide timely and meaningful reports to staff, management, and commissioners.  This requires a considerable developmental and managerial capacity.
• For reporting on the service as a whole, an approach focusing on key performance indicators (e.g. number of out of district placements, number of deaths from respiratory diseases, number of holidays away from the district, number of people receiving direct payments) might give more relevant information.

About the study:

A variety of methods were used for this study.

• Information on the implementation of Goal Attainment Scaling was obtained through a combination of questionnaires and interviews with staff at three different points in time.  Researcher diaries were also kept which captured the less formal feedback on the project from conversations, team meetings, the day to day issues in implementation and so on.  These information sources were analysed qualitatively to identify key themes and issues, and in the case of the questionnaires, quantitatively to establish the relative strength of different opinions among the staff teams.
• The overall number of goals set (in relation to referral rate) was used as a measure of the extent of implementation.
• Quality of GAS goals and scales was established through a study in which three raters made ratings on a number of different dimensions of good practice in goals setting and scale construction. Agreement statistics were calculated, and the pattern of agreements and disagreements was also reviewed.
• The GAS data itself was also analysed statistically to identify the pattern of outcomes obtained.

Back to Collaboration Home