Evaluation of the national mental health strategy

Approach to the evaluation

Page last updated: December 1997

Evaluating the outcomes of Australia's first National Mental Health Strategy presented a formidable challenge. Questions at the core of extensive worldwide research and debate – such as how services should be delivered, what works for whom, and under what conditions - underlie the national reform process. The committee was mindful that its evaluation needed to be focused, and identify the essential policy messages for the future.

The twelve policy areas for reform outlined by the National Mental Health Strategy provided the starting point for the evaluation. The approach taken by the committee sought to build upon the findings of its earlier report and conduct a selective review of those policy areas that would inform about the effectiveness of the Strategy more broadly.

Evaluation focus areas
Defining outcomes
Summary of method

Evaluation focus areas

Four areas were selected as evaluation focus areas on the basis that they are illustrative of the key elements of the national strategy and goals for mental health. The policy areas selected for review were:
  • promotion and prevention
  • linkages between mental health services and other sectors
  • service mix and
  • the rights of consumers and carers
These four areas cover 15 of the 38 objectives defined in the National Mental Health Strategy. Of the objectives not addressed specifically within the focus areas, the committee's earlier evaluation reported substantial progress to have been made in many of these by June 1995, and identifiable initiatives had commenced for the remainder.8 The outcomes of these latter initiatives will not become evident until after the conclusion of the National Mental Health Strategy.

To complement the detailed evaluation within each of the four focus areas, the committee also reviewed available data relevant to all national objectives. Appendix 3 presents a summary of the committee's assessment of progress against all 38 national objectives.
Top of page

Defining outcomes

The committee's brief for the evaluation required the outcomes of the Strategy to be measured against two criteria – effectiveness and appropriateness.

'Effectiveness' criteria relate to how well a particular program has met its stated aims and objectives. In most areas of public policy, this means assessing the difference made to society as a result of the policy – that is, whether it has made things better or worse. Extending this approach to the mental health field, the effectiveness of a national policy might be judged by indicators such as reduced prevalence rates of mental illness and disability, improved quality of life for mental health service consumers, increased employment rates, reduced suicide prevalence and so forth.

However, most of the objectives defined in the National Mental Health Strategy are concerned with the process and structure of reform. This emphasis, on how reform would be progressed rather what it achieved, reflects the stage of mental health reform in Australia at the time the Strategy was devised.

The committee considered that monitoring the extent to which these objectives have been met, as occurs in the National Mental Health Report, was a necessary element of the evaluation but not sufficient to build a picture of outcomes at the community and individual consumer levels. To achieve this, the objectives defined in the Strategy needed to be translated into terms which express the intended benefits for the community.

Similar interpretation of the concept of 'appropriateness' was required. Evaluating the appropriateness of a national reform program requires judgements to be made as to whether particular actions are 'right' or 'suitable'. In essence, this requires a decision as to whether the actions taken were the best options available to meet the stated objectives, as well as consideration of the positive and negative consequences of those actions.

Decisions about appropriateness must also consider the context in which a particular policy is implemented. For the National Mental Health Strategy, aspects to consider include the inadequate state of services at the commencement of the Strategy, the brief , five year timetable given to achieve the objectives, the relative newness of the Commonwealth-State partnership on mental health services and the lack of international precedents for a National Mental Health Policy.

Finally, appropriateness and effectiveness criteria are not necessarily in agreement. To take an example from the mental health field – relocating a stand alone inpatient unit to a poor quality general hospital would achieve the objective of mainstreaming, but may be deemed inappropriate if it is unacceptable to consumers and carers. Clearly, complex issues need to be examined when assessing whether a national policy is both appropriate and effective.

The committees' approach integrated effectiveness and appropriateness issues into a set of key 'outcome questions' that were used to drive the evaluation.

The focus areas and the key evaluation questions associated with each are listed in Table 2.
Top of page

Table 2: The four evaluation focus areas and key outcome questions

Table 2 is presented as text in this HTML version for accessibility reasons. It is presented as a table in the PDF version.

Evaluation focus area - Rights of consumers and carers

Evaluation aimed to establish: "Has the Strategy...
  • generated improved respect for the rights and dignity of people affected by mental illness?
  • strengthened the support available to carers?
  • enabled consumers and carers to participate in mental health service planning at the individual and policy levels?

Evaluation focus area - Service mix

Evaluation aimed to establish: "Has the Strategy...
  • produced a mix of services that:

    • promotes high standards and quality in service delivery?
    • meets the range of needs of people affected by mental illness?
    • leads to improved outcomes for consumers of those services?

Evaluation focus area - Linkages between mental health services with other sectors

Evaluation aimed to establish: "Has the Strategy...
  • led to improvements in access by people affected by mental illness to programs and services needed to participate more fully in community life?
  • improved access to employment and quality housing?

Evaluation focus area - Promotion and prevention

Evaluation aimed to establish: "Has the Strategy...
  • facilitated community understanding about mental illness and reduced stigma and discrimination directed at people affected by these illnesses?
  • reduced community prevalence rates of mental disorder?
  • reduced the impact of mental disorder through early intervention and promotion programs?

Research components

Outcome data that could inform the evaluation are not collected routinely by Australian mental health services, although, as indicated later, steps towards this have been initiated under the Strategy. As a result, little of the data gathered to monitor the National Mental Health Strategy directly addresses the key outcome questions.

The Steering Committee recognised that no single approach would address adequately the key outcome questions. Four supplementary research studies were therefore commissioned, each contributing a unique perspective to the evaluation. These comprised the building blocks for the current report (see Figure 1).
Top of page

Figure 1: Research elements used for the evaluation

Text equivalent below for Figure 1: Research elements used for the evaluation
Text version of Figure 1
This diagram has the evaluation in the centre, with the following four research elements providing input to it:
  • Area case studies - Review of impact of National Mental Health Strategy implementation in 4 local communities
  • Review of available data - Evaluation of available data relevant to the effectiveness of the National Mental Health Strategy
  • National stakeholder consultations - Consultation and survey of views of national peak bodies
  • International expert commentary - Review by US Centre for Mental Health Services of appropriateness of national mental health policy settings

Area case studies

These were 'in depth' studies of local populations, designed to assess the impact of services changes at the 'ground level'.

Previous reviews of mental health services have sampled the views of a wide range of stakeholder groups about the National Mental Health Strategy, but these have been drawn from disparate areas throughout Australia. The new element introduced through the Area Case Studies was to evaluate the appropriateness and effectiveness of the Strategy from the various perspectives of stakeholders living within the same local community.

Four sites were chosen from separate areas throughout Australia, each of which was served by an integrated area-based mental health service. The areas selected for the study represented 7% of the Australian population in 1997.

Specific criteria used to select the four study sites were designed to identify sites which approximated the type of service models promoted by the Strategy, rather than being typical of the 'average' service currently available. These criteria are summarised in Table 3.
Top of page
In each area, qualitative and quantitative data were gathered to examine the impact of service changes on consumers, carers, staff and external organisations. Data collection methods comprised a combination of consultations, one-to-one interviews and written surveys with key stakeholders. In each of the sites, consultations were conducted with:
  • management and staff of the local mental health service
  • consumers and consumer organisations
  • family members, carers and carer organisations
  • private medical practitioners
  • providers of accommodation and employment services
  • police and ambulance services and
  • other key informants.
In all, the Area Case Studies consulted over 400 individuals drawn from these varied backgrounds. In addition, approximately 900 people responded to the written survey.

The Canberra-based consulting group, Purdon Associates Pty Ltd, was contracted to conduct the Area Case Studies during the period April to August 1997.
Table 3: Criteria used for selection of sites in area case studies
Table 3 is presented as a list in this HTML version for accessibility reasons. It is presented as a table in the PDF version.

Criteria used for selection of sites in area case studies:
  • 'Area service network' - Services organised on the basis of a service network, serving a defined catchment population.
  • 'Comprehensive services' - Site has all basic elements of a comprehensive service in place, approximating the levels of service provision advocated in papers commissioned under the National Mental Health Strategy.
  • 'Average resource levels' - Sites were neither 'rich' nor 'poor' relative to Australian resource levels.
  • 'Availability of data' - Sites were required to have in place a basic information system.
  • 'Consent to the evaluation' - Study sites were invited to participate, not 'conscripted'.
  • 'State and Territory balance' - Study sites were drawn from four different States and Territories.
  • 'Urban-Rural mix' - 25% of the site sample was based in rural areas.
  • 'Mix of development stages' - Collectively, the site sample included services at different stages of reform.
Top of page

National stakeholder survey and consultation

Complementing the local perspectives on change gathered in the Area Case Studies, a separate round of consultations and surveys was conducted with organisations providing national representation of one or more key groups.

Organisations approached included national peak bodies representing health professionals, consumers and carers, State and Territory peak mental health consumer bodies and groups with lead agency responsibilities under the Strategy. A total of 182 organisations were invited to participate through one of three means.

First, organisations were requested to complete a survey instrument on their perception of changes that have occurred over the period of the National Mental Health Strategy. Using a set of statements that reflected the outcome aspirations of the Strategy in each of the four focus areas, organisations were asked to rate the situation as they saw it in 1992, the beginning of the Strategy, and again for 1997.

Second, organisations were invited to prepare written submissions to elaborate their views of the appropriateness and effectiveness of the Strategy, and their views about future priorities.

Finally, approximately 15% of the organisations were invited to discuss directly with the committee their views on the National Mental Health Strategy.

The Centre for Health Program Evaluation, a unit of the University of Melbourne Department of Public Health and Community Medicine, was commissioned to assist with this aspect of the evaluation. Consultations and surveys were conducted between May and August 1997.

Review of national data

Available data collections held at the national level were reviewed to determine whether they could contribute to the evaluation.

Most important here were the source data gathered by the Commonwealth used to report progress of the Strategy in the annual National Mental Health Reports. Although not released at the time the Committee prepared this report, the Commonwealth gained State and Territory consent to make available the most recent data relating to the 1995-96 year, due for publication in the 1996 National Mental Health Report.

Review of this data was managed within the committee, with input from Bill Buckingham & Associates, the consultant engaged to assist the committee in supervising the overall evaluation research program.

International expert commentary

This final element of the methodology sought an international perspective on Australia's National Mental Health Strategy. The aim was to seek expert mental health commentary on:
  • the appropriateness of the Strategy as a framework for reform when compared to developments overseas and
  • the most and least successful aspects of the National Mental Health Strategy, in terms of conception and policy development.
This element of the methodology put Australian developments in a broader context, to determine whether the policy framework is consistent with current international thinking and research about mental health services.

The review was conducted by Dr Ronald Manderscheid, Deputy Director of the Centre for Mental Health Services, a central unit of the United States Department of Health and Human Services.

The Centre is the lead organisation on a range of mental health projects auspiced by the World Health Organisation.

Extensive documentation relating to the Strategy was initially forwarded by the committee to the Centre, supported by a number of teleconferences.

Dr Manderscheid visited Australia in September to meet with the Evaluation Committee and a wide range of individual providers, consumers and carers prior to preparing his report.
Top of page

Summary of method

The methodology aimed to bring together complementary views of the National Mental Health Strategy that would:
  • identify the main changes since the commencement of the Strategy in each of the focus areas
  • determine which aspects of the Strategy were working effectively and which were not, from the perspective of those who have a national view
  • assess the extent of 'on ground' support for the types of changes advocated by the Strategy in local communities where these have been implemented and
  • determine whether Australia's policy settings for mental health are in keeping with 'best practice' approaches emerging overseas.
The evidence gathered through this process illustrated many examples of innovation and service improvement in Australia's mental health services arising from the National Mental Health Strategy. It also highlighted areas in which the Strategy has made minimal progress or failed to deliver expected results.

Based on this work, the following sections summarise the committee's principal observations and conclusions in each of the focus areas. The report concludes with a summary statement of the issues the committee considers to be the main priorities for the future national mental health action.

As a final report, this paper does not attempt to summarise the evidence, but rather draws together conclusions based on the information gathered in the research studies commissioned for the evaluation. As indicated earlier, each of the commissioned studies is published as a separate volume of the evaluation report.
Top of page