National Immunisation Program evaluations
Program evaluation is essential in order to understand the impact of immunisation programs and to determine if goals and objectives set out prior to program implementation were successful. It is also useful in order to determine successes and challenges encountered during program implementation and delivery that can be used to guide future program planning.
Since 2006, NCIRS has been contracted to the Australian Government Department of Health and Ageing to evaluate the introduction of new vaccines onto the National Immunisation Program. Program evaluations are to commence within three years of the implementation of a new publicly funded immunisation program. The specific timeline involved varies with the individual evaluation; however, in general, they should be completed four years after program implementation. The scope of each program evaluation varies, but core components should be included in all.
The program evaluations include three main components – process, outcome and impact: The evaluation reports are divided into the following sections:
a) Process evaluation
Process-based evaluations are undertaken, usually through key informants, to assess and understand how a program was conducted. Evaluating the process of program planning, implementation and delivery provides contextual information for the outcome evaluation. Process evaluation should consist of standardised interviews with key informants in order to:
- describe how the program was planned and delivered, including communication strategies
- identify the strengths and challenges of program implementation and delivery
- develop recommendations for future programs.
b) System description
This is a description and assessment of the surveillance systems used to monitor and evaluate the outcomes and impact of the program, including vaccination coverage and epidemiology. These may include specifically designed stakeholder surveys, the National Notifiable Diseases Surveillance System, the National Hospital Morbidity Database, the Adverse Drug Reaction System database, Australian Childhood Immunisation Register and national serosurveillance data.
c) Outcome/impact
This section evaluates the direct outcomes of vaccination program implementation as well as the resulting impact on disease. It assesses the extent to which the program is meeting predetermined goals or objectives and having the desired effects. Core components are:
- estimation of the vaccine coverage in targeted age groups (outcome)
- description of the adverse events following immunisation (outcome)
- calculation of vaccine effectiveness and identification of vaccine failures (outcome)
- identification of the changes in the epidemiology of the disease over time, including overall incidence, age-specific and regional-specific incidence, as well as serotype/serogroup-specific and ethnicity-specific incidence, if appropriate (impact)
- identification of changes in population seroepidemiology, if appropriate (impact)
- an assessment of the overall impact of the program (impact).
Program evaluation cannot be done by NCIRS in isolation. It requires extensive collaboration and consultation with a broad range of organisations and expert groups, including the Department of Health and Ageing, Communicable Diseases Network Australia, National Immunisation Committee, Medicare Australia, Australian Institute of Health and Welfare, Therapeutic Goods Administration and others.
Immunisation Program Evaluations conducted by NCIRS
- 1998 Measles Control Campaign – completed in 2000
- 2001 young adult measles, mumps and rubella (MMR) campaign – completed in 2003
- National Indigenous Pneumococcal and Influenza Immunisation (NIPII) evaluation – completed in 2004, unpublished
- 2001–2003 National Q-fever evaluation – completed in 2004, unpublished
- 2003–2006 Meningococcal C program – Parts 1 and 2 completed in 2007, unpublished
- Adolescent pertussis immunisation program evaluation – completed in 2009, unpublished
- Pneumococcal immunisation program evaluation – in progress
