Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Factors that enable and hinder the implementation of projects in the alcohol and other drug field

Factors that enable and hinder the implementation of projects in the alcohol and other drug field P rojects have limited scope to achieve their intended impact if they are poorly implemented. We report here on a study designed to identify enabling factors critical to successful project implementation and barriers that impeded implementation within a sample of 127 completed projects funded by the Alcohol, Education and Rehabilitation (AER) Foundation between 2002 and 2008. Since its inception in 2001, the AER Foundation (now known as the Foundation for Alcohol Research and Education) has provided time‐limited funding for more than 1,000 projects intended to reduce the burden of harm arising from alcohol and/or inhalant misuse in Australia. Records of funded projects held at the AER Foundation provide rich information on successes and difficulties encountered in relation to funded activities in the alcohol and other drug (AOD) field. Projects in the study sample entailed diverse activities. Some were treatment programs; others comprised workforce training or community education and prevention activities. An additional group of projects involved strategies to enhance the efficiency of AOD agencies through, for instance, policy development. All projects in the study sample were implemented within AOD agencies, or were designed to address AOD misuse and were implemented in health and welfare agencies. While substantial research effort is devoted to identifying the outcomes of AOD interventions, few studies systematically explore elements of successful project implementation across a range of activities. Evaluations of AOD and other health interventions attend to issues of implementation; however they usually concern one intervention type. Other research provides theoretical rather than empirical analyses, offering useful typologies of factors that contribute to intervention success. Drawing on this literature and on a heterogeneous sample of 127 AER Foundation funded projects, this study provides empirically‐based categories of barriers and enablers to project implementation ranked according to the frequency with which they occurred within the study sample. Additionally, the study identifies a small number of barriers and enablers that were statistically associated with meeting all funding objectives for projects in the study sample. Methods Realist synthesis methodology attempts to grapple with the complex factors influencing a social intervention's impact by reviewing underlying mechanisms or sequences of events that enable interventions to work, rather than assessing specific interventions or programs. For example, Pawson explored the effect of ‘incentivisation’ (providing incentives to people to use services or engage in programs) across the contexts of smoking cessation, housing and correctional services. An adapted version of realist synthesis enabled us to explore a range of variables associated with successful and less successful project implementation across projects involving different activities, as discussed in greater detail elsewhere. Ethical approval to conduct the research was obtained from the University of Melbourne Human Research Ethics Committee. The 127 projects included in our eventual study sample were selected in order to maximise similarity of reporting. We excluded projects where final reporting was unavailable, which were funded for less than $20,000, or which funded research or infrastructure. The AER stores project documentation electronically and in paper files. We analysed funding agreements, interim and final reports (and external evaluations for the 41 projects where they existed) to identify and record barriers and enablers to project implementation for each project. Data analysis entailed first and second coding cycles. Lower level (more specific) enabling and barrier factors were identified for each project and coded. Searches were conducted to develop a brief literature review concerning implementation of health and welfare interventions. Enablers and barriers to successful project implementation were then developed through an iterative process involving clustering factors into conceptually coherent categories which also reflected themes emerging from our literature review. Enabler and barrier factors for each project were then coded into these categories. We also identified which projects were successfully implemented, defined as having met all project funding objectives. Data were collected through Access and exported to SPSS for analysis. Fisher's Exact Test was used to assess relationships between successful project implementation and each of the enabling and barrier factors, with a p value of 0.05 or less denoting significance. Values are stated for factors significantly associated with successful implementation. Table 1 shows the enablers and barriers identified through this process, with examples of factors included in each of these categories listed in italics: 1 Enabler and barrier categories identified. Enablers Barriers External communication and relationships Support from partner agencies or from participating communities, used existing networks Engaging communities and partner agencies Lack of partner agency or community interest in or commitment to project, partner agency withdrawal Staffing and leadership Employed suitable staff or contractors, staff or management provided leadership, staff training activities undertaken Identifying and retaining staff Delay in staff recruitment, staff or management turnover, staff lacked required skills Project planning and design Evidence‐based model, good fit to needs, flexible design, appropriate scoping, holistic approach Project planning and design Poor fit, inadequate scoping, poor timing Organisational governance and capacity Organisation already experienced in project work, good policies & procedures, effective internal reference group Governance Lack of management involvement, reference group unrepresentative or ineffectual Sensitivity to service users and settings Model culturally appropriate, employed culturally/gender appropriate staff Meeting cultural needs of specific demographic groups Resources or approach not culturally appropriate, failed to engage specific demographic groups Staff team communication and relationships Staff engaged and enthusiastic about project, staff consultation mechanisms Clarity of staffing roles Staff roles unclear, staff conflict, staff didn't prioritise project involvement, placed additional stress on staff Participatory approach to service delivery Target group involved in development, used role models or peer approaches, activity based approaches Complexities of service users Participation poor, challenging behaviours Funding and resourcing Well funded, used existing resources, gained additional funding, partner agency contributed resources Funding and resourcing Submissions for ongoing funding unsuccessful, other funding problems Research and data collection Well documented, effective data collection, ongoing research Research, evaluation and data collection Poor data collection systems, poor response to evaluation, datasets missing Service system issues not identified as an enabler by projects in the study sample Wider service system challenges Lack of other services, inter‐professional problems, philosophical differences in addressing AOD Note: Enablers are listed alongside a corresponding barrier. They are presented in order of enabler to implementation most frequently mentioned in project records to least frequently mentioned . Eight case studies were selected from the overall study sample to give greater depth to our analysis by exploring implementation across a mix of project types, urban and non‐urban settings, and implementation success. Selection criteria included availability of external evaluations, recent funding and organisational agreement to participate. A total of 22 interviews were conducted with a thematic analysis of interview transcripts used to identify arguments about factors associated with project implementation. Inevitably, the study entailed limitations, not least of which concerned the heterogeneity of projects in the sample. Barrier and enabling factors were clearly articulated for some projects; in others we were required to identify them from available documentation. Barriers and enablers may be categorised in many ways, and ‘successful implementation’ is also open to various interpretations. These decisions reflect the study team's wish to identify definitions that both reflect the data and make sense to AOD agencies and funding bodies. Further, some barriers and enablers do not apply equally across the study sample (e.g. the barrier ‘meeting cultural needs of specific demographic groups’) and the importance of these issues as major themes for implementation may thus be underrepresented. Percentages are provided for each barrier and enabler category for consistency of reporting. Findings should be interpreted with these limitations in mind. Enablers to project implementation As noted above, evaluations of AOD and other public health interventions identify a range of issues implicated in successful project implementation. For example, in the US, Guydish et al. recommended an extensive period of implementation planning for substance abuse treatment programs to account for practical considerations and ensure consistency in the theoretical paradigm underpinning operations. A review of factors that contribute to long‐term community AOD harm‐prevention projects found that honouring community values, cultivating key leader support, and utilising Indigenous staff were key characteristics of successful projects. Reflection on the implementation of centralised intake models for AOD treatment across three jurisdictions (the US; Ontario, Canada; and Victoria, Australia) noted the importance of preparatory activities so treatment sectors were ready and amenable to change. Key success factors in community‐based lifestyle interventions have been identified as: extensive reach, adoption and engagement; sufficient time for social and societal changes to take place; substantial commitment from all stakeholders; and sufficient funding. Strempel and colleagues documented five examples of ‘best’ practice in Australian Indigenous AOD projects, concluding that each entailed “clearly defined and effective management structures and procedures; trained staff and ongoing staff development programs; good multi‐strategy and collaborative approaches; strong leadership; adequate and continuing funding; and clearly defined realistic objectives aimed at the provision of appropriate services that address community needs”. These frameworks informed our identification of enabler and barrier categories, as outlined below and illustrated in Figure 1 . 1 Enablers to project implementation by frequency noted in project documentation (n=127). External communication and relationships Community‐based initiatives are unlikely to be effective without support and participation from the communities concerned or integration with other elements of the service system. An evaluation of partnership strategies in Victorian primary care found that partnerships were most successful when there was: a common purpose; effective communication and clarity about roles and relationships; a positive attitude among stakeholders and member agencies towards the partnership and each other; supportive policy, planning and resources; and relevant effective skills and leadership. Collaboration requires substantial investments of time and energy. Many projects in our study sample relied on partnerships with other agencies or engagement with communities and their representatives to implement activities. Across all projects in our study sample, ‘external communication and relationships’, emerged as the most commonly cited enabler, reported in relation to 73% (n=93) of projects. Within this category, partner agency enthusiasm or having already established relationships with key agencies was the most commonly cited enabling factor, noted in relation to 52% (n=66) of projects. Community enthusiasm for the project and for the organisation conducting the project was cited as an enabling factor in relation to 28% (n=36) of projects. Political or government support was a further factor that facilitated project implementation. As described above, each project in the study sample was classified as successfully implemented or not, according to whether all objectives stipulated in funding agreements had been met. Barrier and enabler factors were then tested for statistical association with successful implementation. For the majority of successfully implemented projects (78%, n=99), reports or evaluations noted community and/or elder enthusiasm (in the case of projects in Indigenous settings) for the organisation and project, while only 22% of projects not successfully implemented showed evidence of this support. This was a statistically significant difference (p=<0.05, two tailed Fisher's exact test). We also found that evidence of a partner agency's enthusiasm for the project and positive regard for the organisation conducting it was significantly correlated with successful implementation ( p ≤0.05, two tailed Fisher's exact test). A hallmark of successful partnerships in case studies was that expectations were clearly defined (for instance through partnership agreements) and that agreements were responsive to program development. Drawing on a network of influential supporters and advisers was also important as the following stakeholder comment indicates: [The project] steering committee assisted with national representation, including small and large agencies. This gave really thorough grounding to the direction of the project . Staff leadership A credible leader can contribute to program success through effectively instigating and managing change. Attributes commonly associated with effective leadership include passion and tenacity, high level interpersonal skills, vision and goal setting capabilities, sound experience, political skills, integrity, curiosity, flexibility, patience, and the ability to take risks and learn from the consequences. Employing the right staff to drive project activities and ensure a profile for a project emerged as critical to successful implementation, particularly where this needed to occur within tight timelines. Staff continuity was crucial in providing ongoing momentum for projects in the study sample. Project leadership by staff and management, and staff development, was an enabler for 61%, (n=78) of projects in our study sample. Under ‘staff leadership’, the most frequently reported enabling factor was the inclusion of training or mentoring within the project, recorded in relation to 33% (n=42) of projects. Other important enabling factors included recruiting a skilled staff member or ‘project champion’, reported in relation to 28% (n=36) of projects, and staff enthusiasm for the project, reported in relation to 17% (n=21) of projects. Employing skilled staff to operate a project was significantly related to successful project implementation ( p ≤0.05, two tailed Fisher's exact test). An enabling factor identified in many of the successfully implemented case studies was the commitment and energy of staff. For example, stakeholders interviewed in relation to one case study commented on staff enthusiasm: They're very positive action‐orientated people . Project planning and design Numerous well‐evidenced and well‐honed models and frameworks are available to guide planning, implementing and evaluating projects: many are universally applicable, while some are styled specifically for particular program types and settings. Program theory, logic modelling and similar approaches help to map activities, outcomes and indicators, and aid both implementation and evaluation. Examples that can be found in the AOD and health promotion fields are: a resource kit for workforce development and a Victorian web‐based compendium of tools. Project planning and design was recorded as an enabler in relation to 61%, (n=78) of projects in our study sample. Under this category the most frequently mentioned enabling factor was the utilisation of an established or evidence‐based project model, cited in relation to 25% (n=32) of projects. Appropriate scoping of project duration was another frequently identified element of successful implementation. For projects which involved service delivery for clients or service users, having clear rules or expectations of participants was observed to be an enabling factor, as were: using an holistic approach; employing a diverse range of activities or approaches and employing a multidisciplinary design. Despite the importance of using an established model, design flexibility was also important. A stakeholder who had been involved in a case study project involving young people believed that having capacity to adjust the project design as activities evolved was important: [Staff] listened and learned. It evolved. They had a framework with what they did but they didn't let it control them. It [the project] was adaptable . Organisational governance and capacity The literature indicates that organisational culture is crucial to effective program delivery. Organisations that are flexible enough to respond to the needs of employees may increase their capacity to attract and retain valuable staff. Governance processes and capacity of the organisation operating the project were mentioned as an enabler in relation to at least half of the projects in the study sample (53%, n=67). In ‘governance and capacity’, the most frequently cited enabling factors were that the organisation already had experience in this kind of work (25%, n=32) or the utilisation of an effective steering committee or working group (20%, n=25). The existence of good governance structures also advantaged project implementation, as did introduction of new software or other communication technology. Case study analysis highlighted the importance of clearly defined roles in the management and development of programs and resources. Sensitivity to service users and settings Designing programs to fit their intended contexts is critical to success. Success may be undermined where programs that have proved exemplary in other settings are reproduced without adequately considering the new implementation context (including cultural requirements of participants and other resources available). For projects which entailed service provision or otherwise involved direct contact with clients or community members, sensitivity to service users and settings emerged as an important enabler, cited in relation to 42% of projects in the study sample (n=54). In one case study project, for instance, focus groups were used to inform development of bilingual resources which were subsequently reviewed by community members. Overall, 44 projects in the study sample (35%) cited having a culturally appropriate model as a key element in their success. Staff team communication Positive relationships between members of staffing teams are strongly implicated in organisational efficiency. Allocating staff roles clearly and equitably is essential to reducing workplace conflict. Staff team communication was noted in 38% of projects (n=48) to be essential in ensuring smooth implementation. It is difficult to implement a project where staff do not feel engaged in and supportive of changes to be made. Staff consultation about any new project was viewed as critical by a range of agencies. Some projects entailed activities such as workshops to discuss and plan new initiatives. Clarity around roles of staff within the project was also identified as an enabling factor to implementation. Utilising a participatory approach There are many challenges involved in encouraging service user participation and retention particularly in the AOD field. Adopting a participatory approach was cited in relation to 36% of projects in the study sample (n=46) as enabling project implementation through encouraging service user engagement. Some projects benefited from adopting participatory approaches such as using role models (i.e. sports people) or implementing peer education or mentoring of service users by ex‐service users. Many projects involving work with young people observed that they used activity‐based approaches to effectively engage participants. Involving the project's target group in development of resources and approaches was significantly correlated with successful project implementation ( p ≤0.05, two tailed Fisher's exact test). This included focus testing educational resources with community members, including service users on committees and implementing consumer or participant feedback mechanisms. Funding and resourcing Funding and resourcing was cited in relation to 36% of projects (n=46) as an enabler to project implementation. Some agencies gained concurrent additional funding or accessed new resources such as vehicles during the lifespan of the project. A small number obtained grants to operate the project subsequent to the conclusion of AER funding. A statistically significant relationship emerged between the receipt of in‐kind support by partner organisations and agencies, and successful project implementation ( p =<0.01, two tailed Fisher's exact test). This is probably a measure of partner agency commitment. Stakeholders interviewed in relation to two case studies suggested that the AER supported project implementation by extending funding in instances where unanticipated delays occurred. Research and data collection Current Australian health policy strongly highlights research evidence and continuous learning and improvement as levers for reform. Health policy and practice are expected to be evidence‐based and to demonstrate accountability through measuring performance against identified indicators of effectiveness. These imperatives necessitate an ongoing commitment to research and evaluation. Locally conducted research and evaluation can also assist in identifying service priorities. Research and data collection was observed as an enabling factor for 28% of projects (n=35) in the study sample. Of these, 24% of projects (n=30) identified research or evaluation that informed project development as an enabling factor. Barriers to implementation Barrier factors for each project were analysed to identify key categories of impediments to successful project implementation. These are discussed separately from enablers as the categories do not directly mirror each other. Figure 2 illustrates these barriers. 2 Barriers to project implementation by frequency noted in project documentation (n=127). Project planning and design Just under half the projects (49%, n=62) encountered barriers related to project planning and design. In 17% (n=21), this was related to some degree of misfit between the project design and the setting in which it was implemented. This frequently occurred when adopting a project design that had been successfully implemented by another agency without sufficient amendment to fit the new setting. Many of the difficulties encountered in relation to project design were associated with project scoping. In 19% of projects (n=24), activities took longer than anticipated. In a further 10% (n=13) it was concluded that goals had proved to be unrealistic. Identifying and retaining staff Managers of AOD agencies in Australia reported that remuneration levels, stress and burnout, poor organisational support, and stigma associated with AOD use impeded staff recruitment. Difficulties relating to identifying and retaining staff (39%, n=50) was the next major barrier for projects in the study sample. Within this barrier category the most frequently reported factor was staff turnover, reported in 23% of projects (n=29). Additionally, 18 projects (14%) cited delays in staff recruitment as barriers to implementation. Nine (7%) projects found that staff appointed lacked the skills required to undertake the work involved. Had the study sample been larger (one cell in this matrix was less than five), employment of staff who lacked required skills may well have emerged as significantly associated with project failure to achieve all set objectives. Several case study projects experienced project staff turnover which resulted in loss of expertise and delays in project implementation. As one stakeholder commented: Staff turnover is a huge problem in this area of work. You lose a lot of people and their expertise goes with them . Engaging communities and partner agencies Just as relationships with external agencies and communities proved critical in enhancing program implementation, they also led to difficulties in implementing 32% (n=40) of projects. Thirteen per cent (n=17) of project reports and evaluations complained that project partner agencies did not sufficiently prioritise the project, had been difficult to engage, or failed in some way to deliver their promised contribution. In a slightly smaller additional percentage (12%, n=16), the difficulties observed in relation to partner agencies were more serious. In these instances, relationships with partner agencies and communities had broken down to the extent where agencies pulled out of the project or refused to refer clients, or communities terminated their involvement. Across the study sample the breakdown of partner agency relationships was significantly associated with failing to achieve all project objectives. (While the difference was not statistically significant using the Fisher's exact test ( p =0.056, two tailed), the difference was significant using the Pearson Chi‐square test (χ 2 =4.1; df=1; p <0.05).) Case study analysis confirmed the finding that failure to develop clear agreements prior to entering partnerships led to subsequent difficulty in implementation. One case study project encountered serious difficulties when workers at the partner agency were reluctant to implement part of the project. A stakeholder from the agency concerned suggested that this was due to philosophical differences between the agencies, a perceived lack of consultation over the project design and because staff of his agency felt that their expertise was undervalued by the other organisation. Complexities of working with service users AOD service users often have complex needs which must be considered and addressed in designing and implementing projects. Complexities of working with service users (reported by 29% [n=37] of projects within the study sample) was another barrier to service implementation, although it may be argued that these problems should have been anticipated and addressed in project design. Within this category, the most frequently cited barriers were that participation levels were poorer than anticipated (11%, n=14) and that difficulties in clients’ lives, such as homelessness, affected both their engagement in the project and on their AOD use outcomes (10%, n=13). Project staff were challenged by some forms of participant behaviour or when clients attended services in intoxicated states. Some reports complained that participants were reluctant to discuss AOD or change their behaviour. Wider service system problems AOD services rarely lead to change in any individual's behaviour in isolation from other factors. A range of elements of the service system are often required to address any AOD issue. Wider service system issues were seen to constitute a barrier for 29% (n=37) of projects. Within this category, the most frequently observed difficulty was that services in non‐urban areas experienced problems in relation to limited service system support; for instance appropriate referral agencies were not available (9%, n=11). A small number of reports and evaluations cited reluctance of professional groups to work together. Differences in philosophical approaches to AOD within the service system (for instance, favouring abstinence or harm reduction as treatment goals) produced tensions in relation to some projects. Research, evaluation and data collection Services often encountered difficulties in meeting data collection and reporting requirements. Some found that their administrative systems were inadequate to collect data on project activities or that responses to surveys were poor. Although research and evaluation was not one of the most frequently cited barriers to project implementation (24%, n=30), one factor emerged strongly within this category: 35 projects (28%) experienced problems relating to designing or implementing evaluations. Many of the projects experiencing this difficulty reported that staff lacked time or expertise to conduct reviews or evaluations or to collect data throughout the project. Clarity of staffing roles Any new project must be integrated with existing activities at the agency concerned in order to ensure the project work is completed without producing unreasonable demands on staff. Clarity of staffing roles and difficulty integrating the project into the work of the agency emerged as a barrier in 22% of projects (n=28). In some projects, staff roles were unclear, or conflict arose over who should take responsibility for specific tasks. In others, reports or evaluations concluded that a project had suffered because agency staff not directly employed on the project did not prioritise involvement or were resistant to participation, particularly where introduction of new policies was involved. Staff were most resistant when they felt inadequately consulted about changes. Meeting cultural needs of specific demographic groups Agencies operating AER funded projects were generally highly cognisant of the necessity to meet the needs of specific demographic groups such as Indigenous people, people living in rural or remote areas and young people in project design and implementation. Nonetheless some reports and evaluations found that resources or approaches were not culturally or age appropriate, or that inadequate consultation had occurred with key target groups prior to project instigation. This emerged as a barrier in relation to 14% (n=18) of projects. Some projects attracted sufficient numbers of participants but found that specific demographic groups (in one project this was young men) were not effectively engaged in programs or influenced by educational messages. Organisational governance and capacity Barriers relating to the organisational governance and capacity of agencies in which projects were implemented were recorded in relation to 12% of projects (n=15). Organisational governance and capacity also emerged as a problem in several of the case study projects which failed to meet all funding criteria. In one instance, a project officer was employed within a partner organisation, with another organisation managing funding. This caused difficulties when disagreements emerged regarding project directions. Funding and resourcing Spooner and Dadich found that non‐government organisations experience financial deficits that impede service delivery. The time‐limited nature of AER funding led to difficulties, particularly for projects involving direct service delivery. An evaluation report from a treatment project quotes a staff member describing the effect of time‐limited funding on her relationships with service users: The funding…that's a barrier… because we don't really know where we're at. Sometimes it can be hard for your own motivation levels. You know, I'm doing all this work with a young person, but in a couple of months I might not be able to do that any more . Conclusions and implications Although findings should be interpreted in light of methodological limitations outlined above, this study builds on knowledge about effective project implementation in two important ways. First, drawing on relevant research and a large and diverse sample of AOD projects, the study identifies an empirically‐based framework of nine enabler and ten barrier categories of project implementation. Identified categories span the life of a project and concern: planning prior to implementing projects; staffing; relationships with communities and other agencies; service user issues; the wider service system; research and data collection; clarity of roles; meeting cultural needs of service users; agency characteristics; and funding. Three categories emerged as both the most frequently observed enablers and as the most frequently observed barriers. These were: issues concerning partnerships with both communities and other agencies; staff leadership, recruitment and retention; and project planning, generally related to designing, costing and developing accurate timeframes for activities. The framework of enablers and barriers outlined here could be used in a range of ways by both AOD agencies and by funding bodies. For example, agency staff developing funding applications might use the framework to demonstrate implementation strengths and subsequently to monitor project implementation. Funding bodies could use the framework to design funding application forms, requiring agencies to report on how projects will avoid barriers and optimise factors associated with successful implementation and to develop project performance indicators. Particular attention should be given by both funding bodies and agencies to project partnerships, staffing and planning. The second way that the study contributes to the evidence base on project implementation in the AOD field is, however, perhaps more important. We identified statistically significant associations between successful project implementation (defined as achievement of all funding objectives) and a partner agency's or a relevant community's enthusiasm for the project and for the organisation conducting it, and a partner agency's donation of in‐kind resources or services. Further, breakdown of relationships with external agencies during the course of the project was the one barrier factor found to be significantly associated with failure to meet all set objectives. Other factors statistically associated with successful implementation were employment of skilled staff and involving the project's target group in development of resources and approaches (such as instituting consumer feedback opportunities or focus testing educational materials with potential users). The importance of building relationships with agencies and communities and ensuring opportunities for participants to be part of intervention planning and implementation are claims that are frequently made in the literature, generally based on observational research, but for which there is little evidence of the nature provided here. The need to employ skilled staff is perhaps more self evident. These are activities that may easily be overlooked in the rush to achieve more tangible elements of AOD project implementation success – such as producing resources or treating clients. The study could be used to persuade policy makers and funding bodies of the value of investing project hours and resources in developing and maintaining relationships with project partners and relevant communities, providing consumers or participants with opportunities for input into project activities, and making workplaces attractive to excellent staff. This is not just because all three are legitimate and important activities in themselves, but also because they potentially influence the degree to which projects are subsequently successful in meet their funding objectives. Acknowledgements This study was commissioned by the Alcohol, Education and Rehabilitation (AER) Foundation. We acknowledge the contributions of Peter d'Abbs, Robin Room and Tracey Purdam. Thank you also to AER staff, authors of reports and evaluations concerning AER funded projects, and to interview participants. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Australian and New Zealand Journal of Public Health Wiley

Factors that enable and hinder the implementation of projects in the alcohol and other drug field

Download PDF
8 pages

Loading next page...
 
/lp/wiley/factors-that-enable-and-hinder-the-implementation-of-projects-in-the-V0UIH3YAgi

References (37)

Publisher
Wiley
Copyright
© 2012 The Authors. ANZJPH © 2012 Public Health Association of Australia
ISSN
1326-0200
eISSN
1753-6405
DOI
10.1111/j.1753-6405.2011.00804.x
pmid
22313708
Publisher site
See Article on Publisher Site

Abstract

P rojects have limited scope to achieve their intended impact if they are poorly implemented. We report here on a study designed to identify enabling factors critical to successful project implementation and barriers that impeded implementation within a sample of 127 completed projects funded by the Alcohol, Education and Rehabilitation (AER) Foundation between 2002 and 2008. Since its inception in 2001, the AER Foundation (now known as the Foundation for Alcohol Research and Education) has provided time‐limited funding for more than 1,000 projects intended to reduce the burden of harm arising from alcohol and/or inhalant misuse in Australia. Records of funded projects held at the AER Foundation provide rich information on successes and difficulties encountered in relation to funded activities in the alcohol and other drug (AOD) field. Projects in the study sample entailed diverse activities. Some were treatment programs; others comprised workforce training or community education and prevention activities. An additional group of projects involved strategies to enhance the efficiency of AOD agencies through, for instance, policy development. All projects in the study sample were implemented within AOD agencies, or were designed to address AOD misuse and were implemented in health and welfare agencies. While substantial research effort is devoted to identifying the outcomes of AOD interventions, few studies systematically explore elements of successful project implementation across a range of activities. Evaluations of AOD and other health interventions attend to issues of implementation; however they usually concern one intervention type. Other research provides theoretical rather than empirical analyses, offering useful typologies of factors that contribute to intervention success. Drawing on this literature and on a heterogeneous sample of 127 AER Foundation funded projects, this study provides empirically‐based categories of barriers and enablers to project implementation ranked according to the frequency with which they occurred within the study sample. Additionally, the study identifies a small number of barriers and enablers that were statistically associated with meeting all funding objectives for projects in the study sample. Methods Realist synthesis methodology attempts to grapple with the complex factors influencing a social intervention's impact by reviewing underlying mechanisms or sequences of events that enable interventions to work, rather than assessing specific interventions or programs. For example, Pawson explored the effect of ‘incentivisation’ (providing incentives to people to use services or engage in programs) across the contexts of smoking cessation, housing and correctional services. An adapted version of realist synthesis enabled us to explore a range of variables associated with successful and less successful project implementation across projects involving different activities, as discussed in greater detail elsewhere. Ethical approval to conduct the research was obtained from the University of Melbourne Human Research Ethics Committee. The 127 projects included in our eventual study sample were selected in order to maximise similarity of reporting. We excluded projects where final reporting was unavailable, which were funded for less than $20,000, or which funded research or infrastructure. The AER stores project documentation electronically and in paper files. We analysed funding agreements, interim and final reports (and external evaluations for the 41 projects where they existed) to identify and record barriers and enablers to project implementation for each project. Data analysis entailed first and second coding cycles. Lower level (more specific) enabling and barrier factors were identified for each project and coded. Searches were conducted to develop a brief literature review concerning implementation of health and welfare interventions. Enablers and barriers to successful project implementation were then developed through an iterative process involving clustering factors into conceptually coherent categories which also reflected themes emerging from our literature review. Enabler and barrier factors for each project were then coded into these categories. We also identified which projects were successfully implemented, defined as having met all project funding objectives. Data were collected through Access and exported to SPSS for analysis. Fisher's Exact Test was used to assess relationships between successful project implementation and each of the enabling and barrier factors, with a p value of 0.05 or less denoting significance. Values are stated for factors significantly associated with successful implementation. Table 1 shows the enablers and barriers identified through this process, with examples of factors included in each of these categories listed in italics: 1 Enabler and barrier categories identified. Enablers Barriers External communication and relationships Support from partner agencies or from participating communities, used existing networks Engaging communities and partner agencies Lack of partner agency or community interest in or commitment to project, partner agency withdrawal Staffing and leadership Employed suitable staff or contractors, staff or management provided leadership, staff training activities undertaken Identifying and retaining staff Delay in staff recruitment, staff or management turnover, staff lacked required skills Project planning and design Evidence‐based model, good fit to needs, flexible design, appropriate scoping, holistic approach Project planning and design Poor fit, inadequate scoping, poor timing Organisational governance and capacity Organisation already experienced in project work, good policies & procedures, effective internal reference group Governance Lack of management involvement, reference group unrepresentative or ineffectual Sensitivity to service users and settings Model culturally appropriate, employed culturally/gender appropriate staff Meeting cultural needs of specific demographic groups Resources or approach not culturally appropriate, failed to engage specific demographic groups Staff team communication and relationships Staff engaged and enthusiastic about project, staff consultation mechanisms Clarity of staffing roles Staff roles unclear, staff conflict, staff didn't prioritise project involvement, placed additional stress on staff Participatory approach to service delivery Target group involved in development, used role models or peer approaches, activity based approaches Complexities of service users Participation poor, challenging behaviours Funding and resourcing Well funded, used existing resources, gained additional funding, partner agency contributed resources Funding and resourcing Submissions for ongoing funding unsuccessful, other funding problems Research and data collection Well documented, effective data collection, ongoing research Research, evaluation and data collection Poor data collection systems, poor response to evaluation, datasets missing Service system issues not identified as an enabler by projects in the study sample Wider service system challenges Lack of other services, inter‐professional problems, philosophical differences in addressing AOD Note: Enablers are listed alongside a corresponding barrier. They are presented in order of enabler to implementation most frequently mentioned in project records to least frequently mentioned . Eight case studies were selected from the overall study sample to give greater depth to our analysis by exploring implementation across a mix of project types, urban and non‐urban settings, and implementation success. Selection criteria included availability of external evaluations, recent funding and organisational agreement to participate. A total of 22 interviews were conducted with a thematic analysis of interview transcripts used to identify arguments about factors associated with project implementation. Inevitably, the study entailed limitations, not least of which concerned the heterogeneity of projects in the sample. Barrier and enabling factors were clearly articulated for some projects; in others we were required to identify them from available documentation. Barriers and enablers may be categorised in many ways, and ‘successful implementation’ is also open to various interpretations. These decisions reflect the study team's wish to identify definitions that both reflect the data and make sense to AOD agencies and funding bodies. Further, some barriers and enablers do not apply equally across the study sample (e.g. the barrier ‘meeting cultural needs of specific demographic groups’) and the importance of these issues as major themes for implementation may thus be underrepresented. Percentages are provided for each barrier and enabler category for consistency of reporting. Findings should be interpreted with these limitations in mind. Enablers to project implementation As noted above, evaluations of AOD and other public health interventions identify a range of issues implicated in successful project implementation. For example, in the US, Guydish et al. recommended an extensive period of implementation planning for substance abuse treatment programs to account for practical considerations and ensure consistency in the theoretical paradigm underpinning operations. A review of factors that contribute to long‐term community AOD harm‐prevention projects found that honouring community values, cultivating key leader support, and utilising Indigenous staff were key characteristics of successful projects. Reflection on the implementation of centralised intake models for AOD treatment across three jurisdictions (the US; Ontario, Canada; and Victoria, Australia) noted the importance of preparatory activities so treatment sectors were ready and amenable to change. Key success factors in community‐based lifestyle interventions have been identified as: extensive reach, adoption and engagement; sufficient time for social and societal changes to take place; substantial commitment from all stakeholders; and sufficient funding. Strempel and colleagues documented five examples of ‘best’ practice in Australian Indigenous AOD projects, concluding that each entailed “clearly defined and effective management structures and procedures; trained staff and ongoing staff development programs; good multi‐strategy and collaborative approaches; strong leadership; adequate and continuing funding; and clearly defined realistic objectives aimed at the provision of appropriate services that address community needs”. These frameworks informed our identification of enabler and barrier categories, as outlined below and illustrated in Figure 1 . 1 Enablers to project implementation by frequency noted in project documentation (n=127). External communication and relationships Community‐based initiatives are unlikely to be effective without support and participation from the communities concerned or integration with other elements of the service system. An evaluation of partnership strategies in Victorian primary care found that partnerships were most successful when there was: a common purpose; effective communication and clarity about roles and relationships; a positive attitude among stakeholders and member agencies towards the partnership and each other; supportive policy, planning and resources; and relevant effective skills and leadership. Collaboration requires substantial investments of time and energy. Many projects in our study sample relied on partnerships with other agencies or engagement with communities and their representatives to implement activities. Across all projects in our study sample, ‘external communication and relationships’, emerged as the most commonly cited enabler, reported in relation to 73% (n=93) of projects. Within this category, partner agency enthusiasm or having already established relationships with key agencies was the most commonly cited enabling factor, noted in relation to 52% (n=66) of projects. Community enthusiasm for the project and for the organisation conducting the project was cited as an enabling factor in relation to 28% (n=36) of projects. Political or government support was a further factor that facilitated project implementation. As described above, each project in the study sample was classified as successfully implemented or not, according to whether all objectives stipulated in funding agreements had been met. Barrier and enabler factors were then tested for statistical association with successful implementation. For the majority of successfully implemented projects (78%, n=99), reports or evaluations noted community and/or elder enthusiasm (in the case of projects in Indigenous settings) for the organisation and project, while only 22% of projects not successfully implemented showed evidence of this support. This was a statistically significant difference (p=<0.05, two tailed Fisher's exact test). We also found that evidence of a partner agency's enthusiasm for the project and positive regard for the organisation conducting it was significantly correlated with successful implementation ( p ≤0.05, two tailed Fisher's exact test). A hallmark of successful partnerships in case studies was that expectations were clearly defined (for instance through partnership agreements) and that agreements were responsive to program development. Drawing on a network of influential supporters and advisers was also important as the following stakeholder comment indicates: [The project] steering committee assisted with national representation, including small and large agencies. This gave really thorough grounding to the direction of the project . Staff leadership A credible leader can contribute to program success through effectively instigating and managing change. Attributes commonly associated with effective leadership include passion and tenacity, high level interpersonal skills, vision and goal setting capabilities, sound experience, political skills, integrity, curiosity, flexibility, patience, and the ability to take risks and learn from the consequences. Employing the right staff to drive project activities and ensure a profile for a project emerged as critical to successful implementation, particularly where this needed to occur within tight timelines. Staff continuity was crucial in providing ongoing momentum for projects in the study sample. Project leadership by staff and management, and staff development, was an enabler for 61%, (n=78) of projects in our study sample. Under ‘staff leadership’, the most frequently reported enabling factor was the inclusion of training or mentoring within the project, recorded in relation to 33% (n=42) of projects. Other important enabling factors included recruiting a skilled staff member or ‘project champion’, reported in relation to 28% (n=36) of projects, and staff enthusiasm for the project, reported in relation to 17% (n=21) of projects. Employing skilled staff to operate a project was significantly related to successful project implementation ( p ≤0.05, two tailed Fisher's exact test). An enabling factor identified in many of the successfully implemented case studies was the commitment and energy of staff. For example, stakeholders interviewed in relation to one case study commented on staff enthusiasm: They're very positive action‐orientated people . Project planning and design Numerous well‐evidenced and well‐honed models and frameworks are available to guide planning, implementing and evaluating projects: many are universally applicable, while some are styled specifically for particular program types and settings. Program theory, logic modelling and similar approaches help to map activities, outcomes and indicators, and aid both implementation and evaluation. Examples that can be found in the AOD and health promotion fields are: a resource kit for workforce development and a Victorian web‐based compendium of tools. Project planning and design was recorded as an enabler in relation to 61%, (n=78) of projects in our study sample. Under this category the most frequently mentioned enabling factor was the utilisation of an established or evidence‐based project model, cited in relation to 25% (n=32) of projects. Appropriate scoping of project duration was another frequently identified element of successful implementation. For projects which involved service delivery for clients or service users, having clear rules or expectations of participants was observed to be an enabling factor, as were: using an holistic approach; employing a diverse range of activities or approaches and employing a multidisciplinary design. Despite the importance of using an established model, design flexibility was also important. A stakeholder who had been involved in a case study project involving young people believed that having capacity to adjust the project design as activities evolved was important: [Staff] listened and learned. It evolved. They had a framework with what they did but they didn't let it control them. It [the project] was adaptable . Organisational governance and capacity The literature indicates that organisational culture is crucial to effective program delivery. Organisations that are flexible enough to respond to the needs of employees may increase their capacity to attract and retain valuable staff. Governance processes and capacity of the organisation operating the project were mentioned as an enabler in relation to at least half of the projects in the study sample (53%, n=67). In ‘governance and capacity’, the most frequently cited enabling factors were that the organisation already had experience in this kind of work (25%, n=32) or the utilisation of an effective steering committee or working group (20%, n=25). The existence of good governance structures also advantaged project implementation, as did introduction of new software or other communication technology. Case study analysis highlighted the importance of clearly defined roles in the management and development of programs and resources. Sensitivity to service users and settings Designing programs to fit their intended contexts is critical to success. Success may be undermined where programs that have proved exemplary in other settings are reproduced without adequately considering the new implementation context (including cultural requirements of participants and other resources available). For projects which entailed service provision or otherwise involved direct contact with clients or community members, sensitivity to service users and settings emerged as an important enabler, cited in relation to 42% of projects in the study sample (n=54). In one case study project, for instance, focus groups were used to inform development of bilingual resources which were subsequently reviewed by community members. Overall, 44 projects in the study sample (35%) cited having a culturally appropriate model as a key element in their success. Staff team communication Positive relationships between members of staffing teams are strongly implicated in organisational efficiency. Allocating staff roles clearly and equitably is essential to reducing workplace conflict. Staff team communication was noted in 38% of projects (n=48) to be essential in ensuring smooth implementation. It is difficult to implement a project where staff do not feel engaged in and supportive of changes to be made. Staff consultation about any new project was viewed as critical by a range of agencies. Some projects entailed activities such as workshops to discuss and plan new initiatives. Clarity around roles of staff within the project was also identified as an enabling factor to implementation. Utilising a participatory approach There are many challenges involved in encouraging service user participation and retention particularly in the AOD field. Adopting a participatory approach was cited in relation to 36% of projects in the study sample (n=46) as enabling project implementation through encouraging service user engagement. Some projects benefited from adopting participatory approaches such as using role models (i.e. sports people) or implementing peer education or mentoring of service users by ex‐service users. Many projects involving work with young people observed that they used activity‐based approaches to effectively engage participants. Involving the project's target group in development of resources and approaches was significantly correlated with successful project implementation ( p ≤0.05, two tailed Fisher's exact test). This included focus testing educational resources with community members, including service users on committees and implementing consumer or participant feedback mechanisms. Funding and resourcing Funding and resourcing was cited in relation to 36% of projects (n=46) as an enabler to project implementation. Some agencies gained concurrent additional funding or accessed new resources such as vehicles during the lifespan of the project. A small number obtained grants to operate the project subsequent to the conclusion of AER funding. A statistically significant relationship emerged between the receipt of in‐kind support by partner organisations and agencies, and successful project implementation ( p =<0.01, two tailed Fisher's exact test). This is probably a measure of partner agency commitment. Stakeholders interviewed in relation to two case studies suggested that the AER supported project implementation by extending funding in instances where unanticipated delays occurred. Research and data collection Current Australian health policy strongly highlights research evidence and continuous learning and improvement as levers for reform. Health policy and practice are expected to be evidence‐based and to demonstrate accountability through measuring performance against identified indicators of effectiveness. These imperatives necessitate an ongoing commitment to research and evaluation. Locally conducted research and evaluation can also assist in identifying service priorities. Research and data collection was observed as an enabling factor for 28% of projects (n=35) in the study sample. Of these, 24% of projects (n=30) identified research or evaluation that informed project development as an enabling factor. Barriers to implementation Barrier factors for each project were analysed to identify key categories of impediments to successful project implementation. These are discussed separately from enablers as the categories do not directly mirror each other. Figure 2 illustrates these barriers. 2 Barriers to project implementation by frequency noted in project documentation (n=127). Project planning and design Just under half the projects (49%, n=62) encountered barriers related to project planning and design. In 17% (n=21), this was related to some degree of misfit between the project design and the setting in which it was implemented. This frequently occurred when adopting a project design that had been successfully implemented by another agency without sufficient amendment to fit the new setting. Many of the difficulties encountered in relation to project design were associated with project scoping. In 19% of projects (n=24), activities took longer than anticipated. In a further 10% (n=13) it was concluded that goals had proved to be unrealistic. Identifying and retaining staff Managers of AOD agencies in Australia reported that remuneration levels, stress and burnout, poor organisational support, and stigma associated with AOD use impeded staff recruitment. Difficulties relating to identifying and retaining staff (39%, n=50) was the next major barrier for projects in the study sample. Within this barrier category the most frequently reported factor was staff turnover, reported in 23% of projects (n=29). Additionally, 18 projects (14%) cited delays in staff recruitment as barriers to implementation. Nine (7%) projects found that staff appointed lacked the skills required to undertake the work involved. Had the study sample been larger (one cell in this matrix was less than five), employment of staff who lacked required skills may well have emerged as significantly associated with project failure to achieve all set objectives. Several case study projects experienced project staff turnover which resulted in loss of expertise and delays in project implementation. As one stakeholder commented: Staff turnover is a huge problem in this area of work. You lose a lot of people and their expertise goes with them . Engaging communities and partner agencies Just as relationships with external agencies and communities proved critical in enhancing program implementation, they also led to difficulties in implementing 32% (n=40) of projects. Thirteen per cent (n=17) of project reports and evaluations complained that project partner agencies did not sufficiently prioritise the project, had been difficult to engage, or failed in some way to deliver their promised contribution. In a slightly smaller additional percentage (12%, n=16), the difficulties observed in relation to partner agencies were more serious. In these instances, relationships with partner agencies and communities had broken down to the extent where agencies pulled out of the project or refused to refer clients, or communities terminated their involvement. Across the study sample the breakdown of partner agency relationships was significantly associated with failing to achieve all project objectives. (While the difference was not statistically significant using the Fisher's exact test ( p =0.056, two tailed), the difference was significant using the Pearson Chi‐square test (χ 2 =4.1; df=1; p <0.05).) Case study analysis confirmed the finding that failure to develop clear agreements prior to entering partnerships led to subsequent difficulty in implementation. One case study project encountered serious difficulties when workers at the partner agency were reluctant to implement part of the project. A stakeholder from the agency concerned suggested that this was due to philosophical differences between the agencies, a perceived lack of consultation over the project design and because staff of his agency felt that their expertise was undervalued by the other organisation. Complexities of working with service users AOD service users often have complex needs which must be considered and addressed in designing and implementing projects. Complexities of working with service users (reported by 29% [n=37] of projects within the study sample) was another barrier to service implementation, although it may be argued that these problems should have been anticipated and addressed in project design. Within this category, the most frequently cited barriers were that participation levels were poorer than anticipated (11%, n=14) and that difficulties in clients’ lives, such as homelessness, affected both their engagement in the project and on their AOD use outcomes (10%, n=13). Project staff were challenged by some forms of participant behaviour or when clients attended services in intoxicated states. Some reports complained that participants were reluctant to discuss AOD or change their behaviour. Wider service system problems AOD services rarely lead to change in any individual's behaviour in isolation from other factors. A range of elements of the service system are often required to address any AOD issue. Wider service system issues were seen to constitute a barrier for 29% (n=37) of projects. Within this category, the most frequently observed difficulty was that services in non‐urban areas experienced problems in relation to limited service system support; for instance appropriate referral agencies were not available (9%, n=11). A small number of reports and evaluations cited reluctance of professional groups to work together. Differences in philosophical approaches to AOD within the service system (for instance, favouring abstinence or harm reduction as treatment goals) produced tensions in relation to some projects. Research, evaluation and data collection Services often encountered difficulties in meeting data collection and reporting requirements. Some found that their administrative systems were inadequate to collect data on project activities or that responses to surveys were poor. Although research and evaluation was not one of the most frequently cited barriers to project implementation (24%, n=30), one factor emerged strongly within this category: 35 projects (28%) experienced problems relating to designing or implementing evaluations. Many of the projects experiencing this difficulty reported that staff lacked time or expertise to conduct reviews or evaluations or to collect data throughout the project. Clarity of staffing roles Any new project must be integrated with existing activities at the agency concerned in order to ensure the project work is completed without producing unreasonable demands on staff. Clarity of staffing roles and difficulty integrating the project into the work of the agency emerged as a barrier in 22% of projects (n=28). In some projects, staff roles were unclear, or conflict arose over who should take responsibility for specific tasks. In others, reports or evaluations concluded that a project had suffered because agency staff not directly employed on the project did not prioritise involvement or were resistant to participation, particularly where introduction of new policies was involved. Staff were most resistant when they felt inadequately consulted about changes. Meeting cultural needs of specific demographic groups Agencies operating AER funded projects were generally highly cognisant of the necessity to meet the needs of specific demographic groups such as Indigenous people, people living in rural or remote areas and young people in project design and implementation. Nonetheless some reports and evaluations found that resources or approaches were not culturally or age appropriate, or that inadequate consultation had occurred with key target groups prior to project instigation. This emerged as a barrier in relation to 14% (n=18) of projects. Some projects attracted sufficient numbers of participants but found that specific demographic groups (in one project this was young men) were not effectively engaged in programs or influenced by educational messages. Organisational governance and capacity Barriers relating to the organisational governance and capacity of agencies in which projects were implemented were recorded in relation to 12% of projects (n=15). Organisational governance and capacity also emerged as a problem in several of the case study projects which failed to meet all funding criteria. In one instance, a project officer was employed within a partner organisation, with another organisation managing funding. This caused difficulties when disagreements emerged regarding project directions. Funding and resourcing Spooner and Dadich found that non‐government organisations experience financial deficits that impede service delivery. The time‐limited nature of AER funding led to difficulties, particularly for projects involving direct service delivery. An evaluation report from a treatment project quotes a staff member describing the effect of time‐limited funding on her relationships with service users: The funding…that's a barrier… because we don't really know where we're at. Sometimes it can be hard for your own motivation levels. You know, I'm doing all this work with a young person, but in a couple of months I might not be able to do that any more . Conclusions and implications Although findings should be interpreted in light of methodological limitations outlined above, this study builds on knowledge about effective project implementation in two important ways. First, drawing on relevant research and a large and diverse sample of AOD projects, the study identifies an empirically‐based framework of nine enabler and ten barrier categories of project implementation. Identified categories span the life of a project and concern: planning prior to implementing projects; staffing; relationships with communities and other agencies; service user issues; the wider service system; research and data collection; clarity of roles; meeting cultural needs of service users; agency characteristics; and funding. Three categories emerged as both the most frequently observed enablers and as the most frequently observed barriers. These were: issues concerning partnerships with both communities and other agencies; staff leadership, recruitment and retention; and project planning, generally related to designing, costing and developing accurate timeframes for activities. The framework of enablers and barriers outlined here could be used in a range of ways by both AOD agencies and by funding bodies. For example, agency staff developing funding applications might use the framework to demonstrate implementation strengths and subsequently to monitor project implementation. Funding bodies could use the framework to design funding application forms, requiring agencies to report on how projects will avoid barriers and optimise factors associated with successful implementation and to develop project performance indicators. Particular attention should be given by both funding bodies and agencies to project partnerships, staffing and planning. The second way that the study contributes to the evidence base on project implementation in the AOD field is, however, perhaps more important. We identified statistically significant associations between successful project implementation (defined as achievement of all funding objectives) and a partner agency's or a relevant community's enthusiasm for the project and for the organisation conducting it, and a partner agency's donation of in‐kind resources or services. Further, breakdown of relationships with external agencies during the course of the project was the one barrier factor found to be significantly associated with failure to meet all set objectives. Other factors statistically associated with successful implementation were employment of skilled staff and involving the project's target group in development of resources and approaches (such as instituting consumer feedback opportunities or focus testing educational materials with potential users). The importance of building relationships with agencies and communities and ensuring opportunities for participants to be part of intervention planning and implementation are claims that are frequently made in the literature, generally based on observational research, but for which there is little evidence of the nature provided here. The need to employ skilled staff is perhaps more self evident. These are activities that may easily be overlooked in the rush to achieve more tangible elements of AOD project implementation success – such as producing resources or treating clients. The study could be used to persuade policy makers and funding bodies of the value of investing project hours and resources in developing and maintaining relationships with project partners and relevant communities, providing consumers or participants with opportunities for input into project activities, and making workplaces attractive to excellent staff. This is not just because all three are legitimate and important activities in themselves, but also because they potentially influence the degree to which projects are subsequently successful in meet their funding objectives. Acknowledgements This study was commissioned by the Alcohol, Education and Rehabilitation (AER) Foundation. We acknowledge the contributions of Peter d'Abbs, Robin Room and Tracey Purdam. Thank you also to AER staff, authors of reports and evaluations concerning AER funded projects, and to interview participants.

Journal

Australian and New Zealand Journal of Public Health – Wiley

Published: Feb 1, 2012

There are no references for this article.