Over the past several years, a growing accountability movement has emerged in the fields of both prevention and education. As researchers and federal agencies invest increasing amounts of time, money, and energy into identifying and disseminating information about research-based programs, developers face mounting pressure to document the effectiveness of their interventions. Schools, in turn, are encouraged to select and implement only those programs that have demonstrated success, in the hopes that such programs will improve the health and safety of their students.

Yet many programs that have proven their effectiveness under strict, research conditions have difficulty doing so when delivered under "real world" conditions within schools or community agencies. A growing body of literature examines the reasons for these differences, which can often be attributed to quality of implementation.

Quality implementation refers to the degree to which an intervention is delivered as it was intended to be delivered. Programs that are implemented with a high level of integrity or fidelity are delivered with strict adherence to their original design. Programs that are delivered with high levels of fidelity will be more likely to achieve their intended outcomes.

However, many researchers in the field assert that successful implementation actually depends on some degree of adaptation. Programs must be adapted to accommodate complex and often idiosyncratic school settings, as well as to meet the needs of a variety of target audiences.

Given the tension between fidelity and adaptation, a third group of researchers have proposed a compromise: They suggest that program modification is acceptable as long as a program's active ingredients or core elements are delivered as planned. Core elements are intervention components that must be maintained without alteration to ensure program effectiveness. They may include basic program structure (e.g., number of sessions, setting), content (e.g., inclusion of certain types of skill-building activities), or method of delivery (e.g., modeling and practice of skills, role plays).

Quality implementation could thus be defined as:

The effective delivery of a program's core components to its target audience.

This definition recognizes that, in order to achieve your intended outcomes, you must not only look at what you are implementing (the core elements), but also how and to whom they are delivered. All of these factors will ultimately affect the quality of your implementation efforts.

Across the prevention literature, findings support the conclusion that implementation influences program outcomes. Consider these examples:

Numerous evaluation studies have revealed the potential of the Life Skills Training (LST) program to produce positive outcomes among youth. Research has also shown that LST has no measurable effect when less than 60 percent of the curriculum is implemented. Yet, implementation of research-based prevention programs -- including LST -- often falls short of this level. In fact, some studies have documented that between 20 and 80 percent of a prevention program's activities may be left out when implemented within a school or community settings. (Gottfredson, D.G. et. al., 2000)
An evaluation study of the Midwestern Prevention Project/ Project STAR assessed student outcomes according to "level" of implementation. Findings showed that students at "low" implementation schools, when compared to non-intervention controls, demonstrated 18 percent less cigarette smoking, 25 percent less alcohol use, and the same level of marijuana use. By comparison, students at the "high" implementation schools, when compared to controls, demonstrated 43 percent less cigarette smoking, 34 percent less alcohol use, and 33 percent less marijuana use. (Durlak,J.A., 1998)

Poor implementation can, and often does, diminish program impact. Yet many schools and communities that seek out research-based programs precisely because of their documented potential to work sacrifice this potential to poor implementation. According to the National Study on Delinquency Prevention in Schools: "Individual prevention activities are not being implemented with sufficient strength and fidelity to be expected to produce a measurable difference in the desired outcomes."

Clearly, schools are not willfully sabotaging their implementation efforts. So what's going wrong?

Perhaps the most difficult part of implementing a prevention program is figuring out if your school can implement a program with fidelity or whether some degree of adaptation will be necessary. If you decide that adaptation is needed, you must then determine the types and degrees of program adaptations that will boost, rather than detract from, program effectiveness. For example, allowing a teacher to present materials in her own style may promote buy-in for your program, but it may also compromise the quality of the presentation. Similarly, squeezing a 6-week curriculum into three 50-minute health classes might be the only way to integrate prevention into your school day, but what are you giving up in the process? Finally, what do you do if a program's core components are not clearly defined?

While there are no easy solutions to these issues, you may find this general approach to implementation helpful:

First, determine if the research-based program you have selected can be implemented with fidelity. Your priority should be to replicate with fidelity a research-based program that has been proven effective through rigorous evaluation.
If you choose to adapt a program, be sure to retain the core elements of the original research-based intervention. Doing so will increase the likelihood that you will achieve program outcomes similar to those achieved in a research setting.
Prepare all educators who are implementing the program to adhere to these elements to the greatest extent possible.
Finally, be sure the prevention program meets the needs and draws on the assets of the young people you serve. The ultimate goal of program adaptation is creating a program that respects and responds to the needs of all members of the target population.

As you work to strike the right balance for your school, make sure to do so in consultation with program representatives and school and community partners. Together, you may be able to find a way to increase your school's capacity to meet program requirements so that fewer adaptations are necessary. Remember, if you do end up altering language or replacing activities within a program session, make every effort to retain the essence of the lesson -- including the kind of information delivered and the methods employed. Also, make sure to document any changes that you make.

Quality implementation is a dynamic, highly subjective process. Numerous factors affect and influence program delivery: Some you will have control over, some you will not. Over the next few days, you will have the opportunity to explore and discuss some of the steps that you can take to improve program implementation in your school, such as selecting programs that provide structure and support for quality implementation (Day 2), shaping the school environment so that it is conducive to quality implementation (Day 3), and monitoring program implementation so that you can correct problems as they arise (Day 4).

Please read the following scenario, then share your reactions in the Discussion Area.

Judy is a middle school coordinator working in a school with a 97% Native American population on a remote reservation. As she searched for an appropriate research-based prevention program for her school, two significant problems emerged. First, there were no programs designed for or evaluated with a Native American population, and the few that tried to tailor their activities to this population did so only superficially. According to Judy, "They took a regular house, made it into a teepee, and added a couple of feathers." Second, most of the program materials were too hard for her students to read, since most of the students in the school read below grade level. Unfortunately, programs written at a lower level included graphics and examples geared to much younger children.

Discussion Questions

What are Judy's options?

Have you experienced any similar challenges identifying a research-based prevention program for your school? If so, what did you do?

What are some other factors that might lead a school to want or need to adapt a research-based prevention program?

This completes today's work.

Please visit the Discussion Area to share your responses to the discussion questions!

References

Dane, A.V. & Schneider, B.H. (1998). Program Integrity in Primary and Early Secondary Prevention: Are Implementation Effects Out of Control? Clinical Psychology Review, 18 (1), pp. 25-45.

Durlak, J.A. (1998). Why Program Implementation is Important. Journal of Prevention and Intervention in the Community, 17 (2), pp. 5-18.

Graczyk, P.A., Domitrovich, C.E, & Zins, J. E. (in press). Facilitating the implementation of evidence-based prevention and mental health promotion efforts in schools. In M. Weist, S. Evans, & N. Tashman (Eds.), School Mental Health Handbook, a volume in the series Issues in Clinical Child Psychology (M. Roberts, Ed.).

Greenberg, M.T., Domitrovich, C.E., Graczyk, P., & Zins, J. (2001, January). A Conceptual Model of Implementation for School-Based Preventive Interventions: Implications for Research, Practice, and Policy. Report to the Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, US Department of Health and Human Services.

Gottfredson, G.D., Gottfredson, D.C., Czeh, E.R., Cantor, D., Crosse, S.B., & Hantman, I. (2000). National Study of Delinquency Prevention in Schools. Ellicott City, MD: Gottfredson Associates, Inc. Available at http://www.gottfredson.com/national.htm.