6. Implement and Determine the Effectiveness of a Staff Development Program

This kind of project requires two kinds of evaluation:

1. A formative evaluation to determine if the new staff development program has been implemented.

2. A summative evaluation to determine if the newly implemented program has improved such things as teacher knowledge, motivation, and skills, and student interest, participation, and achievement.

I Formative Evaluation

The task here is to determine how well (or, indeed, if) the new program has been implemented. This is not something that can be taken for granted.

Thus, you will need to assemble credible, objective evidence (not simply your own judgments) on the following types of questions:

1. Does the program exist?
2. Is it operating as it is supposed to?
3. If not, what changes are needed to make it operational?
4. Have the necessary books, materials, and equipment been purchased and made available to the teachers?
5. Are the teachers for whom the program is intended participating in the training?
6. Do they like the training program, and do they think they are learning from it?

This kind of evaluation will tell you whether the staff development program is operational and potentially capable of having an impact. It is essential to complete this type of evaluation before undertaking a summative, impact, or outcome evaluation. It makes no sense to evaluate the effectiveness of a program that does not exist-even though we regrettably see many instances of this.

II  Summative Evaluation

The two basic questions here are:

>Have there been changes in teacher behavior in the classroom and improvements in student interest, participation, and achievement?

>Can any changes or improvement that have occurred be confidently attributed to the new staff development program rather than other factors?

On the question of whether the staff development program has resulted in changes in teacher behavior, classroom observations of teacher styles and activities should be carried out before and after they experience the training.

On the question of whether there have been improvements in student interest, participation, and achievement, and whether the improvements can be confidently attributed to the changed behavior of the teachers, there are two preferred evaluation designs:

(1) Administer pre- and post-tests to both a group of students in the classrooms of teachers who have gone through the staff development program and a comparable control group of students in classrooms of teachers who haven't. If the gains of the treatment group are statistically and educationally greater than those of the control group, you can confidently conclude that these gains were the result of the new staff development program. Simply administering pre- and post-tests to the treatment or program group without a control group will not do. Scores may have increased because of other conditions or factors besides the new program. For comparison, you need some measure of change under the conditions of non-treatment. This is provided by the control group. (See the section on An Example of the Most Common Pitfall in Evaluating Education Programs in the Short Course on Evaluation Basics.)

(2) As an alternative, if a control group is not available, you can use what is known as the interrupted time-series design. If test scores are available for several years or periods prior to the introduction of the new program, and there is a marked improvement in these scores immediately following the introduction of the new program that is sustained in following years or periods, you can confidently conclude that the gains were the result of the new program.

For more information on these two designs, see the section on Alternative Summative Evaluation Methods in the Short Course on Evaluation Basics, and references 1 (Campbell), 4 (Cook & Campbell), and 8 (Rossi & Freeman) in the Evaluation References.

Design Selections | Evaluation Support Home
Applied Research Center | EdD Major | Ed Leaders Home/ Basic Statistics