Section 3 Evaluating, Implementing, & Managing Instructional Programs & Projects

1. Chapter 10 discusses evaluation in instructional design and provides you with two evaluation models, the CIPP and Kirkpatrick models for evaluation. Search for at least two other models used for evaluation and summarize these models. Describe how you would use them to evaluate your instruction. Reflect on what other questions that instructional design evaluation should address besides whether the instructional design leads to comparable amounts of learning and learner satisfaction as traditional methods. What else would be useful to know?

One other model used for evaluation was developed in the late 1970s by Peter Rossi and his colleagues. According to this model known as Rossi’s Five Domain Evaluation Model, “each evaluation should be tailored to fit local needs, resources, and type of program”. It was Rossi’s belief that “the evaluation questions constitute the core, from which the rest of the evaluation evolves”. Therefore, the model emphasizes “five primary evaluation domains”: needs assessment, program theory assessment, implementation assessment, impact assessment, and efficiency assessment. With the domain of needs assessment, the question of whether or not there is a need for this type of program might be raised. Under program theory assessment, one might pose the question “Is the program conceptualized in such a way that it should work?”.  Implementation assessment is the domain used to determine if the program was carried out or “implemented” correctly and according to plan. The forth domain of impact assessment deals with cause and effect to evaluate whether or not the program “had an impact on its intended targets”. The final domain, efficiency assessment is designed to address the idea of whether or not the program was “cost effective”.

 Essentially, Rossi’s Five Domain model of evaluation looks to see if there is a need for the program, if the program will work, how it can be carried out, if it accomplished the desired task, and if it was efficient. Although I have not had any recent classroom teaching experience, I can see how considering these five steps would help me to better evaluate lessons that I teach. The times that I have conducted lessons, I have not always stopped to consider if the lesson was implemented exactly as planned. This is often due to the fact that there are many consistently changing variables within a classroom setting that can affect the successful implementation of a program or lesson. When I think about the way that budgets are steadily declining, I can see where the last domain in this model would be most helpful because if the program were not cost effective, it would most likely not be able to be successfully implemented in a school setting where budgets are often tight.

Another evaluation model mentioned in this chapter is that of Robert Brinkerhoff. His evaluation method known as the Success Case Method (SCM) is “focused on finding out what about a training or other organizational intervention worked”. In other words, according to this model, an effective method for determining what “works” is by comparing successful cases to unsuccessful ones. Like Rossi’s Five Domain Evaluation Model, there are also five steps to this model of evaluation. The first step in the process is to focus and plan the success case study. After this has been completed, the next step involves constructing a visual “impact model”. The third step involves conducting surveys or questionnaires to determine the best and worst cases. After the successful and unsuccessful cases have been identified, the fourth step is to conduct “in-depth interviews with multiple success cases”, and “nonsuccess cases”. This step allows for the identification of “facilitating and inhibiting use factors”. The final step requires documenting and communicating the evaluation findings.  This final report should contain “detailed data and evidence”, as well as a “rich narrative” that tells how the program was successful and ways for it to be even more successful in the future.

I like the fact that the Success Case Method of evaluation involves feedback from individuals involved in the program or those receiving the instruction. In my opinion, this would be very helpful when evaluating my lessons or instructional designs. I think that the idea of surveying the participants or students would be a fairly simple method for providing initial feedback on the success or failure of my instruction.

When evaluating instructional design, I think that it might also be helpful to address the question of how different learners or groups were affected by the program, not just the targeted group. In addition to that, I think that it would be good to consider ways in which the design could be adapted. This might be extremely helpful in areas where there is inefficiency with the instructional design due to budget restrictions or lack of available resources.

2. Chapters 12 & 13 focus on project management and how to manage projects when resources are scarce. You have been assigned to develop a series of professional development sessions focusing on technology use in the classroom for teachers during a time of economic decline. How will you use Situational Leadership to facilitate this project and manage scarce resources?

With situational leadership, effective leadership is “task-relevant”, meaning that the leader is expected to adapt his or her leadership style to that which works best with the individuals he or she is in charge of leading and the task at hand.  According to Dr. Paul Hersey’s model of situational leadership, there are four phases for “developing leadership skills and influencing individuals”. These phases are based upon the following three factors: “the amount of guidance and direction a leader gives, the amount of socioemotional  support a leader provides,  and the readiness level followers exhibit in performing a specific task, function, or objective”.  When approaching the task of facilitating this series of professional development sessions, I think that it is important to determine the goal of sessions and evaluate the current skill levels of participants, as well as the resources that are available to conduct the sessions.

As the leader, communication throughout a project is essential to the success or failure of the project. Speaking from limited experience, I know that when I have participated in training sessions, I want the instructor to provide direct and detailed instructions, but I also do not want to feel intimidated or overwhelmed by the process. I view the situational leadership approach as being able to adjust your leadership style to that which works best with your participants in order to accomplish the goal. At the beginning of the program, my leadership and instructional approach should be one that is direct and detailed, while being observant of the participants’ response. As the training progresses, my leadership style should adjust to reflect the fact that the learners are gaining knowledge and therefore, will look for me to provide clarification that they are making effective progress. At this point in the process, I would like the participants to feel as though we are working together to achieve the goal. Providing training to help teachers incorporate technology into their classrooms is a great goal, but doing so during a time of economic decline will greatly impact the ability of the teachers to use what they learn because the resources will be scarce due to lack of funds.  This is why it is important to evaluate the scope of the project at the beginning to determine if the project is manageable with the current resources. As the leader, I need to be mindful of those factors and provide ways that the teachers can adapt what they learn based on the available resources. Ultimately, my role as a leader should be to provide instruction, direction, and motivation to help the teachers meet the goal of the project. In this position, it most important that I am able to build positive relationships and adapt to changes and challenges that may arise.