EFFECTIVENESS AND EVALUATION OF PROFESSIONAL DEVELOPMENT

Thomas R. Guskey

24 April 2002

Everyone gets into trouble when success is paired with accountability. Accountability is a political issue being defined by politicians who have no educational training. Some of the things they are mandating aren’t educationally sound. Everyone is included in the assessments. States average around 30 (of 100) in their student testing. High stakes are for educators; not for students. Students can hold schools hostage.

High stakes for students and not for educators don’t work well either. Inadequate opportunity to learn or inadequate curriculum have been charged.

Accountability has to do with improvement. If you do nothing and stay the same, you’re in sanctions. If you aren’t making it, you have to develop an improvement plan. An improvement plan eliminates the need to disaggregate by demographics.

ESEA: The idea of looking at improvement greatly influenced the state of Tennessee. William Sanders. “Value added.” They analyze results from one year to the next—value added—gain scores. They began probing data more deeply. You’re better off being in a rich district than a poor district. Then they discovered there’s a big difference between teachers. They have taken accountability to the individual classroom level. Testing occurs annually. The teachers who get good results get good results year after year after year, and teachers who get poor results get poor results year after year after year. It doesn’t matter who you give to good teachers; they get good results. Whether or not they get good results has nothing to do with years of experience—look at this with respect to our teacher compensation system.

Dallas Independent School System uses this system to rank order all teachers in the system! There’s a difference in 50 percentile points in three years between children who had good teachers and those who had bad.

Now we have a President from Texas and a Secretary of Education from Texas, and that’s how we got ESEA. We’re going to test children every year and measure adequate yearly gain.

Every school has models of excellence. Every school also has teachers who need help. We need to develop peer coaching and mentoring. Pair a good teacher with a poor teacher.
What’s on the horizon? The federal government seems to be pressing for politically expedient means.

Professional development’s role in a systemic change effort: One premise: change is a prerequisite to improvement. If you always do what you’ve always done, you’ll always get what you’ve always got. Our students are changing. Change is highly contextualized. What works on one setting won’t necessarily work in another.

What makes PD effective? Please refer to the staff development questionnaire in handouts. Identify your most effective PD experience; identify your least effective PD experience. Circle the elements that seem to make a difference for you.

Five Levels of PD Evaluation:
1. Participants’ reactions to the experience
2. Participants’ learning from the experience
3. Organization support & change
4. Participants’ use of new knowledge and skills
5. Results: Student Learning Outcomes

This is a hierarchy. Effective PD impacts student learning. Grading on the curve and selecting a valedictorian are antithetical to identified student learning outcomes. Graduate students Summa cum laude or Magna cum laude. The first has to do with competition. Our job is not to select talent; our job is to develop talent.

We need to plan backwards. If you want to end up with student learning, you have to start there to reach that goal.

Move PD from the district level to the building level. Each building has a PD team to determine what they’re going to do. They need to develop a PD plan. What level of evaluation are you using to develop PD? Most people only document—we had these workshops with these participants on this topic. They do an opinion survey to see if people liked it. The state of our practice and where we’re pressed to be are very different.

What topic is the most effective part of your PD experience? The lists of best and worst are nearly identical. The topic can be engaging, but presented in an ineffective way. How do we select a topic of what we’re going to include? First thing we do is conduct a needs assessment and then we derive a list of topics. These needs survey are misnamed. You don’t get needs but problems, concerns, dilemmas and wants. If you focus here, you’re going to do stress reduction workshops from here to eternity. These are really symptoms of needs that lie underneath. The number one topic identified by teachers is classroom management. There’s a powerful relationship between engaged time and student learning. In a 50 minute class period, some kids are engaged for 40 of 50 minutes; some for only 4! That’s a good argument against more hours and more days. We need to work on engaging the 4-minute student to 20-minutes. If you manage learning well, you don’t have to worry about managing learners. Bloom understood the need.

Needs surveys give you important information; you need to look underneath to interpret them.

You’ve been told that whatever you do is research-based. Any consultant you hire will tell you his/her ideas are research-based. There are very few ideas out there today that can really be said to be research-based. (Bob Sternburgh and the Triarchal Model has a solid research base behind it.)

There’s no evidence to show that brain-based learning will lead to better learning. We must always ask if there’s evidence to support this and evidence in our context. The one topic that consistently does this is PD designed to increase teachers’ content knowledge. This consistently leads to better student learning.

The challenge is understanding more deeply what is underlying the symptoms. What does this relate to in terms of our kids? 90% of evaluation is considered in the planning stage. If you plan well, evaluation pretty much takes care of itself. Include the critical questions in the planning process.

In program planning, we have a long history of great limitation in our success. We plan PD from a district or regional level, so we moved to a site-based model. We have about ten years of research on this, and we’re doing the same rotten stuff at the site level as we did at the district/regional level. District level planning might not be relevant in the context. Combine district and site-based people. Give teachers the opportunity to share across buildings.

Program participants: should PD be mandatory or voluntary. The research literature is really mixed. In examining the literature deeply, it really depends. It requires a mix. If you want to share information, a one-group presentation works very well. It gives everyone a shared knowledge base and vocabulary that facilitates communication, and it helps dispel rumors. When you move from information sharing to implementation, you must move from mandatory to voluntary. Otherwise, some work to sabotage. Who will volunteer? The people who volunteer are the best, the most effective you have on staff already. The most resistant don’t volunteer. Go with the thoroughbreds! Go with the fastest horse you can ride. Support them in their efforts. When they start getting good results, publicize them and others will be more interested in getting involved. This guarantees larger scale implementation in a shorter period of time. If you get 30-40% to implement, this is a good percentage. These are the early implementers. Train them and support them and publicize their efforts.

It isn’t who they are but who they do that makes a difference (inside vs outside). It’s sometimes important to bring in outside expertise, but you must assure they’re going to do a credible job. Check their references.

What about group size? It depends. If it’s an information sharing session, large group works well. If you’re moving to implementation, smaller groups work well. Group size makes a difference depending on your purpose.

Program timing indicates that summer is the best time. This is an excellent time to do PD. It’s cheaper to do it in the summer, even if you pay them. You can train teachers and subs at the same time. It’s easier to get consecutive days. It removes teachers from their ongoing classroom responsibilities. The major disadvantage is that opportunities for implementation must be delayed. Second in terms of effectiveness is release time during school. The worst time is always after school. Teaching is hard work. You can do follow up and problem solving after school, but don’t expect deep engagement.

Length of training is directly related to the magnitude of change required. Many changes today require structural changes in schools. It isn’t the structure that makes the difference, but what you do within the structure that makes a difference. There’s no evidence to support blocking, looping, etc.

We must change what happens within the structure to make a difference. These may require massive changes by teachers. In a 90-minute period, you must have three distinctly different instructional activities to be effective and make a difference for students.

Good PD looks a lot like good teaching. There’s not a lot of difference between what works for adults and what works for kids. They need opportunities to share, diverse topics, etc.

What about extent and complexity of change? A balance must be met. Mutual adaptation: when any change is involved, people must change to use the adaptation and the change must be flexible enough to let people adapt to it.

One of the things that consistently involves effective experiences is follow-up. The problem is not with the workshops but that they’re one shot. There have to be opportunities for follow-up.

What does make PD effective?
Begin with a clear focus on learning and learners! If this is your goal, this is where you start. In terms of PD planning, the order is reversed (5 Levels of PD Planning). You can’t be satisfied with a single measure here. No single measure tells the whole story. If your goal is to increase the dropout rate, your achievement level will go down. No single measure tells you the whole story. Suppose you want students to consider college as an option. You need them to take more challenging curriculum. As a result, more students will take the ACT and this means your average ACT score will go down. This is what we have to help board members understand. We need to consider multiple indicators. Not everyone trusts the same information. What do you trust? What do you identify as important indicators? Teachers and administrators have almost exactly opposite. Administrators trust nationally normed and state tests. Teachers say these take too long to come back. They trust their own tests to know if they’re making a difference. When you go to block scheduling, you cut discipline 60%--students are in the halls less. This is important, too, but it isn’t an impact on student achievement.

There must be clear goals along with assessment procedures to document progress. Ralph Tyler’s “Basic Principles of Curriculum and Instruction.” You must decide what you want students to learn and you must decide what evidence you’ll accept that they’ve learned it. However, we base student learning on time. We are time based: credit hours, Carnegie units, etc. We measure seat time for students and for teachers in PD. Tyler said, “Teaching is not the sort of thing you can go out in the wilderness and do. We have to see teaching and learning as absolutely linked.”

Solution: 3-step planning:
? How does this activity relate to the school mission?
? What are the intended student learning outcomes?
? What evidence best reflects those outcomes?

Hopefully, your school mission has to do with student learning.

Goals prevent distraction by peripheral issues that waste time and divert energy.

2. Engage in rigorous self-analysis and do it immediately. Ask thoughtful questions about the information. Self-analysis requires the courage to ask tough questions and the skill to find honest answers. Who is learning well? Who isn’t learning well? Why? What can we do about it? This has profound implications for what we do about PD. Working with a colleague may show us that their best is different from my best, so if it isn’t working for me, I should seek the advice of a colleague.

Reaching your goals requires a clear sense of where you are. If half the kids in the class miss an item, it’s a teacher problem. No matter how well I taught it, they just didn’t get it.

Those who make a difference are called “positive deviants.”

3. Recognize change is an individual and organizational process. With respect to Level 3, the principal’s role is vital. They are vital to the success of these efforts. Principals must be present during the entire PD activity. Few have shared with principals how they can do a good job. There are two aspects of individual change: Remember the Stages of Concern (1. Personal; 2. Management; 3. Impact). The first concerns are very personal. What does this mean for me? How much time does this demand on my part? Then they move to management concerns: How can I make this work in my classroom with my kids? How will this affect my grading? My planning? Then they move to impact. What will this do for my students. Most of the planners start with impact, but people are back at the personal level. We must address these at the personal level first. How will you know? A needs survey! This is exactly what you’re doing—addressing the personal concerns and addressing them in concrete ways. What would you say are the biggest obstacles to preventing all of your kids to learn and learn well. It boils down to motivation. The advantage we have is that there is no area in education that’s been studied and researched so well. Kids persist in activities those activities in which they experience success, even some success.

Think about Pac Man. What’s the purpose of gobbling up dots? When you gobble up dots, you get more dots. It’s not the reward. Success is the reward. Every time you play, you get a chance to improve your score. Those who have succeeded, there’s a little song that plays.

“I have some really easy ways for you to do this.” This offers a solution. You don’t say, “I have an additional burden for you to bear.” You must be willing and able to address the personal concerns in very real and concrete ways.

Consider the order of change:
? teacher attitudes and beliefs
? teaching practices
? student learning

We don’t want change for the sake of change, but we want change for improvement. In what order do these changes occur? You have to know this so you know where to put your time and energy. In America there’s a long and distinguished history that the order of change is as above. The problem is that modern research says that isn’t the way it happens. With experienced teachers, it’s teaching practices that impact student learning and this changes teacher attitudes and beliefs. If you try to change their attitudes and beliefs, you’re doomed. If you can get to the stage, “I’ll give it a try,” they’ll see the change in student learning and their attitudes and beliefs will change. If there’s a drastic change in student learning, there’s a remarkable change in teacher attitudes and beliefs. The second thing this makes clear to leaders is that the most vital aspect to PD is not the initial training, but the follow up! This is also the most neglected aspect of PD. This is when questions will be raised and concerns brought to the table. Exactly the same order of change holds for students. Rent and watch, “Remember the Titans.” This is shown directly in this movie.
The person you bring in to work with staff must be seen as credible, and your staff members will judge their credibility very early in the session. The second aspect of credibility is that the ideas presented must be credible: they must have personal meaning and be able to be interpreted to the context of their classroom. The third aspect is you can’t bring someone to the district who is racist or sexist but they get to leave; this person damages your credibility and it suffers more than the consultant’s! If they refuse to give you references or recommendations, go to someone else. You must protect your credibility at all costs. You must follow accommodation, follow-up support and pressure! There’s an implementation dip (Michael Fullan); to get better, you must first get worse. There must be a sensible combination of support with pressure. If we can’t find solutions here, we’ll find solutions outside but we must go on (pressure). Bloom found this is evident in all talent areas. Tennis player example: baseline players vs serve and volley. Someone must intervene so that the player comes up to the net. We like to revert back to the tried and true, what worked before, but we have to keep the pressure up to improve. When problems are experienced, intervene and help the person find a solution.

Evaluating Professional Development
At each of the five levels we must consider:
A. What questions are addressed?
B. How will information be gathered?
C. What is measured or assessed?
D. How will the information be used?

All five levels of professional development evaluation are important. Each level builds on those that come before. Thus, it’s important to gather information at each level. In planning, the order is reversed (5, 4, 3, 2, 1).

1. Participants’ Reactions:
A. What questions are addressed?
Did they like it?
Was their time well spent?
Were the refreshments fresh and tasty?

B. How will information be gathered?
Questionnaires and surveys administered at the end of the session(s).

C. What is measured or assessed?
D. How will the information be used?
To improve program planning, design and delivery. We probably do this one best. No matter how well it was planned, designed and delivered, there will be someone who hates everything you did. 90% of the people will love what you do; 10% will hate what you did.

What do you need to learn next? Assessment-driven instruction.
2. Participants’ Learning
A. What questions are addressed?
Did participants acquire the intended knowledge and skills? If we really want to make a difference with kids, we need a different operating system.
Did participants acquire the intended knowledge and skills? You should be able to identify up front what knowledge and skills you want the participants to acquire.
B. How will information be gathered?
Paper and pencil instruments
Performance assessments/tasks
Adapt the performances to fit the goals. “This year let’s do portfolios?” Why? What will this do for you?
C. What is measured or assessed?
New knowledge and skills of participants
D. How will the information be used?
To improve the content, format and organization of the program

Level 3. Organization Support & Change
A. What questions are addressed?
What are roadblocks? Time, bus schedule, furniture

The factory model looks at reactions, ____, ____, impact

Things break down because of organizational structures.

This is implementation advocated, supported and facilitated!

Follow-up needs to be about once a month. More complicated needs more; less complicated needs less. The principal is critical! This is a structured opportunity to address issues. Set it up, structure it, keep it focused. If program is going to gain school-wide implementation, the results of the early implementers must be shared. The good way is to focus on the results; the bad way is to focus on the people. It’s not who they are but what they do that makes a difference. When you focus on results, it’s evident anyone can get those results.

What about resistors? Shooting them compounds the problem—you have a lame horse with a gunshot wound! Problems need to be addressed quickly and efficiently. People need sufficient resources. Successes are recognized and shared. Small things can make a very powerful difference. Ask them how it’s going. Recognize the extra time people are putting in. Did it impact organization climate?

B. How will information be gathered?
District and school records
Minutes from follow-up meetings
C. What is measured or assessed?
The organization’s advocacy and support

D. How will the information be used?
To document and improve organizational support. If things break down at level 3, intervene at level 3.
To inform future change efforts.

4. Participants’ Use of New Knowledge and Skills
A. What questions are addressed?
Did participants effectively apply or implement the new knowledge and skills?
B. How will information be gathered?
The same rules for rubrics apply. If I can’t tell you what effective implementation looks like, then it doesn’t exist. Include the scoring rubric with the PD. What does effective implementation look like?

5. Results: Student Learning Outcomes
A. What questions are addressed?
What evidence makes most sense to you?
Use multiple indicators. Collect information on a wide variety of things. To a board of education, use anecdotes and testimonials. Present kids to say supporting things. From a technical perspective, these are very subjective. Board members must believe the evidence.
Did it influence their attitudes and behaviors? Are students more confident as learners? Is attendance improving? Is the number of dropouts decreasing?
B. How will information be gathered?
C. What is measured or assessed?
Student learning outcomes: cognitive, affective, psychomotor.
D. How will the information be used?
To focus and improve all aspects of program design, implementation, and follow-up. Ask the stakeholders what evidence they believe, support, trust. Then you’ll know what to look for. Be sure to include evidence of unintended consequences. If you introduce a writing program, make sure other core areas aren’t neglected.

Three things to keep in mind:
We shared lots of information today. If additional questions or concerns come up, use his address, phone number or e-mail address. What are you going to do with the ideas that you presented? Develop an action plan with three things you’re going to do with what you learned today. When we come back together, we’ll ask, What have you accomplished?

1. Consider Stages of Concern.
2. Share information with principals so that we can embed PD in SI plans.
3. Use the “Order of Change” to impact professional practice.