| BASICS
FOR USING STANDARDS AND ASSESSMENTS
Thomas R. Guskey
10/23/2002
Our focus today is standards and assessments. Next time is
grading and reporting. Take a moment for yourself to think
about two issues: given that the session is focused on standards
and assessments, why did you come today? To be successful
for you, what would you like to get out of today?
How do we bring efforts together? How do you change attitudes
about assessment? How do we create an understanding of assessments?
How does assessment drive instruction so that it’s understandable
for everyone? How do we create a common rubric so that writing
can be assessed across departments? How do you merge the two
masters: standards and accountability? Are standardized assessments
more valid than district assessments? How should we report
assessment results? Contextured assessment vs decontextured
assessment…
No Child Left Behind
A lot of what’s being done today in assessment, especially
with respect to reform, happened in Kentucky. Kentucky has
led the way in the nation in educational reform. Kentucky
was the first state to have a statewide assessment program;
it focused on language arts and math, had MC and a few CR
questions. It generated an aggregated score for each school,
and Kentucky used this to rank order all the schools in the
state. This generated all kinds of problems because of the
great diversity all over the state. This created all sorts
of dilemmas. Educational funding is also inequitable. The
state department decided not to rank the schools any more,
but the newspapers did it. A group of 66 poor schools filed
a class action suit to say that school funding was inequitable,
and they won. The state’s system of education was judged
inequitable (1988-1989). The KERA (Kentucky Education Reform
Act) was passed and was very comprehensive. It was the first
reform effort in the country: it dealt with finance, governance,
curriculum and instruction. Standards were established for
all children in six different domains: reading, writing, language
arts, social studies, science, fine arts, practical living.
A statewide assessment was also developed. Five people helped
design the system: Jason Millman, Doris Redman, Pat Forgione,
Edward Keiffer, Grant Wiggins. They developed a phenomenal
assessment system that included three different components:
there were portfolios in each domain; on-demand assessment
with MC and CR items; performance events (a group of evaluators
would come to your school and a group of children would be
selected randomly to perform); non-cognitive indicators—dropout
rates, ADA, and follow up information—were they gainfully
employed, going to school, etc. They had to define success.
They had to draw lines: what is and what isn’t success.
This had to be tied to accountability. Technical issues are
solvable. Accountability is a political issue. They had four
levels: novice, apprentice, proficient, distinguished. All
children were included in the assessments, no exceptions.
If a child isn’t present on the day of assessment, they
give the child the lowest score possible on the assessment.
They averaged the school scores. You were given 20 years to
get to 100%. The first year tests were given was 1992. The
average score for schools was 30%. They bi-annual goal was
an increase based on the difference divided by 10. If you
went beyond this goal, you were awarded up to $3000/teacher.
If you didn’t make it, the principal and teachers could
be fired. Teacher tenure was eliminated in the State. If a
school isn’t making it, there are intermediate steps
and resources available. No school has reached that level
of sanctions yet. The accountability is on the part of teachers,
not students.
California created a similar system. In one school students
didn’t want speedbumps and held the district hostage.
The district lost $56,000 when the students blew off the test.
Children must have adequate opportunity to learn. The Kentucky
system has to do with improvement. If a school does nothing
and stays the same, sanctions are in order. This eliminates
the need to adjust for social, economic or demographic characteristics.
William Sanders in Tennessee thought the Kentucky model was
worthwhile. He started analyzing results in terms of gain
scores and called this “value added accountability.”
It doesn’t matter where you are but how much you gain.
He found that gain patterns were inconsistent and unrelated.
It’s better to be in a rich district than a poor one;
the big difference is between teachers. They’ve taken
accountability to the individual classroom level. He’s
presented this to the chief state school officers three times.
They can attach a name to classroom level results. The teachers
who get good results get good results year after year after
year. The same thing goes for teachers who get poor results.
He found it is somewhat related to who has had advanced training
in their subject areas. It doesn’t matter what students
the good teachers have; they always get good results.
Texas adopted this model. Texas used this to rank order teachers
in a system. They also determined what teachers created the
top third, middle third, bottom third. They examined children
who started at the 40th %ile and if these children had teachers
in the top third, they ended up in the 82% range. The effects
are cumulative. Teachers make a difference! If we’re
looking for models of excellence, we don’t have to look
far. If we’re looking for teachers who need help, we
don’t have to look far, either.
NCLB has similar parameters. We’re testing 3rd grade
through 8th grade and one year beyond. We’re going to
disaggregate the data. Our President and Secretary of Education
are from Texas. We’re absolutely moving in this direction.
Systemic Change
• Change is a highly complex process
• Professional development is essential
Michael Fullan – Leading in a Culture of Change
Baselines are being established in every state based on the
standards set by the state. Schools have until 2014 to reach
100%. In Kentucky, no school has reached the 100% level yet
though some are close.
Change is a prerequisite for improvement! Everything is based
on this premise. You can’t get better without any change.
One of the efforts in the reform effort is to clarify what
we want students to learn: standards.
In Standards-Based Education there are four premises:
1. These ideas are not new!
• Ralph W. Tyler – 1949
• “Basic Principles of Curriculum and Instruction”
• Two fundamental decisions:
a. What do I want students to learn?
b. What evidence would I accept to verify their learning?
Education isn’t based on student learning; it’s
based on time. It’s Tyler who said, “I taught
them, they just didn’t learn it” is the same as
saying, “I sold it, they just didn’t buy it.”
If they didn’t learn anything, how can you say anything
was taught? We have to see teaching and learning as absolutely
linked.
2. The Ideas are more important than the vocabulary we use!
In education today, we’ve become involved in a war of
words, and it’s detracting us from important work we
need to do. Confusing vocabulary:
Objective goal outcome standard benchmark
Competency proficiency performance expectation aspiration
New year’s resolution
It doesn’t matter what you call it! The ideas are more
important than the vocabulary. We must make definite decisions
about what we want students to learn and what evidence we’ll
accept that they’ve learned it.
3. Good ideas can be implemented poorly!
• How do the ideas translate into practice?
• How will we know if they work?
Teachers don’t plan in terms of goals or standards
or objectives; they plan in terms of materials and activities.
What that says is that if we want these things to find their
way into practice, we have to find a way into implementation.
We must be concerned about implementation: how it’s
enacted in practice.
4. Success in education hinges on what happens at the classroom
level. Any successful change will occur at the smallest level.
It doesn’t matter what happens at the national, state
or district level; if it doesn’t happen at the classroom
level, it will fail!
In the 1970s teachers started to be held accountable for
performance on national standardized tests. Teachers started
teaching to the test. When the powers that be learned this,
they started changing the tests to be higher level. This only
brought about slight changes in the classroom.
Carol Ann Tomlinson’s The Parallel Curriculum. EMU’s
Pat Williams-Boyd is strong in differentiated instruction
at the middle level. Deborah Wahlstrom is a good presenter,
very practical, takes the mystery out of using data in the
classroom.
If we want to make a difference at the classroom level,
how can we do this? If we press on these issues on the teacher
and ask what obstacles there are, there are two answers that
are always given:
1. Student motivation
2. Diversity among students
In this case, diversity means the range of skills and abilities
in the classroom. How can we have them all learn well since
they’re starting at such different points?
Student motivation:
There one thing more closely tied to motivation than anything
else: The key to motivation is success! They avoid with a
passion those things that they feel they can’t be successful
at or will fail. Walk into a video arcade. We’re told
in methods classes to make things relevant and interesting.
But Pac Man is about gobbling up dots—it’s repetitive
and isn’t relevant. It’s not the reward but the
success. Every time they play, they can improve their score.
They play with passion and intensity. Success is more important
than interest or relevance.
Diversity among students:
Teachers often lack the time, support and strategies to assist
all learners.
Individualized programs are often impractical in group-based
classroom.
If kids are going at their own rate, how are they going to
learn everything they need to learn. They can go at their
own rate, but by the end of the year, they must be here. Next
year’s teachers won’t be happy if they aren’t
at this place. How can they learn what they’re expected
to learn if they go at their own rate?
Needed: A practical strategy to help all students succeed!
Solution: Mastery Learning – A practical strategy to
help all students succeed!
This helps all students succeed in a standards-based format.
Mastery Learning: An Overview
How to get started – Mid 1960s we really tried to individualize
instruction. As teachers started using them, they discovered
hardships. Bloom came along and he was concerned with the
same issue. He walked into schools and watched teachers teach.
He found consistency across grade levels and subject areas;
there was lots of consistency, even in assessment. They took
the materials and divided them up into learning units that
corresponded with chapters in textbooks. The teacher introduced
and taught a unit, they reviewed, they took a test and went
over the test, then went on to the next unit. How many really
learn excellently? Teachers thought 30%. Students who didn’t
learn the first unit well, didn’t learn the second unit
well, particularly if it was sequential. Bloom found that
only 20% of the students really learned excellently. Bloom
discovered the Bell curve for these students. The Bell curve
is the distribution of achievement in traditional classrooms.
Bloom said, “Hold on! This is the random distribution
of achievement without any intervention!” If we intervene,
we should have more students get better scores. He said, “Teaching
is an intervention. It’s designed to make a difference.”
If we get a Bell curve, we haven’t made a difference.
The basis of mastery learning:
1. Tutoring instructional techniques are the best.
2. Learning strategies of successful students.
With these two sources of information in mind, he went back
to the classroom. He kept the idea of things being divided
into units. At the quiz point, he said we need to make a change.
We’ll use this as a source of information. This will
communicate to kids what’s important to know. He called
it formative.
Assessment information or feedback
1. What students are expected to learn.
2. What each student has learned well.
3. What each student needs to learn better.
Formative should inform the student about what’s important.
Mystery tests teach two things: you can’t trust the
teacher and it doesn’t matter how hard you work. Learning
for most kids is a guessing game. We were good at school because
we were good a guessing what the teacher thought was important.
We need to pair with the feedback those things we deem important.
This is not reteaching, meaning we repeat the same thing but
louder. Bloom suggested a process that looked like this:
Instructional process in mastery learning classrooms
Unit 1 – First formative assessment
Feedback and correctives
Second formative assessment
At the end you’d get a J curve. You aren’t lowering
standards or dumbing down the curriculum. This is called the
mastery learning process.
Unit 1 – Formative Assessment A
Enrichment -> Unit 2
Correctives -> Formative Assessment B (parallel to the
first but not identical) -> Unit 2
Enrichment is a challenging, exciting learning opportunity
for those who get it the first time.
1. One-to-one tutoring is just the process described. You
constantly ask the student questions. When s/he makes a mistake,
you stop and point out the error and try to explain it in
a different way, then ask another question to see if the student
understands before going on. This is what an excellent coach
does.
2. Strategies of successful students
They look at what they do wrong, correct the test, save the
test to review.
Essential Elements of Mastery Learning
1. Feedback, correctives, and enrichment
2. Congruence among instructional components (alignment)
Implementing Mastery Learning is in its second printing and
it’s practical for teachers.
In order to implement this, we must incorporate the feedback
loop. In order for a teacher to incorporate these, a teacher
must offer regular and specific feedback on their progress.
If they do this only once a year, you’re going to lose.
We have to go over assessment results weekly. Any time any
kind of assessment is administered, it’s an opportunity
to learn. How many kids missed each assessment item? If it’s
a great percentage, it’s a teacher problem. If kids
don’t get it, it didn’t work! Whether it works
is defined by what our students can do. We have to set aside
ego issues. If over half the kids in class miss something,
it’s something I’m doing as teacher. Principals
should ask every Friday as teachers leave the parking lot
what they learned as a teacher this week. We should aslk students
every day what they learned in school today; if they reply,
“Nothing,” we send them back into the school.
Correctives must include:
1. Different presentation
2. Different involvement
Children don’t learn in the same way.
Correctives include:
• Different learning styles
• Different learning modalities
• Different types of intelligence
• NOT repetition of the initial instruction
Enrichment must be:
1. Rewarding
2. Challenging (valuable learning experience and need not
be related directly to the unit content)
Best resources can be found in materials for G/T. They’re
gamelike and challenging. They involve high level skills.
G/T – Midwest Publications; Dale Seymour; Games magazine.
Apt initials—take a famous person’s initials and
develop a phrase using the initials. Resignation Made News
= Richard M. Nixon. Wonderkid and Musician = Wolfgang Amadeus
Mozart. After you’re done, take your own initials and
develop a phrase that describes you.
Mastery learning is not: teach > test > reteach >
retest. Don’t use a 2-test format! The feedback corrective
enrichment process is essential. You can use the first test
to predict the score on the second test unless you do correctives/enrichment.
Essential element #2
Congruence among instructional components or alignment.
Teaching and learning are composed of three major parts:
Learning objectives > instruction > feedback and correctives
> competent learners (evaluation)
Among these components there must be congruence. Mastery
learning is not: teaching to the test. Not so for this reason:
If a test dictates to the teacher what is to be taught, then
you must teach to the test. This isn’t the same thing.
Every teacher at every level in every subject level must clearly
identify what’s to be learned. Teaching is directional.
Teaching is taking students from where they are to where we
want them to be. We’re not teaching to a test but testing
what we teach. Every time you give a test, you’re teaching
students what’s important to learn.
Essential Elements of Mastery Learning show:
1. Implementation is flexible
2. Applications are broad
Implementation is extremely flexible; there’s no one
best way to teach it. Format of tests can be very different.
Mastery learning takes the guesswork out of learning. The
formative assessment permits individualization. They only
work on what they didn’t do well on. The correctives
are your ticket to the second assessment. He gives his students
guidance and feedback on their learning errors. He clearly
communicates the standards to his students.
Common Concerns Regarding Mastery Learning:
1. Mastery Learning and Grading Standards?
2. The fairness of a second chance?
3. Mastery learning’s relation to real life?
4. Motivation and the second chance?
If I do this, if I extend my units or instruction, I’m
going to have to limit the amount of content I cover. This
is true in the early units; it will require more time. Students
need time to be oriented to this process. All of this is new.
They’ll ask, Where’s the trick? Some orientation
is necessary. When they first begin, teachers must take time
for the correctives to be done in the classroom under the
guidance of the teacher. If not, they won’t do it. It’s
done in class and under my direction and guidance. I want
them to see they get something out of it. When I first begin,
it’s highly structured. First thing I do with the correctives
group is reteach. When they’re working on something,
I go up and down the rows. Enrichment activities must be rewarding,
challenging, things the students can work on fairly independently,
self-directional, and can be completed within the time I’ve
allotted. After awhile, that structure can be relaxed. Start
out with correctives as 1 ½ hours per unit, then reduce
it to 1 per unit, then 1 per two units, ½ per four
units, and finally I give it to them as homework. Review is
now shifted to the correctives phase. The kids who don’t
need to review can do enrichment. You must see pacing your
units in more flexible terms. My first units are 7 day; my
later units are 5 days. At mid-term I’m one unit behind,
but by the final, I’m one unit ahead!
That’s exactly what Kindergarten’s like: you
get there in different ways and at different times, but you
all get there.
What happens if the child doesn’t make it the second
time or third time or fourth time—if you keep going
back, you’ll teach the same unit all year. Move on.
This process is student-based and teacher-paced. If you insist
on 100% before going on to the next thing, you’re going
for perfection. Kids learn things differently and in different
sequences. You should move on if something isn’t developmentally
appropriate for a child at that time. You need to spiral in
the curriculum. Refer to items you taught in unit one during
unit 2. It’s one way to capture what’s difficult
even if you did move on. Do the same on the exam. If you miss
this, you’ll do correctives on it again. When we move
on, we have to see progress in different ways. Some kids need
to master skills before they can move on; others don’t.
To most kids multiplication is a new thing they have to memorize.
They don’t see it as an extension of addition. Remember
where you want to be at the end and see your pathway and students’
pathways in a flexible way. Multiple opportunities to learn
means you spiral and provide opportunities at different times.
Do I have to change my grading standards with mastery learning?
No, unless you grade kids on the curve = you don’t know
if anyone learned anything. You have to grade based on standards.
The fairness of a second chance is understood by writing
teachers. First draft and get feedback; revise according to
the feedback; rewrite and get more feedback, etc. Am I supposed
to give them the same high grade if they get a good score
on the second test? This isn’t like what life is like
at all. We let all other professionals practice before they
actually do what it is they’ve learned (pilots, surgeons,
etc.). However, we expect kids to perform perfectly the first
time. Can kids learn successfully without making mistakes?
Thomas Edison had over 1000 failed attempts to make the light
bulb. He says he learned 1000 ways how not to make the light
bulb. These are the kinds of skills we want to develop in
lifelong learners.
If I tie it to the grade, they’ll try harder. Did you
pass your driver’s license test on the first time? What
do we say to those who don’t? They can’t drive
in the rain? Maybe the reason they didn’t get it the
first time isn’t there fault; maybe it was mine? What’s
the motivation for the first if they get a second? Bloom emphasized
enrichment. Getting the certificate of mastery proved motivational.
It’s tangible recognition of success! It means someone
recognizes they’ve done a good job.
Another concern that was raised is, does this imply we have
to start every unit with a pre-test? Bloom said not to use
them. Pre-tests have value if we use them at the start of
the school year. Don’t use them at the beginning of
every formal unit. It only confirms what the teacher already
knows (they don’t know this material). It also starts
their experience with a failure. Use the pre-test as formative
A and the post-test as formative B, and your concern is how
to construct the enrichment and correctives in between. Rather
than Monday, give the pre-test on Wednesday after two days
of teaching and the post-test on Friday after days of enrichment
and correctives.
If we’re going to use this well, here are the steps
I think we should take. Discuss.
Developing a Table of Specifications (Outlining Learning
Goals)
Two Important Questions:
1. What do I want students to learn?
2. What do I want students to do with what they have learned?
When you look at implementation, think big but start small.
Start at the individual classroom level/school level. If you
approach implementation voluntarily, not everyone will do
it. If you get 30-40% to do it, you’re doing well. Those
who are likely to volunteer are the best, most effective,
the most talented you have on your staff. Those most likely
to resist are those who don’t want to change anyway.
Go with the thoroughbreds. Publish their results. You won’t
get up to 100% even in the 2nd year. You’ll still have
holdouts in the 3rd year. Three years into the process 90%
are using it. The principal/department or grade level chair
sits down with them. “I think you should get similar
results.” Share the results of the teachers who are
using these techniques. Otherwise, this person is hurting
kids; isn’t professional. Not everyone will have the
will at the start. Make sure you collect the evidence! Think
five years down the line.
One of the major challenges you’ll face: you have standards,
goals, benchmarks. How do you translate these into instructional
practice? In instructional planning, ask teachers “How
far will you go before you’re sure students are with
you?” In this unit, you must ask, “What do you
want students to learn? What do you want students to do with
what they have learned?” Use the Table of Specifications.
Are there any vocabulary terms they’ll need to learn?
What facts do you want them to learn? What rules & principles
do you want them to learn? What processes & procedures
(a series of steps) you want students to know as a result
of this unit? The first four columns answers, What do you
want students to know?
What do yu want students to do with what they know? Do you
want them to explain it in their own words? This is translations.
Give them a paragraph and pick out errors = translations.
If they write the paragraph and pick out the errors = application.
See what we’re doing in math that has relevance in science
= analysis and synthesis. This is Bloom’s taxonomy.
Teachers use their instructional materials as a guide to develop
this instructional table. This will only fill out the first
four columns. Teachers get nervous when they see empty boxes,
so they fill out the “translations,” “applications,”
and “analysis and synthesis” boxes themselves.
Marzano refers to this as unpacking the standard. This is
a unit plan. The most important part is to pull out the assessment
you use for the unit and see how well it matches. This is
a very humbling experience. They’ll discover that all
of their assessment is in the knowledge area. They see that
they have to change their assessments to more performance.
This helps bring about congruence. You don’t have to
have something in every column! Sometimes early units are
heavily laden with lower level skills. Later units deal with
problem solving and not vocabulary, for example. This is a
basic framework for bridging the gap between what the standards
and benchmarks provide and the materials deliver and the assessments
evaluate. This helps us unpack, combine units. This is the
way teachers plan.
Guidelines for Success
If assessments are going to make a difference, if you look
at schools where they’ve used assessments to make a
difference, here’s what they’ve done:
1. Think big, but start small. Don’t require too much,
too soon from teachers and administrators. Do not try using
these ideas in all the subjects in the very first year because
if you do, you’ll die! Have a clear vision of where
you want to be 3-4 years from now. Start with this first year,
add something else second year, add something new third year.
Teachers consistently get better results the second year as
they work the bugs out. Have a vision, long term.
2. Ensure that assessments become an integral part of the
instructional process. Quizzes and tests should be learning
tools, not simply evaluation devices that mark the end of
learning.
Implication #1: Assessments must be sources of information
for students and teachers.
Implication #2: Assessments must be followed by high quality
corrective instruction. Just going over the test isn’t
enough.
Implication #3: Students must be given a second chance to
show improvement! It gives students a second chance at success.
Quote: “Spectacular achievements are always preceded
by unspectacular preparation.” Roger Staubach. You don’t
get a second chance in the Super Bowl, but up to that point,
you do.
3. Link new assessments with existing classroom assessment
practices. Blend traditional approaches with alternative assessments.
This year we’re going to do portfolios. Why? Make sure
form follows function. Adapt assessments to fit the learning
goals/standards/whatever you’re trying to achieve.
A complex problem:
2/3 + ¾ = 5/7
Assessment formats – Traditional assessments (1. T/F;
2. Matching; 3. Multiple choice; 4. Completion; 5. Essay/short
answer.
Do you really understand the concept of adding fractions
with unlike denominators?
No matter how clever we are, students will always give us
answers that are difficult to score. There is no such thing
as a perfect assessment.
Essay/short answer are now called constructed responses.
Traditional assessments all have advantages and disadvantages.
1. T/F are good, particularly if you have low level skills.
Put T/F in the left hand margin rather than have them write
a T or an F. They’re restricted to low level skills.
Don’t ask kids to correct it to make it true. They can
guess and get 50% right.
2. Matching. They’re restricted to low level skills.
One important aspect of matching: how you format them on the
page is important. Don’t put answers on the left. We’ve
taught them to read left to right. They can read a long response
and scan a list of words. It takes 1/3 the time. The less
time you take for this, the more time you have for instruction.
Publishers usually format them wrong. If it’s formatted
by the publisher wrong, work backwards. Read the long series
and scan the long series of words.
3. Multiple choice: Include common errors to diagnose learning
problems. If a child answers 1.2 = .23 = either 3.5 or .35,
the child has a problem with placement of decimals. You’re
not trying to be tricky but to diagnose a problem.
4. Convergence theory: Stephen Douglas, Abraham Lincoln, James
Monroe, Robert E. Lee. What’s the question? Often just
by analyzing the relationship between the answers, you can
identify where they converge. Some share a Civil War history.
Some share the presidency. Lincoln has two of these, thus
he’s the answer. High school students can be trained
in convergence theory. They can get 60-70% right without having
the questions! The usefulness as a learning tool is dependent
on immediate feedback. They can get this by creating a double
answer sheet. As they take the assessment, they write the
answers twice. They hand one in. We go over it immediately
and they correct their papers. They know what they got right
and what they got wrong. They know who’s in enrichment
and who’s in correctives. Teacher makes an assessment
analysis (# of errors/item). First look at the questions themselves.
Are they poorly worded? If no, then I didn’t teach it
very well. If many students chose the item in error, they’re
guessing. This means I need to make a change here.
Alternative assessments:
5. Skill demonstrations
6. Oral presentations
7. Task performances and complex problems
8. compositions and writing samples
9. laboratory experiments
10. projects and reports
11. group tasks or activities
12. portfolios
The key to success with alternative assessments: Clearly
specified performance criteria or scoring rubrics. These need
to be clear enough so that the student knows how he can get
a 10 (of 10).
Rubrics:
1. List the criteria for a piece of work or “what counts.”
2. Articulate graduations of quality for each criterion from
“Excellent” to “Poor.”
Heidi Goodrich. The magic numbers are between 3-5. Understanding
rubrics. (1996) Educational Leadership, 54 (4), 14-17.
Simple guideline for developing graduations of quality:
4 Yes
3. Yes, but…
2 No, but
1. No
Why use rubrics?
1. They’re powerful tools for teaching and assessment.
2. They help students become more thoughtful judges of their
own work.
3. They reduce the time teachers spend evaluating students’
work.
4. They allow teachers to accommodate differences in heterogeneous
classes.
5. They’re easy to use and explain.
6. They improve objectivity in scoring.
Tips for designing rubrics:
1. Begin with models of excellence
2. Avoid unclear language (e.g., “A Creative Beginning”)
3. Avoid unnecessary negative language
4. Involve students in the process
|
