CEPR - Center for Economic and Policy Research

Multimedia

En Español

Em Português

Other Languages

Home Publications Blogs Beat the Press Penalizing Teachers for Low Test Scores

Penalizing Teachers for Low Test Scores

Print
Wednesday, 19 September 2012 04:39

Morning Edition had a segment this morning in which it touted new research from John List, a University of Chicago economist, that found that student test scores rose the most when teachers were given a bonus at the start of the year which they would lose, if their students didn't score at a certain level. The argument is that teachers will more fear losing money they have already received than they value getting a bonus at the end of the year.

While this is an interesting result, there is an important problem with this approach that was not mentioned in the discussion (apart from issues raised about teachers teaching to tests). In general, workers do not like pay systems where their pay can be cut in ways that are unpredictable. (Test results are highly erratic, even the best teachers often have classes that do poorly.)

It is likely that if school systems had pay structures where the pay of teachers who did not meet certain standards was retroactively cut by 5-10 percent at the end of the year, they would have a more difficult time attracting teachers. This means that schools would have to offer higher average pay to attract the same teachers. Whether or not the higher pay offset whatever benefits came from this mechanism for structuring compensation would have to be examined, but it certainly is not obvious that it would.

 

Comments (17)Add Comment
Perverse incentives
written by bakho, September 19, 2012 6:39
The focus on testing and test scores takes the focus off education and causes teachers to redirect efforts to "teaching the test". Broadening "elective" education is reduced. Teachers have maximum incentive to over-prepare and over-drill students to ace the test. This comes at the expense of teaching other things. Plus, the pay is an incentive to cheat. Teachers with good students are dealt a winning hand. Teachers with poor students are dealt a losing hand. This system is fundamentally unfair. When a system is viewed as unfair, compliance will decrease. There are many ways to game the test system. This simply encourages teachers to game the system in whatever ways they can.
Incentives
written by AED, September 19, 2012 8:17
Unpredictability is certainly a problem. But there's a deeper question: What motivates people?

Intrinsic motivation is superior to extrinsic. The latter undermines the former.

The TED lecture "Daniel Pink on the surprising science of motivation"( http://www.youtube.com/watch?v=rrkrvAUbU9Y ) provides an introduction. For a better understanding, see, for example, Alfie Kohn's Punished by Rewards: The Trouble with Gold Stars, Incentive Plans, A's, Praise, and Other Bribes.
...
written by Kat, September 19, 2012 8:46
Shout out to Dean today from hero Diane Ravitch (and she is a hero to me for simply uttering the little heard phrase "I was wrong.").
http://dianeravitch.net/2012/0...he-strike/
Interesting, But....
written by Aaron, September 19, 2012 8:53
The coverage of the study makes it difficult to figure out its context. The Chicago Heights school district appears to offer eight K-8 schools, one at-risk learning center for pre-kindergarten, and an early childhood development center for preschoolers. The study speaks of selecting 150 teachers from nine schools, suggesting the inclusion of the eight K-8 schools and the pre-kindergarten program. Although the study intended to focus on math, among the teachers randomly selected for inclusion were social studies teachers, language arts teachers and interventionists. The study thus included a reading score element, but bases its conclusions only on math scores. It seems reasonable to conclude that the study represents a small first step toward determining the effectiveness of bonus programs, even if first impressions seem promising.

The study, however, raises concerns about how such a bonus plan might work in practice. Yes, a teacher who is at risk of having to pay back money (or have her wages reduced such that the school can recoup the money) may, as the authors theorize, spend additional time with the kids who are having difficulty, so that those kids don't drag down the test scores. But what does that mean for the other kids who, it's reasonable to infer, get less time and attention from the teacher? Also, if the teachers cover multiple subjects (as is common in lower elementary) what is the impact on classroom hours spent on other subjects, as opposed to the subject that drives the bonus? If a teacher sees a bonus about to slip away, how much time and energy will the teacher shift from other subjects and activities onto the subject that drives the bonus?

To what degree are the math skills tested in the earliest grades truly math, as opposed to memorization - and to the extent that a classroom might focus on memorization, does that translate into improvement of the child's capacity and understanding of math or does it function principally to improve test scores? If memorization is playing a significant role, how well do the kids retain the information after the test?

Those questions, of course, can be addressed by larger, more scientific studies, and a focus on schools and grades in which there are specialist math teachers (or, for parallel studies, specialist teachers in other fields who can be tested in their field). But there's another big concern implicated by districts that have implemented high stakes testing: cheating. How do you control for cheating within the context of a large-scale implementation of this type of program?

I agree that people are unlikely to like this type of compensation scheme. I take some issue with the argument, "schools would have to offer higher average pay to attract the same teachers" because we are talking about bonus, not base compensation. If it is a bonus, the school district should be able to offer base compensation no higher than other districts, as by definition a bonus would be given in addition to the base pay.

In the interest of fairness, the same type of incentives should be applies to administrators.
Death might be a motivator too
written by Gary, September 19, 2012 9:17
We could just shoot teachers whose students don't meet certain educational guidelines. That would be an even stronger motivator, and much cheaper than bonuses, negative or positive. And quite in line with how we treat teachers. Uh-oh, this will likely become a real policy suggestion.

Aaron - I think Dean's point is that average total pay (base + bonus) would need to be higher to compensate for the riskiness of a portion of that pay - whether it's structured as a cut from salary or as a bonus on top of salary.

The interesting part is the behavioral economics here - that cuts may motivate better than straight bonuses due to asymmetrical risk aversion. Leaving aside whether the "performance" is real (a big thing to leave aside!), the question is whether the pay-cut regime would end up requiring higher or lower average pay than a bonus regime to achieve the same incentive effect.
Peace of Mind??
written by SS, September 19, 2012 10:04
There are enough crazy or fragile individuals in the world, including teachers, without subjecting people to this type of stress. I would hate to see some of the real beauties I have dealt with over they year worrying about a retroactive pay cut.

Imagine if economists were subject to retroactive, review and adjustment, the profession might dry up.
...
written by Andrew Clearfield, September 19, 2012 11:30
I agree: this pay structure would be a strange and possibly ineffective way to improve teacher performance. A more straightforward approach would be to BLOODY PAY TEACHERS MORE TO START WITH, so that we actually get competent people in the classroom. There is a decent correlation between teacher pay (relative to other jobs in the country) and student performance across the world. Maybe it is not sexy to simply throw money at teachers with no strings attached, but if other countries are a guide, it seems like doing just that would work pretty well.
Testing the testing !
written by Gerry Flaychy, September 19, 2012 11:32
Does each teacher has the same quality and quantity of students as any other teacher; the same means and services at their disposition; the same socio-economic environment ?

Does each teacher has the same formation; the adequate formation and experience for what he has to teach and vice versa ?

Testing the teachers without testing first everything else is pure nonsense.
endowment effect?
written by freebird, September 19, 2012 11:35
This result might be attributable to the endowment effect. If teacher A gets her bonus upfront, she may consider this part of her base pay, while teacher B may earn exactly the same at the end of the year, but would consider the bonus as found money. So teacher A may have the illusion that she earns more than teacher B. Forfeiting a bonus is much less damaging to self-esteem than taking a paycut. Stock traders see this too I think, as it's less painful to miss a winning play than to buy a loser, even though the net delta is exactly the same. Somehow opportunity cost isn't viewed as a loss. This also leads to our asymmetrical views on inflation/deflation.
...
written by Eric377, September 19, 2012 11:44
"(E)ven the best teachers often have classes that do poorly." Really? Seems unlikely, if the word 'often' means what I think it means.
Obvious Question
written by Hugh Sansom, September 19, 2012 12:11
The most obvious question regarding John List's claims doesn't even require a reading of the paper, but only an awareness of prevailing incentive structures in American workplaces. The bonus structure touted (bonus up front with threat of loss) is barely implemented at all in the US. I'm not aware of even one instance. Why? Does List have some reason to think that teachers would be uniquely responsive to this among all American workers?

A couple of other points:

1. Assuming that the List study is actually sound, how would it play out over several years?

2. Even if the study is sound, the question remains of what constitutes good performance and a good measure of that performance.

3. It is striking, as always, how some of the loudest champions of such incentive structures change their tune when the issue is closer to home. Are Chicago profs going to applaud incentive pay tied to student performance? How about performance-based rewards for Wall Street, where bonuses are paid even when performance is abysmal?
My dog ...
written by David, September 19, 2012 4:34
My dog will do many things for a "scooby snack," with great enthusiasm. Just because the dog jumps through a hoop doesn't mean she knows how to do Calculus. Multiple choice math tests are notoriously biased toward socioeconomic factors; poorly written it is very easy for students to mimic knowledge rather than knowing, just by knowing test-taking strategies that they can learn at Sylvan (if their parents can afford to send them there). Should a teacher be penalized because their classes have 60% of the advantages from someone at a private school or magnet school? There's a lot to question here.
A License to Cheat
written by Scott F, September 19, 2012 10:29
The problem with "incentives" is that the more powerful they are, the more incentive there is to cheat - to do whatever it takes (and I do mean WHATEVER) to get the required result. I thought we already learned this from Michelle Rhee's DC tenure.
Good on Ed
written by James, September 19, 2012 10:45
Dean,

You look great, well-spoken, and articulate. Good job.
...
written by urban legend, September 20, 2012 12:22
Talk about "who-gives-a-shit?" research! Imagine the bureaucracy needed to administer a take-back program. Imagine the anger. Imagine the disputes stretching out for years and all the expense of litigators. It ain't never going to happen, so what is the point of it?

Interesting question: How come the charter school effort is exclusively in big cities? Suburbs that routinely have great high schools -- in major part because students are more affluent -- have little interest in charters. I suspect the difference reveals fairly clearly the true agenda of the so-called "reformers (whether in the R or D column)." It has nothing to do with what's best for the kids.
Probability Question
written by matthew, September 20, 2012 1:36
Is somebody out there good at probability math? What are the odds of a teacher getting good or bad test results simply through randomness? Let's say students' abilities (coming into the class) are one through five (five being the best). Let's say you have an average class size of 25 students, with one class per year over 10 years, and that the students are selected randomly in terms of their ability. So what are the odds of a given teacher ending up with particularly "bad" or "good" students (that is, that deviate from the middle score of three)? I hope this question is clear.

It seems obvious that this starting advantage or disadvantage would likely affect test scores.
...
written by liberal, September 20, 2012 7:02
matthew asked
Is somebody out there good at probability math? What are the odds of a teacher getting good or bad test results simply through randomness?


That's a very good question, though it's not so much a question of applying statistical knowledge as it is looking at the data.

I forgot what research concludes, but IIRC if you plot out the data the strength of teacher influence on test scores is actually pretty weak.

Of course, for teachers that's a two-edged sword...

Write comment

(Only one link allowed per comment)

This content has been locked. You can no longer post any comments.

busy
 

CEPR.net
Support this blog, donate
Combined Federal Campaign #79613

About Beat the Press

Dean Baker is co-director of the Center for Economic and Policy Research in Washington, D.C. He is the author of several books, his latest being The End of Loser Liberalism: Making Markets Progressive. Read more about Dean.

Archives