Measuring Teacher Effectiveness

I have a new paper on teacher effectiveness out today. So far as I know, it’s the first ‘value-added’ study to be conducted outside the United States.

Estimating Teacher Effectiveness From Two-Year Changes in Students’ Test Scores
Using a dataset covering over 10,000 Australian primary school teachers and over 90,000 pupils, I estimate how effective teachers are in raising students’ test scores from one exam to the next. Since the exams are conducted only every two years, it is necessary to take account of the work of the teacher in the intervening year. Even after adjusting for measurement error, the resulting teacher fixed effects are widely dispersed across teachers, and there is a strong positive correlation between a teacher’s gains in literacy and numeracy. Teacher fixed effects show a significant association with some, though not all, observable teacher characteristics. Experience has the strongest effect, with a large effect in the early years of a teacher’s career. Female teachers do better at teaching literacy. Teachers with a masters degree or some other form of further qualification do not appear to achieve significantly larger test score gains. Overall, teacher characteristics found in the departmental payroll database can explain only a small fraction of the variance in teacher performance.

The results imply that a 75th percentile teacher can achieve in three-quarters of a year what a 25th percentile teacher can achieve in a full year; and that a teacher at the 90th percentile can achieve in half a year what a teacher at the 10th percentile can achieve in a full year. For what it’s worth, I don’t think this dispersion is wider than what one would find among plumbers, dentist, architects or bricklayers. But it does indicate that – at least as measured by test score gains – all teachers are not created equal.

One context in which this might especially matter is with regard to Indigenous children. In the paper, I estimate that Indigenous primary school students perform approximately two grades below their non-Indigenous counterparts. Assuming that the impact of having a more effective teacher persists over time, and that Indigenous children typically get teachers at the 25th percentile, these results imply suggests the black-white test score gap in Australia could be closed in seven years by giving all Indigenous pupils teachers at the 75th percentile.

About these ads
This entry was posted in Economics of Education. Bookmark the permalink.

46 Responses to Measuring Teacher Effectiveness

  1. invig says:

    While the results are interesting, and even excellent (in terms of opening up awareness) the problem is the assumption that the response is to pay people more who are better. What if the reward teachers received for performing well was actually detrimental to the performance of the school overall?

    What if those teachers who are better don’t want to get paid (and therefore known to be paid) more than their fellow teachers? What if they would rather see benefits such as official recognition, choice of class or influence over their schools’ policy?

  2. Bill says:

    Andrew,

    Thanks for the continuing focus and quality in what you do. I find your recent posting on estimating teacher effectiveness, compelling reading.

    Bill

  3. Panadawn says:

    Invig, in every other area in the modern economy it is almost universally recognised that paying more for superior performance encourages people to be more productive.

  4. Stephen Hill says:

    I found this was surprising.

    “Teachers with a masters degree or some other form of further qualification do not appear to achieve significantly larger test score gains. ”

    I would have thought (but maybe this would be concentrated at secondary level), teachers with extra study through a Masters would have attained an increased mastery of a subject area that would have flowed through to improved communication of difficult concepts and more flexible lesson design.

  5. Tim Condren says:

    If we take test scores as the determiner of teacher quality and therefore pay scales, how do you prevent, as has happened in England and the United States in recent years (and in Australia in the 1880′s) teachers cheating and teaching to the test to ensure they receive their increased pay scales?

  6. Andrew Leigh says:

    Stephen, the Masters result is the same that other studies find for Texas and New Jersey. I agree that it might be different at a secondary level.

    Tim, this an excellent point (and one that Peter Martin has worried about). I agree that a perfect merit pay system is unattainable – but the question is whether we can do better than the status quo.

  7. Mercurius says:

    Those who support the status quo pay system would actually find a fair bit to support their position from this paper:

    1) The current pay system stops pay increases after about nine years, unless you are promoted to head teacher. P.16-17 of the study concluded that after about 10 years’ experience, there are marginal improvement to teacher performance for each extra years’ experience. Broadly speaking, teacher experience has the greatest effect over years 1-9, corresponding to the years in which their pay increases under the current system. That said, seniority does seem to leads a continual gradual improvement, so seniority is still looking like a good basis for awarding pay increases (see graphs on pages 16-17).
    2) Unless I’m missing something, it seemed that the QLD Department of Education’s (DETA) own system of rating quality teachers (1= Outstanding applicants 2= Quality applicants etc.). was also tied to the better test results. This suggests that (perish the thought) it’s the bureaucrats in the department who can best select the wheat from the chaff. Again, this is a support for the status-quo, and rather goes against those calling for Principals to have hire-and-fire powers or for parents to get involved in rating teachers. Hmmmm…
    3) The study detected an effect in which female teachers did better for literacy. So are we going to stop hearing from the lobby who whinge that “boys’ education is suffering” because of the preponderance of female teachers? No, thought not.
    4) I thought it was a bit unfair of the newspaper reportage to pick up on the Masters degree as not leading to a detectable gain. As the author points out, the study had no way to check if those teachers’ own performance improved as a result of getting a Masters degree, so for all we know teachers who have a Masters degree are better teachers than they might otherwise have been, even if they don’t stand out from the crowd. Also the author did not acknowledge the fact that many Masters-qualified teachers are in fact novice teachers themselves, which the study shows could not on average be expected to stand apart. (Disclosure: I do not hold, nor am I contemplating, a Masters teaching qualification).
    5) The study’s methodology is not applicable to high-school teachers. In high school, students swap teachers continually throughout the day. This fact alone would confound the applicability of these results to any high-school context. It would not be possible under this study’s methodology to work out who should get merit-based pay increases in a high school. Would anybody care to take up the cudgels to argue for a return to different pay systems for Primary and High School? Good luck with that.

    So, the picture that emerges is one which experience (ie. dreaded seniority!!) counts for pay increases, the Department knows best, and women make the better teachers…
    :-)

    Just sayin’, is all.

  8. Tim Condren says:

    Thanks Andrew I agree that we need a better system, but my fear is that the effort and funding required to implement a properly researched and detailed pay system that takes in to account all the variables is beyond the political will of Governments who only want a simple test based system attuned to the electoral cycle.

  9. wpd says:

    Andrew didn’t you find that:

    “Teacher experience is positively correlated with teacher effectiveness”?

    Isn’t it the case that teachers in Australia and elsewhere are paid on the basis of years of service?

    Doesn’t your study suggest that this approach is on the money?

  10. David Zyngier says:

    Andrew where was your paper presented? Is it peer reviewed? Hattie’s work (2002) also found that as Lingard et al (2001) states ” teachers have the ability to tip the balance”. We know that excellent teachers make a difference, but teachers are not THE only problem in relation to student outcomes.

    David Zyngier Monash University

  11. John Butcher says:

    Hi Andrew

    Very interesting and though provoking work. It generally accords with my impressions of the Australian public education system which seems at times to subordinate achievement of educational objectives in favour of manging industrial relations and fiscal policy objectives. In particular, it strikes me that the public education sector is a last bastion of centralised IR and HRM management. In virtually every other area of government activity HRM has been devolved to the business unit level – including (importantly) responsibility for managing performance. Whilst school principals have HRM responsibiltiies, they do not have the capacity to reward or recognise or otherwise manage for performance. Neither school boards nor teachers have much control over recruitment or placement. Teachers are allocated and rotated by head office edict – which seems to me to signal a failure to address the performance issue (ie, it’s about moving the poor-performers on rather than confronting the situation). As for the issue of performance pay for teachers, I think that misses the point. The real issue is about ‘performance management’ – and remuneration is one of a suite a strategies to encourage and reward performance. The question I’d ask is whether principals and Boards have the capacity and capability at present to effectively oversight performance? Certainly, if HRM were to be effectively devolved to the school level some ‘capacity building’ would have to occur. Have there been any comparative studies between, say, Australia and New Zealand to assess the effectivenes of financial and industrial devolution in education? Anyway, thanks for ‘stirring the possum’!

    John

  12. invig says:

    Invig, in every other area in the modern economy it is almost universally recognised that paying more for superior performance encourages people to be more productive.

    Panadawn, I understand that, more often than not, this is certainly the case. What I am saying is that the teaching environment may require a more subtle approach.

    I mean, if you want Mother Theresa to work harder, what do you do?

  13. Mercurius says:

    Hi Andrew

    Have my comments been lost in your spam trap? I spent a good hour or so reading your study this morning and made (what I thought was) a series of fairly measured observations about the results. I’d be grateful if they could be reproduced on your site for consideration. I submitted them I think between 8:00am-9:30am today. Is your spam filter gummed up?

    Best regards

    Mercurius

  14. backroom girl says:

    “Assuming that the impact of having a more effective teacher persists over time, and that Indigenous children typically get teachers at the 25th percentile, these results imply suggests the black-white test score gap in Australia could be closed in seven years by giving all Indigenous pupils teachers at the 75th percentile. ”

    Andrew – do you have a theory why it is that indigenous kids (and I suspect also underprivileged) non-indigenous kids get worse teachers to start with? If it is because better teachers don’t want to (and don’t have to) work in “bad” schools (or schools in bad areas) in the first place, can this problem really be solved by performance pay per se? Or is the problem that, if you are a teacher in Sydney you get paid the same amount whether you are a teacher in a nice or even very nice suburb as you do if you are in the worst suburb in Sydney. To say nothing of having to worry about where your own kids are going to go to school, if you do the humanitarian thing and agree to go and work in the bad locations.

    So what would you think of paying all teachers more to go and work in bad areas in the cities, in small country towns and/or indigenous communities? Do you think that would help raise the general standard of teaching on offer to disadvantaged kids in general? I would guess that the teachers unions would be strongly opposed to such a policy, but maybe I’m wrong about that. Maybe they would prefer it to performance pay, in the end.

  15. Grugs and Harro says:

    Andrew,

    We are keen to know what happened to the comments submitted by ‘Mercurius’. From his posting, it seems that he has made some pertinent observations about your study. It may well be that we share these concerns (eg RE research design, statistical results, theory formulation, data interpretation, conclusions and policy recommendations).

  16. Andrew Leigh says:

    Mercurius, thanks for your comment, which was caught by my Akismet spam filter (I have no idea why – it has its own odd algorithms). I’ve now rescued it. As to the substance of your comment, my simple response is that yes, experience and departmental ratings matter – but they still explain for only about 1% of the differences between teachers.

    David, the paper is not peer-reviewed, though I’ve presented it in 4 seminars/conferences (see bottom of the page 1 for details). I understand that disseminating unpublished work is not the norm in some disciplines (medical researchers in particular would frown at it), but it’s certainly standard practice in economics. This is an issue I’ve been thinking about for a while, and might write a blog post on at some point.

    BG, I agree with your theory – if salaries are the same, teachers will choose schools based on compensating differentials. And yes, I think a system of paying loadings for disadvantaged schools would be sensible (some states already do this a bit – including WA, I think).

  17. Grugs and Harro says:

    Andrew – could you please post the comments from Mercurius?

  18. John says:

    Andrew

    You have admitted yourself that your study may not have any correlation with teacher effectiveness and extra qualifications (masters, doctorates) in high school situations; – Does a teacher who has extra qualifications in a particular discipline actually result in increasing the knowledge of high schools students in his/her subject? This is untested by you at this stage.

    My concern however is that the minister for education is now gaining evidence based research (from your study) in supporting her stand for performance based pay for all rather than paying teachers with higher qualifications despite your study investigating primary schools and not secondary. Yet the minister will use your study as evidence to make a blanket rule for all, without accounting for any variables.

  19. Andrew Leigh says:

    Grugs and Harro, when Mercurius’s comment reappeared, it reappeared above. To be precise, it’s here.

  20. David Balfour says:

    I’m not sure why Backroom Girl thinks that a teachers’ union would be opposed to paying teachers more to go to what used to described here (WA) as Difficult to Staff Schools. That description has now been replaced by the Country Teaching Program and the Metropolitan Teaching Program. The State School Teachers’ Union of WA has negotiated – over quite a few years now – financial and professional incentives to go to these schools. Currently, the financial benefits range from around $750 a year to $8 500 a year depending on location, etc.

    In addition, there is the Remote Teaching Service where, again depending upon location, the additional salary ranges from $10 000 to $15 000 a year.

    Regards

    David Balfour

  21. Andrew Leigh says:

    John, I’m only aware of one study of this type that looked at high schools. It found that teachers with a Masters or PhD did no better in raising student scores.

    Standard education background characteristics, including certification, advanced degrees, quality of college attended, and undergraduate major, are loosely, if at all, related to estimated teacher quality.

    I haven’t caught what the Minister has said today, but I’ve tried to be honest about the limitations of my study in regard to policymaking. I’m not an evangelist for merit pay, but I do think it’d be worth running some carefully constructed trials of merit pay programs, to test the claims of the proponents and detractors.

  22. Bruce Bradbury says:

    Andrew,

    I have some concern over the interpretatation of the ‘fixed effects’. I don’t think these concerns are a problem for the conclusions about the characteristics that might be associated with good teachers. However, I think they are relevant to your interpretation of the variation in the fixed effects. For example, when you talk about moving from the 25th to the 75th percentile of teacher ability.

    The first is a ‘class effect’. If a given group of students ‘click’ and hence do well over a given two-year period, will this be included as part of the teacher effect? Second (and related), how ‘fixed’ are the teacher fixed effects. For example, if teachers tend to perform well one year and poorly the next (possibly because they work with some students better than others) will this show up as a variation in teacher fixed effects?

    If so, teachers at the 75th percentile might not continue to be at this level over time – and so it is not really sensible about replacing a group of 25th percentile teachers with 75th percentile ones.

  23. Kevin Cox says:

    A small anecdote that I may share with you that may (but probably doesn’t) shed some light on why Masters degrees do not appear to make better teachers.

    I recently attended a workshop on working out the value of early stage companies. One valuer said he had several rules of thumb. One of which was:
    The value of a high tech company is the number of PhD’s multiplied by 1M minus the number of MBA’s multiplied by 500K

    Perhaps the problem is with what is taught in a typical masters program where the emphasis is often on learning not research. Perhaps this is a message to the developers of Masters programs.

    It would be interesting to see whether PhD’s make better teachers.

    I don’t think we should concern ourselves too much about the economics and pay of teachers. What we should concern ourselves with is what are the characteristics of good teachers that make them so good. My guess is that desire for a higher salary is way down on the list of characteristics. Perhaps it is high levels of empathy, curiosity, child like behaviour, and being fun to be with. That is, before we go looking a performance based pay let us look at what it is that makes a teacher great and forget about pay and “power”.

  24. Ken Lovell says:

    ‘Invig, in every other area in the modern economy it is almost universally recognised that paying more for superior performance encourages people to be more productive.’

    This is certainly true in the sense that it passes what our prime minister calls ‘the common sense pub test’. However that does not mean it has been empirically substantiated.

    For extra pay to cause improved teacher performence, several conditions would have to be met:
    - teachers would have to know what factors contribute to performance – something which is poorly understood but which Andrew’s findings suggest is not related to credentials
    - these factors would have to be within the control of teachers (for example, to the extent that performance depends on innate characteristics or on experience, it is beyond the ability of a teacher to change it)
    - they would also have to believe that these matters were within their control
    - teachers would have to value the additional rewards on offer more than they value the cost of doing whatever is required to achieve the improved performance stipulated by the performance pay scheme – even assuming they know what it is that they have to do.

    All these issues need to be carefully investigated before anyone can draw meaningful conclusions about the probable effects of linking pay to performance.

  25. Garry Grant says:

    A first gut reaction as a practicing classroom teacher from reading the press in SMH early this morning was – “Is good teaching solely about raising test scores? What type of classrooms/teaching methodology/pedagogy could we end up with should the salary of teachers become dependent on raising test scores?” History suggests they may not be pleasant places to be!! Having had this as a first reaction to press I’m about to have a read of the paper, as it does raise interesting issues.

  26. Lorelei says:

    Another gut reaction, or rather a question:
    Where does the funding for your study come from? Was it funded by the Commonwealth government in any way?

  27. Pingback: Club Troppo » Missing Link: YHBT. YHL. HAND.

  28. wpd says:

    “if salaries are the same, teachers will choose schools based on compensating differentials”.

    Where, pray tell do beginning teachers get to choose schools, at least in the public system? Certainly not in Queensland IMHO.

    Or am I mistaken?

  29. Chris Curtis says:

    Beginning teachers in Victoria choose their schools and have done so for about ten years. In fact, all teachers do, in that Victoria has had a highy inefficient local appointments “system” for that long.

  30. Nik says:

    I think your paper is quite thorough overall, the main concern I’ve heard on the news is that the tests your study are based on are quite narrow. Maybe addressing these concerns or acknowledging limitations might help.

    I hope the next progression from your research is how to design a system that identifies those teachers whom are better at improving students and putting them where they are needed most i.e. indigenous communities. Especially given that these teachers may be reluctant to leave a high paying job (based on performance pay) in a nice school to go to a remote disadvantaged community. In this sense I think what the person above said about giving salary bonuses for teaching in remote areas is a better go than a system based on performance pay, unless schools with indigenous pupils whom are far behind the state average are given very large budgets to allocate to getting the best talent (which raises issues about whether they should spend on infrastructure and other areas in desperate need).

    This study is making the point that teachers play a huge role in determining student performance and readily observable factors such as qualifications and experience are not the best determinants of a teachers’ productivity. To follow on and say that designing a system based on performance pay will best allocate/ better allocate than the present system teachers to where they are needed most is a big jump lacking theory and evidence and is not dealt with in this study. In fact from the example I’ve presented above and I’m sure if people think it through in their mind unless performance based pay is going to be implemented along with redistributions from rich schools with pupils well above average to poorer schools with students in dire need of good teachers performance based pay could actually reinforce the inequalities we observe in society today rather than help fix them.

  31. Ken says:

    Yes…interesting that an economist purports to be a credible source on teaching.

    Even more incredible I think, is the assertion that ‘one can assume that indigenous students get the worst teachers’ or words to that effect…morem accurate than that is the fact that schools in the NW of WA (i.e. schools with high indigenous student numbers) are also the most transient, and usually lower on average in socio-economic terms.

    The transience, in particular, makes the schools less likely to score well on ‘standardised’ tests, such as WALNA, in which the best scores are achieved by students that learn to suit the test results….usually by teachers experienced in what those test questions will be.

    Ask the teachers, not an economist.

  32. ChrisPer says:

    Ken, teachers have spent several decades being ill served by politicised education theorists, and un-repeatable classroom ‘experiments’. Measurable results which are repeatable can make a difference.

    The hard part is that variables are not independent. Schools with high violence against teachers and other students will score a teensy bit lower, and that is partly dependent on bad family life, which is heavily dependent on substance abuse in the community. These make a situation where teachers are either incredibly altruistic, or forced to be there by the system. the forced ones are pretty transient themselves, new and less experienced.

    Blaming teacher quality (and by implication, pay) for results when social problems mean attendance is very bad and the violence a constant danger, is unreasonable. But it is very natural that they get some lower quality teachers – good quality ones don’t have to go there.

  33. Andrew Leigh says:

    Garry/Nik, I agree with the point about literacy and numeracy not being all that matters in education (Pat Byrne made a similar argument on SBS last night, though she was rather less polite towards me… something about not living in the real world…). But literacy and numeracy scores do matter a lot – for example, we know that those with better literacy and numeracy are more likely to earn higher wages, and less likely to be unemployed.

    Lorelei, the study was funded by an Australian Research Council grant. So it’s federally-funded, but without strings attached.

    More generally, thanks for the many thoughtful comments on this thread. I really appreciate the feedback, and the fact that those who disagree vehemently with the study have been polite in their critiques.

  34. backroom girl says:

    I’m with Nik in that I still think the important issue is how to actually get good teachers teaching the kids who need good teaching the most. While I’m sure it’s true that teachers don’t teach just for the money (that seems self-evident), I think that extra money might still be needed to enable some people to exercise their altruism and take on the extra challenge involved in teaching in a bad school. Not to say compensate them for the extra effort and other disadvantages that they and their families might face as a result. Performance pay could then come on top of that, I guess, but by definition it only comes after the results have been produced, so it’s not likely to be the solution to getting people to willingly take on the hard assignments in the first place.

    I’m pleased to know that WA at least has a program to pay teachers more in difficult schools (didn’t see any other State mentioned though) but what struck me about that comment is that some of the extra amounts didn’t seem all that high – an extra $750 a year to go teach in a bad suburb? If I was a teacher, I don’t think I’d be tempted.

    A few people also made the points that it is beginning teachers who get posted to the country and many of those don’t stay. All the more reason, I would have thought, to have policies that might induce more experienced teachers to actually choose to take up such positions.

  35. Chris says:

    Better teachers achieve better student outcomes. It’s a ‘no-brainer’ really, so what is all the hype ?

    Even if the ‘estimations’ are correct, does it lead us anywhere other than more standardised testing for students and a narrowing of the curriculum?

    The community’s expectations that a full and positive school experience provided by accomplished teachers for students is manifestly more substantial and broader ranging than the notion of assessing teachers through student outcomes on the national testing benchmark agenda is entirely missed in an econometric perspective.

    A number of fundamental questions remain unexamined in this research. I would encourage readers to look at the Independent Education Union of Australia website at http://www.ieu.org.au

    Despite comments by some contributors, education unions have been attempting to progress reward for highly accomplished teachers for over 15 years, but reject the repeatedly failed processes (both in Australia and the US) such as the use of standardised testing.

  36. derrida derider says:

    I’ve got no problem with Andrew’s study – just with how it’s being wilfully misinterpreted by both sides (I see the teachers unions have joined in this morning).

    Andrew has looked closely at what correlates with primary teachers’ success (granted, we can argue about the definition of “success”). He has demonstrated that these associations are strong. But his study says absolutely nothing about what the effects of a merit pay system would be on these correlates, let alone what the effects on actual outcomes would be. Ken Lovell above highlights the sort of things we’d need to know before designing such a system.

    Unfortunately I don’t reckon randomised trials would tell us what Andrew thinks they will. As it can’t possibly be blind there’s a strong risk of a Hawthorne effect for a start. The trials would also have to be very long (you need longitudinal data, and you particularly need to know whether any apparent “success” persists in later life to do cost-benefit calculations). Apart from making the ethical and funding problems much worse, this means there would be a very high risk of differential attrition and recruitment (for both students and teachers) leading to hidden selection bias.

    Randomised trials are no panacea for lots of policy evaluation problems precisely because of these sort of issues.

  37. ChrisPer says:

    One more point -
    It is good that ‘disadvantaged’ students get good teachers. It is bad to deny good teachers to the people that can benefit from them, ESPECIALLY the best students.

    What is needed more is a way to raise the performance of many teachers, so the disadvantaged schools get a better-performing pool to draw on. The best will always be drawn to the schools that reward them, and high-achieving students that show the benefit of teaching are one of the best benefits a teacher can receive. Better facilities, more supportive and effective headmasters, less bureaucracy and better living conditions are parts of the reward structure that motivates teachers.

    Measuring teacher performance objectively will allow teacher development to become much more effective.

  38. Lyle Dunne says:

    Andrew,

    Thanks for the paper – if nothing else it’s a useful reminder (“reminder” on the assumption that everyone already knew) that teachers are not all the same, and teacher quality is an issue.

    I’m curious about whether teacher quality – or perhaps I should say literacy/numeracy teaching ability – is generalisable, or is related to the relationship between teachers and pupils. You analysed the effect of M vs F teachers – did you (and could you) look at whether it matters whether it matters whether the teacher is of the same sex as the pupil; (the “role model” hypothesis?) Personally I’m sceptical of this as an explanation for outcome differences but it would be worth testing I think.

    A related issue is that of Indigenous outcomes. You say:

    “In the paper, I estimate that Indigenous primary school students perform approximately two grades below their non-Indigenous counterparts. Assuming that the impact of having a more effective teacher persists over time, and that Indigenous children typically get teachers at the 25th percentile, these results imply suggests the black-white test score gap in Australia could be closed in seven years by giving all Indigenous pupils teachers at the 75th percentile.”

    It’s not clear whether you’re assuming that the differential is (mainly) a result of teacher quality – presumably not – but either way you’re assuming better teachers would remedy the deficit. This is not self-evident: it may be that there are other environmental factors which limit the performance of Indigenous students, effectively imposing a “ceiling” on outcomes. Also (though this doesn’t detract from the paper) what constitutes a good teacher, in the sense of one who maximises test-score gain, may not be the same for indigenous communities.

    (I’m probably making too much of a point of what seemed to be mainly an illustration of the magnitude of the effect – but it won’t necessarily be read thus.)

    Finally, on higher qualifications, it may be the case that a Master’s degree makes someone a better teacher (or identifies someone as a better teacher) – but also makes someone much more employable in a range of other jobs which pay better. So it may be that the most talented of Master’s graduates end up leaving teaching.

    In which case it might make sense to encourage teachers to get higher degrees, but only if we could ensure they would stay in teaching – and if we know it isn’t just a mechanism for identifying the better teachers.

  39. Kevin Cox says:

    Derrida Derida is correct in saying that Andrew’s latest paper says nothing about the issues raised by many correspondents (including myself).

    In defense of the correspondents Andrew is on record as suggesting that good teachers should be paid more and this is remembered by readers of his latest paper not least the Minister for Education (or her advisers) who on national TV said that Andrew’s paper supported the Federal Governments push for performance based pay. (I expect Andrew may have been surprised by the link)

    From memory Andrew has himself been on national TV suggesting higher pay for good teachers and so the debate and comments here are understandable.

  40. keegan says:

    John may have a point!

    I’d like to raise a point about post graduate qualifications too, ie teachers with masters and Phds, surely they can bring added benefit to classroom learning in high schools. Their added knowledge in science or maths etc could bring a higher level of learning and inquiry to a classroom environment.

    Human capital theory (HCT) tells us that the higher the qualification the higher your output (Belfield, 2000; McMahon, 1998)

    Why are teachers now four year trained and even five year trained? Maybe we should go back to the two year training days of CAEs if the extra training does not make a difference according to economists. But this will then go against their own HCT.

    Why then do teachers in universities, in the main, require Phds? Why doesn’t an undergraduate degree or a diploma suffice for a lectureship at university?

  41. Andrew says:

    I will leave it to others to discuss what long or short term measures or mechanisms would lead from higher pay to better teaching. What I would like to see is some estimate of the reliability of a results-based test to determine that teacher A was better than teacher B. What are the errors associated with the estimates of individual teachers’ relative abilities?

    I say ‘relative’ since it is my understanding that if all teachers became better by the same amount or factor – say all testees obtained 50 marks more after two years – the results of the model would be unchanged. Given this point one should be wary of talking of good and bad teachers.

    Whilst the focus on relative value-added affects forestalls a number of potential objections I wonder whether the method might have occluded possible effects like a tendency to increasing dispersion independent of teacher (e.g. a correlation of low starting point and low progress). Also would the measured teacher effect be less if school level variables were considered?

    These questions become important when conclusions about indigenous education policy are drawn from the paper. (Something that cannot be supported since the analysis took no account of indigenous status and the overall numbers of indigenous students are small enough for a quite different picture to be hidden in the aggregate data.) In the case of remote indigenous schools it would be difficult to separate teacher and community effects. Also in some Queensland schools the average differences between indigenous students and non-indigenous students are of the order of two years. It is hard to see that this is simply a matter of teacher effectiveness.

    Finally with all respect to keegan it has been suggested that the existing teaching courses do not well prepare first year teachers for teaching and that going back to shorter more didactic training and giving them much more direct instruction on what to do would help. An economist would point out here that the market for teaching is highly imperfect – you can’t teach unless you have an approved qualification so we can’t compare the effects of radically different sorts of teacher preparation. (A comparison of this sort could be attempted in kindergartens except that they are resistant to assessment of academic outcomes.) In any case it is hard to see how a higher degree would significantly affect the quality or content of teaching; you should already know the curriculum and how to teach. They are normally done to rise in the service – or to get out of it and teach teachers.

  42. Kylie says:

    Just a thought on the apparent ‘plateauing’ of improvement for teachers after 9 or 10 years experience. This could be related to teachers being quite willing to remain in the same class environment after this level of experience. When you consider the majority of teachers are female, after 9 or 10 years experience, they are likely to be raising a family etc as well as teaching. To keep life as straight forward as possible it is definitely easier to stay in one class, in the same school.
    From my observations this is especially strong in rural schools. iIwould imagine that this results in a level of complacency and stalenenss. Perhaps the ‘plateauing’ could be overcome by requirements such as:
    * teacher exchange programs between rural and urban schools (e.g. month long exchanges etc) to remind teachers that the world is a widely varying place,
    * within schools, requiring teachers to teach across year levels in say a 10 year period. This should also include a requirement of complete new curriculum tasks. (i.e. not simply moving year level and using the same projects/ assessments as have been used in the previous level)

    Also, as a complicating factor, in Victoria at least, the pay system of paying on years of service means it can be difficult for experienced teachers to move schools, as they are more expensive than graduates.

    (Disclosure: I am the wife of a primary teacher in his 9th year of teaching public sector, and the mother of 5 children 4y.o to 12y.o)

  43. Kevin Cox says:

    Here is a short article that addresses motivation.

    http://www.theage.com.au/articles/2007/05/22/1179601410236.html

    May I suggest that the simplest most cost effective solution to getting better teachers is to give them more respect. Pay is one way of showing respect but perhaps not the most effective and maybe performance based pay will be counter productive as it brings competition into what should be a supportive cooperative environment.

  44. Fred says:

    I’m rather intrigued by the mishmash of normative data mixed up with standards referenced data. Presumably the testing done at Year 3, 5 and 7 was standards referenced. That means, as far as I’m concerned, that the old bell curve is not relevant. No matter what you do with teachers, their qualifications, their experience, how interested they are in their subjects etc, if you start talking about means, standard deviations and normal distributions, you will ALWAYS get those who are below average and those who are significantly below average as well as the other end of the spectrum.

    As far as I can see, Andrew’s research skips over this. Consider this hypothetical situation: let’s say every student who did these test gets into the top band of the results on a standards basis. Andrew’s analysis would still have it that some teachers are not as good as others because when the students’ results are put onto the bell curve, we’d still have some who were below the mean and some who were above it.

    I thought we’d moved past this kind of stuff years ago.

  45. Andrew says:

    Although I made a point similar to Fred’s I will defend the real (original) Andrew on this point. Whatever the standard you adopt for year 3, 5, 7 you would expect a bell-shaped dispersion if it is defined by some measurement. If the dispersion is narrow and everyone agrees that the central tendency represents what we want at that age then fine. In fact the dispersion is not fully bell-shaped, it is quite wide on the lower side, and some doubt the central value is that good either. Under these conditions you want to move the lower segment upwards and that’s what I take it Andrew is addressing.

    Also note that Andrew is not putting the results onto a bell curve but the increments in the results. This is quite a different exercise.

  46. Fred is getting close to the point here. To me, the statistics described in the paper (on which the conclusions are based) are not at all convincing. Despite there being evidence that the results of the data testing were statistically significant (ie, they did not arise merely by chance), the percentage of the total variance attributed to the teacher influence on student outcomes is a (very low) – a mere fraction (0.47) of 1%(!) A person observing of the scatterplot in Figure 1 (p. 10 of the paper) could justifiably draw a line along the 0 – 0 line (ie NO relationship), especially if one or two critical outliers were eliminated.

    The paper is most useful – especially given the amount (and quality) of response it has generated. I have been most interested to read the more general comments on how we can identify and reward effective teachers. However, for a Minster of the Crown to base a policy decision on this (and previous US studies) beggars belief. This is a good beginning to a valuable series of on-going research projects. It is not, however, evidence of an ability (much less a ‘need’) to adopt performance-based pay systems for teachers within the life of the present, and even the next federal government.

    A strong word of caution must be made here relating to research carried out by economists (and in my case, accountants). For a start, we are not (in many cases quite justifiably) welcomed with open arms by our colleagues in education. We have a valuable role to play in our research endeavours (especially using quantitative designs), but in the final analysis it is educational theorists who need to make the final call on what is appropriate in terms of education policy.

Comments are closed.