Imagine you’re the CEO of a major company. One of your advisors comes running to you. They say we should change the way the company operates. The changes suggested aren’t minor – in fact they involve sweeping away what your company has done before. What would you do in this situation?
The obvious answer is you would ask for some evidence. After all, you’re hardly going to make big changes without having some hard data to back it up. Now, what if the advisor told you that they didn’t have any hard data, but they did speak to some colleagues who thought it was a good idea. What would you do then? Presumably, short of laughing them out of your office, you’d ask them to gather some quantifiable evidence before mandating for any changes.
This seems like a basic standard to set. Big changes need strong evidence. Unfortunately, the Ministry of Education does not even apply a standard as simple as this one.
Streaming – the practice of separating students by ability in our schools – has come under fire by a recent publication titled Kōkirihia by Tokona Te Raki. It calls for a ban on streaming by 2030 in New Zealand schools on the grounds that it is unethical and racist. The Ministry agrees and wants streaming banned.
Before circling back to the Ministry, let us firstly examine this report. It is important to stress that it was written with the best of intentions. Tokona Te Raki are an advocacy group for Māori issues, particularly in education and employment. Their mission is to end inequitable outcomes for Māori in these two spaces.
These are worthy aims. With regards to education, there has been a gap for decades between the educational attainment of Māori and Pasifika students and their Pākehā and Asian counterparts. Tokona Te Raki’s mission to end this is important.
Unfortunately, this paper has two fundamental flaws. Firstly, the original research relies entirely on dubious qualitative evidence. Secondly, it seriously misrepresents international findings.
Qualitative evidence has very limited reliability. It usually draws on interviews and focus groups interpreted by the researchers. As such, qualitative research conclusions can lack scientific rigor and are subject to bias. There are ways to minimise that bias, but it’s unclear that Tokona Te Raki researchers did so.
Interviews with teachers and students holding a negative view of streaming form the main evidence base for Kōkirihia. The views expressed by the research participants are no doubt sincerely held. However, they are just opinions.
It is hard to avoid the suspicion that the evidence presented in Kōkirihia was cherry-picked to represent the conclusion preferred by its authors.
Citing data from the OECD, Kōkirihia claims that New Zealand streams more than any other country except Ireland. It also highlights New Zealand’s relatively wide disparity between the highest and lowest achievers, suggesting that these things are connected.
This argument is weak at best. Ireland streams more than New Zealand, yet has one of the smallest achievement gaps between the highest and lowest achievers in the OECD in key subjects. If New Zealand’s results suggest that streaming increases disparity, do Ireland’s suggest that streaming decreases disparity? Once again, the suspicion of cherry-picking is difficult to avoid.
That suspicion is reinforced by the authors’ superficial and careless reporting of research literature. A vast body of literature examines the impact of streaming on academic and sociological outcomes for students from many countries. Yet, Kōkirihia’s bibliography contains only 19 sources. This is hardly an adequate survey of the impacts of streaming. And a detailed examination of those sources reveals worse problems.
Early in the report, the authors claim “The research evidence is unequivocal that fixed-ability grouping in any form does not work”. But that assertion is not supported by the research reviewed in the report.
For example, Kōkirihia cites a large-scale study by Professor John Hattie. It is true that Hattie is not pro-streaming and points to qualitative evidence finding inequitable outcomes. However, he found that streaming has no impact on academic outcomes at all according to empirical evidence. He concluded that the quality of teaching and student interactions are much more important than the composition of the classes. Much international research that uses large-scale, quantitative methods is also cited. But these studies had mixed findings.
Yet, they are supposedly evidence that the literature is “unequivocal” in saying that streaming doesn’t work. It is not.
Kōkirihia was, no doubt, written with the best of intentions. But good intentions cannot make up for poor research. It is possible that the assumption made by the authors, that streaming is harmful, is correct. But the report does nothing to substantiate this assumption.
But what does this have to do with the Ministry? Well, in our initial analogy, the Ministry is the CEO and Kōkirihia is the advisor. You would hope that the Ministry would ask for stronger evidence before taking a position.
We asked the Ministry, via an Official Information Act request, what data they have on streaming in our schools. We asked about streaming according to year groups, subjects, type of schools, etc. The answer can be summarised curtly: they have nothing.
What they did have, however, was links to reports by Tokona Te Raki including Kōkirihia. Yet, as we have seen above, reports like Kōkirihia are not substantive enough to mandate for major changes.
To recap: the Ministry has no idea if, where and how our schools use streaming. It is thus basing its position off the findings of reports like Kōkirihia, which use dubious qualitative evidence and misrepresent the international literature. And it is on this shaky ground that they want to get streaming banned.
We don’t need dubious research determining educational policy. What we need is to know not only that streaming occurs but where, at what age, and in what subjects.
Once we have all the data, we need a large-scale quantitative study on the effects of streaming on learning. That study, and not what we currently have, should shape future policy.