Matching Comparative Judgement Data to the Curriculum

In the latest results for the Assessing Writing in Australia project, we were able to see a high reliability in a standardised set of data for Year 2, 4, 6, 8 & 10. This was due to having multiple judges making comparisons for our student scripts instead of one teacher making a subjective call on a writing piece.

This innovative assessment technique is known as Comparative Judgement and you can learn more about it here. I have also discussed it as a key measure of valid and reliable data previously with TFE and in our Writing Connect Discussions every term.

The 3 key pieces of information provided by No More Marking are:

· A scaled score

· A NAPLAN indicator

· A writing age

As Daisy Christodoulou explained, the scaled score is determined through the boundaries drawn on our distribution of each cohort, explained here for Primary and Secondary. What is left for teachers is to now think about how we will align this to a curriculum level.

The scaled score allows us to measure progress over time. The writing age is provided to make sense of this number, but we are cautioned not to directly use a writing age to assign a curriculum level. The best way to think of it, is to see the age as an alternative to grading as Daisy has shared here. I have written more about ways the writing age can provide an important comparison, like for instance with a gender gap, here.

For the purpose of curriculum reporting, however, pulling apart some student information using the predictive NAPLAN score and scaled score, might be more useful. How do we now match up the 3 pieces of information we received for each student, to a curriculum level?

Here is an example of some ways we used the results at Templestowe Heights.

Firstly, we think about what directs our instruction. This is first and foremost curriculum driven. We have pulled apart the Victorian Curriculum into a progression of skills and if we match these against the student writing sample, and the information provided from the Comparative Judgement project, we can make some interesting observations.

This student sits at about the centre of our distribution and has a consistent cohort age. They are also ticking off most of the skills on this list and their classroom samples indicate we can feel confident of this. We can make some general assumptions because of the writing age, that other students who have scored above this scaled score, have achieved similar or better. This is a much quicker way to determine if the majority of the cohort are capable of these skills or conversely, sit below them.

One of the other instructional frameworks we use is The Writing Revolution. Here is a Year 4 sample measured against the TWR skills we have focused on this semester.

This student sample is the same NAPLAN band, however their scaled score and writing age is higher than the previous, which we as a team agreed with. Despite the student writing less, the complexity of their creation demonstrates a deeper interaction with sentence knowledge in the form of description and a balance of syntax consistent with the purpose of their text structure. The punctuation is accurate, with evidence of revising, and we can see one of our explicitly taught vocabulary words present in the text!

Once again, we can make some assumptions that students who have scored above this (of course we always examine this for outliers and inconsistencies) have achieved a greater curriculum progression.

This allows us to make some efficient reporting judgements against curriculum using the progressions that scaffold our teaching. The exciting thing going forward is that over time, when we have multiple data points, we can see what learning is embedded and where the gaps lie so we can direct our instruction explicitly.

Here is an example of this from the Year 4 cohort which participated in the start of year trials, therefore providing us with multiple data points that we can compare.

Improvement is evident in the first line with capitals in the title and a better quality beginning sentence. The student also demonstrates more complex syntax and interestingly, does not list their characters in their second script – something we worked on after seeing this repeatedly in the February project. I look forward to continuing to share further insights.

To find out more about our Australian projects, see details on the No More Marking website or contact me: jeanette@nomoremarking.com or jeanette.breen@education.vic.gov.au