Over-marking of controlled assessments in some schools
- Published on Friday, 02 November 2012 10:59
- Posted by Scott Buckler
The complexity and poor design of GCSE English exams, along with too much emphasis on school-based controlled assessment, led to some schools in England experiencing grade variations this summer, the exams regulator Ofqual has concluded
Combined with pressure created by performance measures that provide incentives to ensure as many students as possible achieve a C grade, this led to significant over-marking of controlled assessments to meet grade boundary targets.
The report by Ofqual sets out in detail the data, analysis and other evidence that Ofqual has considered over the last two months to help it understand why there was unexpected variability in GCSE English results in some schools. It follows an initial report in August.
The report also reviews in depth the issues covered by the August report. It confirms the conclusion in the initial report that grades for GCSE English were set at the right standard in June 2012, and provides the evidence to show this.
The report addresses the concerns about GCSE English awarding. It shows that – although there are problems with the qualification – there are more significant issues.
- It explains why statistics had to be used to enable accurate grading in this new qualification. With many changes to the content, structure and assessment of the qualification, it was essential to use data to check that the standard was being maintained, and in particular to make sure that the 2012 students did not lose out from being the first to sit the new qualification.
- It sets out why different grade boundaries were set in assessments in January and June.Controlled assessment boundaries for the internally-marked controlled assessments were set differently in June, when four out of five of all units were submitted and examiners saw over-marking.
- This meant that the average mark for a piece of work of a particular standard was higher in June than in January, and examiners judging the standard of work set the grade boundaries to reflect that.
- For the exams, the report explains the difficulties of setting grade boundaries in January, and why we conclude the June boundaries were set properly. The evidence does not support the suggestion that June boundaries were unnecessarily tough.
It explains why Ofqual had to intervene in summer 2012 awarding with two exam boards. Had it not done so, standards would have been set at the wrong level. There would not have been consistency between exam boards.
The report acknowledges that, knowing what we know now, regulators and the exam boards could have done more. Some controls, especially moderation tolerances for controlled assessment, could have been run more tightly and communication could have been better in various ways. However, once this GCSE design had been implemented, Ofqual does not believe that the problems seen this summer could have been eliminated, no matter how much more tightly these qualifications had been managed.
The report sets out the action that Ofqual will be taking based on these findings.
Because of the problems with the design of the qualification, and the overmarking in some schools, it sets out why it was right for exam boards to offer the additional November resit opportunity. This will provide students who think they did not get the grade they deserved with an opportunity to demonstrate their achievements. The report explains that it would be impossible to identify any students who may have lost out as a result of some schools over-marking. Any general change to grade boundaries could not be justified, as it would increase unfairness.
Acknowledging the design problems with the qualification, Ofqual will be taking regulatory action to minimise the risk of the 2012 problems recurring in 2013. Exam boards will be required to tighten controlled assessments, and not to issue grades for exams and controlled assessments sat in January 2013. It confirms the decision, announced last year, to end the modularisation of GCSEs in England from September 2013.
The report also raises some wider issues for debate and discussion, particularly around the accountability arrangements.
The regulator concludes that so much weight on one grade in one subject as part of accountability and performance measures created perverse incentives for schools in the way they marked controlled assessment and led to the over-marking.
Ofqual's report includes a review by Capgemini of interviews with 100 schools, many of whom felt they were adversely affected by last summer's results. It highlights the impact of the intense pressure felt by schools and teachers to predict and achieve specific grades.
While no school interviewed considered that it was doing anything untoward in teaching and administering these GCSEs, many expressed concern that other nearby schools were overstepping the boundaries of acceptable practice.
The report states: "The pattern of marks – the unprecedented clustering around perceived grade boundaries for each whole qualification – is striking".
Glenys Stacey said: "It is clearly hard for teachers to maintain their own integrity when they believe that there is a widespread loss of integrity elsewhere. No teacher should be forced to choose between their principles on the one hand and their students, school and career on the other."
Despite the exam boards' warnings to schools, the regulator's report found that some schools assumed wrongly that grade boundaries published for January 2012 controlled assessment units would be carried forward and used for June assessments. When these were combined with grades for students who had sat the written exam in January 2012 they provided apparent assurance of the final grades. When grade boundaries were set by examiners in the summer, and the boundaries were different, because they had to reflect the over-marking, some schools and students were disappointed.
This was the first time the new qualification had been awarded. Schools were familiar with the previous GCSE and teachers were routinely able to predict pupils' grades accurately.
The new English exam suite – GCSE English, English Language and English Literature – was planned in 2007 by the Qualifications and Curriculum Authority, since disbanded, and introduced in 2010. Ofqual became a fully independent regulator in the same year that teaching started in the new qualifications.
From September 2013, GCSE English qualifications will no longer be modular in England. Ofqual is taking regulatory action to prevent a repeat of the problems experienced this year.
- To protect standards of GCSEs in 2013, all January 2013 assessments (whether examination or controlled assessment) will be marked, but not graded, to reduce the risk of marking to meet the expected requirement for a given grade.
- This will enable the grades for January and June to be set at the same time
- Marking moderation tolerance will be tightened, reducing it from the current 6 per cent
- Exam boards will be required to improve their communications with schools
- Ofqual will consider if any further exam design changes are needed to protect standards.
In addition Ofqual will:
- Continue to review controlled assessment – controlled assessment makes up 60 per cent of the GCSE English exam. The review will report, with recommendations, in January
- Advise on exam reform – care must be taken to ensure the introduction of any new exams, such as the proposed English Baccalaureate Certificates, learn the lessons of the design failures in GCSE English
- Use our findings to inform real debate on accountability – strong and well-directed accountability should be possible without assuming that what can be accurately captured in accountability testing represents the totality of what should be taught in schools.
Glenys Stacey said: "Overall, the grading of GCSE English in the summer of 2012 was a fair reflection of the performance of pupils as a whole. However, within the overall picture some schools experienced significant variations.
"Though the content of the exam is sound, and the assessments are in some ways better than the previous GCSE English qualification, the technical design of the exam has flaws. That is why we are taking action to reduce the risk of exam processes and teachers' grade predictions causing difficulties in the summer of 2013 and beyond.
"Our report reinforces the need for strong and independent regulation. We have learnt lessons and found areas where we might have done better. Others must also reflect on the findings of the report and consider how, together, we can continue to improve the rigour and credibility of examinations.
"Striking the balance between accountability and qualification design will never be easy. We would welcome wide discussion of these issues in the context of the current accountability review and also the development of new qualifications, including the English Baccalaureate Certificates."