<\/p>\n\n\n\n
![\"\"](\"https:\/\/oeb.global\/oeb-insights\/wp-content\/uploads\/2019\/11\/alphaplus-1.png\")
<\/figure>\n\n\n\nWe often find ourselves discussing assessment design options with clients, and these discussions can often seem like an ever-growing swirl of options, with each further step down the assessment theory rabbit hole adding new complexity to the decision making. While trialing new assessments remains the best option, followed by post-release qualitative and quantitative performance studies, documenting the \u201cdesign compromises\u201d can help remind assessment designers about the choices they made when considering how a live assessment is working, and what improvements to make next.
<\/p>\n\n\n\n
Many AlphaPlus conference presentations over the last decade have had a common theme of \u201cmake better use of assessment performance statistics\u201d. So often, practicalities mean that assessments have to be launched without the opportunity for adequate trials or validity studies. Performance data provides an essential way to get rapid feedback on how new tests are performing, and to direct scarce resources toward improvement. The acceptance of the need for continuous improvement and the pace of the improvement cycle is often more important than the quality of the initial assessments from the perspective of overall validity.
<\/p>\n\n\n\n
One assessment data lesson that comes around time and again, highlights one of the biggest and most fundamental challenges \u2013 writing good questions. This sounds obvious (it is!) Alistair Pollitt\u2019s bucket brigade analogy [1] : \u201cvalidity (water) in the assessment development and delivery chain (the bucket chain) can only be retained or lost, not added\u201d helps to clarify. Good assessment specifications, strong writing teams, cycles of review, and, of course, trials have helped make \u201cwriting good questions\u201d less of a craft and more of a manageable process, but even the best teams and processes can throw up surprises.
<\/p>\n\n\n\n
The challenge to move many vocational assessments from minimum competence to graded models, with the associated challenge of defining what a level means, and how to align it to a test score reminded me of a project we ran some years ago where the challenge was to write questions for a high stakes test where the questions had to be targeted at the grade descriptors for GCSE mathematics (An \u201cA grade\u201d student \u2018can do\u2019 the following, A \u201cB\u201d grade student \u2018can do\u2019 the following, etc.). We assembled the best maths test writers we had, organised them into peer review teams, with cycles of QA, and asked them to write \u201ceasy\u201d, \u201cmedium\u201d and \u201chard\u201d questions. The result is shown in the chart below. Many of the designated \u201ceasy questions\u201d were harder than most of the \u201cmedium questions\u201d and several of the \u201chard questions\u201d, and so on.
<\/p>\n\n\n\n
![\"\"](\"https:\/\/oeb.global\/oeb-insights\/wp-content\/uploads\/2019\/11\/alphaplus-2.png\")
<\/figure><\/div>\n\n\n\nAfter much analytics, looking back at previous assessment cycles, reworking of test content, etc. we concluded that we had probably done a reasonable job against a tricky assessment specification. The questions performed well in terms of difficulty and discrimination, but not entirely as predicted. Subsequent writing cycles fitted the model better. This highlights for me two really important messages:
<\/p>\n\n\n\n
- Assembling good writing teams, and providing processes that support their work with multiple QA cycles is as critically important as ever in maximising the chance of good assessment outcomes.<\/li>
- Trusting assessment writers to know the difficulty of their questions makes you (the assessment owner) a hostage to fortune. Assessment data provides both the acid test and the process feedback which is critical to improving quality over time.
<\/li><\/ol>\n\n\n\n[1] For example: Creating valid assessments for Curriculum for Excellence<\/a>
<\/p>\n\n\n\nWritten by: <\/em>John Winkley
<\/p>\n\n\n\n