About two years ago, we brought over Mario Rinvolucri to do some workshops for local teachers. In one of his workshops, he mentioned the work of Barry O’Sullivan, Professor of Applied Linguistics at Roehampton University London. As you can see from his profile on the University website, Barry is “particularly interested in issues related to performance testing, test validation and test-data management and analysis.” Unfortunately, I am not great at retaining details and so I have been retelling the anecdote – without any details – for several years. Earlier in the week, I decided to put an end to the vagueness by emailing Barry and asking him to fill me in on the details and he was kind enough to do so. Below is a retelling of his fascinating story that certainly highlights how poorly a test can actually evaluate language proficiency.

A while back, Barry was running a training course for the Ministry of Education of a Northern European country. He was shown a copy of one of the language proficiency tests that they give to immigrants and Barry suggested that he try writing it. They apparently laughed, as they knew he had no knowledge at all of their language. However, their mirth quickly turned to something altogether different as, by the end of the test, Barry had answered 80% of the questions correctly! He repeated the performance and the score with another version of the test, thereby depressing his hosts.

The questions, by the way, were all multiple choice or matching items and Barry assumed he had been writing a reading comprehension test.

A few years later Barry was doing a different training course for the Association of Language Testers in Europe and recalled the story. It turns out that in the crowd was a person who had been in the Ministry of Education group. This person embarrassingly told the group that the test had actually been a test of listening comprehension!

How is this possible, you may be wondering? Well, clearly Barry is an intelligent man. But he also knows how to “read” a test. He shared his strategies with me:

1. Match words in the stem and the options

2. Try to reduce my odds by identifying two options as possibilities (they seemed to be similar in some way)

3. Look for the longest answer if nothing else worked

This story has actually ignited an interest for me in testing and assessment. I am not sure that I have huge insights to share as of yet, but I would like to highlight two rather obvious morals to Barry’s story. The first is that preparing for an exam is, to a greater or lesser extent, about preparing for that specific exam. Students wishing to do well on IELTS or TOEFL, for example, need to study the format of those exams just as much as they need to improve their academic English. Furthermore, there are most definitely strategies or “tricks” for making good guesses when knowledge is lacking.

The second obvious point is that exams are far from perfect. As educators, we need to bear this in mind and take test results with a grain of salt. Furthermore, we should stop (or at least speak with) learners who decide they want to study for tests like TOEFL simply as a means of improving their English (if you have not met any such students, I assure you they do exist; I have met plenty of them in our storefront). Additionally, we need to keep in mind that not all tests are created equal and that they should be carefully examined themselves. I have always been a fan of the Cambridge exams (FCE, CAE, IELTS) because, though they are not perfect either, they are well enough designed that students preparing to take those exams will undoubtedly learn some useful English along the way. I am not so sure the same can be said about TOEFL and I find it worrying that there is far more information available about security than test design for the new PTE Academic. That is not to say that PTE Academic is not a good test as it is quite possible that it is. However, Pearson is largely marketing this test on the basis of its security measures. Security is a big enough of an issue for this marketing tactic to work. However, the actual strength of the test itself absolutely must be considered.

I am looking forward to the upcoming “The Cambridge Guide to Language Assessment”, of which Barry is an editor. He has a number of published papers (see link at the beginning of this blog) and you may want to check out an article of his in hltmag, which is available online.

Posted by Nicole