h2g2 The Hitchhiker's Guide to the Galaxy: Earth Edition

Entry: Principles of Language Testing. - A775893
Author: Solnushka - U138596

Still got to credit the guy I've quoted, but other than that it's ready for you comments.

Oh, it's *ages* since I've done this type of test. I've done loads in my time, including German, French, Latin, Russian and Italian at different levels.

They were mostly four types, oral, where it was one to one, aural, where we listened to a tape and had to answer questions, comprehension, where we were given written text and had to answer questions on it and ones where we had to do stuff on set texts, which was more understanding, say what Corneille (or whoever the author was) meant.

I suppose another type of test would be to see if a candidate spoke enough of a country's language to be able to do a job properly or to apply for citizenship. There may be others.

Do you think it might be helpful to give examples of what the intent of the language testing is?

it should be fairly easy to change this to being about the general principles of testing, as most of it is not specific to language testing.

just a thought.

xroth: I did think about that, but the thing is, I *know* that these are the basic theoretical principles for language tests - I mean, I know that people in the business seem to agree that these are what should taken into consideration, under these headings. Though obviously they tend to go into more detail.

But I have no idea what other disciplines think about them. I agree that they could be applied, but I don't know if they are, or if someone has come up with a different set. I imagine they would be similar in some ways, but perhaps they go by different names or something. I did have a little rootle on the internet, but couldn't find zip on the subject, so on the whole I think it's better if I didn't leave what I know about to talk about something I don't.

Do you think I shoould expand the small comment I have made about their universal application, though?

Zarquon's Singing Fish: I was actually thinking about doing a seperate article on types of test as, trust me, it would make this one way too long to go into more detail than I have here on the subject. But even if I did that, then it wouldn't be a detailed examination of the componants (such as the listening/ reading/ speaking skills) that each test would include, as the ultimate purpose for testing would define those, and every single test would be different depending on what it was for. The thing about these principles is that they are applied simaulaneously to the whole test, each componant/ paper, and each question, regardless of what type of test it is. Have I not made that clear?

I suppose I could do another article going into more detail about the sorts of questions you can get and how to make them really solid, but I really don't want to...

Sorry: 'xyroth'.

OK, I had a look, and actually the sections on practicality and reliability are alraedy not particulalrly subject specific. But I still don't *know* that these principles are used outside of language test construction.

OK Solnushka,

Entry's looking good to me! I think you did make some small changes, yes?

Well, not to the content. I've been doing a bit (more) proof reading though. Which reminds me: how do you spell 'alter' as in 'item of religious significance' rather than 'verb meaning change a bit'?

No, but really: would you like to see a bit more of the practical applications bit? If so, in what areas?

You definately have got the stuff about right, but I need to be at my other location to check the terminology for generality.

That should be possible in a day or two.

Well, if it turns out the principles are universally applicable, and universally applied, I'm goonna need some help in thinking up non-language tests to exemplify. I shall pinch your intelligence test for a start...

intelligence testing is a very good bad example.

it has suffered from both cultural bias in both class and sex forms.

there are also examples of varients which are multiple choice (but which don't have balancing questions to test for guessing or lying).

there is even one varient which can give identical results after you receive a brain injury, even when you are obviously impaired.

as to more general background, some of the reliability attributes are a direct consequence of "the knowledge representation hypothosis". this basically states that there should be something there which you are measuring, and that if you measure it a few times you should get very similar results. a lot of test fail this test.

The thing about 'if you measure if a few times you should get very similar results' is more or less my definition of reliability, as I understand it. I'm not sure what you mean by 'there should be something there which you are measuring', unless we are talking about intelligence, in which case I assume it has something to do with the trickiness of cutting up 'intelligence' into discrete sub skills and then attempting to measure them seperately. And probably failing to identify all of the aspects. I'd put that under 'validity', myself, as it's similar to attempting to test skills like writing with discrete item questions, and also part of the problem of identifying exactly what areas are important to test generally. As for then using the tests out of the context they were originally created for, words fail. That's a big sin for validity. And then the culture bias. I suppose the reason they still have any credibility at all is the fact that it's such a seductive idea.

They remind me of the 'aptitude' tests for language learning, which I haven't mentioned in the article as they are not about testing language at all. I can't imagine how you could ever, with any degree of certainty, identify all the factors which go into 'being good at learning languages'. I mean, there's a lot of discussion about it, and some good points made, but these are not rules to live by. As for breaking them up and testing them

As for these examples:

'there are also examples of varients which are multiple choice (but which don't have balancing questions to test for guessing or lying).

there is even one varient which can give identical results after you receive a brain injury, even when you are obviously impaired.'

... and the point you made in your (excellent, by the way) entry about how they are not very good for testing the extremes.

Those are straight reliability issues, no? And to be fair, not necessarily examples of why intelligence tests are bad generally. There are a lot of very shonky (language) tests out there, which is more a factor of bad design with respect to reliability than an argument that all such tests are bad. They just need to be rewritten.

Anyway, trashing tests is fun, but what I meant to ask is: any news on the cross-subject applicability of the terminology/ groupings I've got? Because if my interpretation (see above) of the knowledge rep. hyp. is correct, then I think language testing does look a bit differently at/ differently catagorise the same problems. Plus, at least with language testing there is less problems with the whole issue of being somewhat sure that there is something there to test in the first place, although I suppose there is a direct correlation between that and the vexed issue of trying to test sub-skills individually, and the fact that fashions change with respect to what is important to demonstrate knowlegde of/ skills in.

right, starting with the knowledge representation hypothosis, it is the fundamental rule of any type of modeling.

the classic example used to demonstrate it is temperature. if you measure the temperature of ice it should always give a near identical result, as should steam, and steam should get a higher score than ice.

remarkably, I have seen various tests which don't fit this rule, and thus are completely worthless.

multiple choice questions work very well for testing beurocrats, but people who are more creative do very badly with them. similarly, they can be guessed at, so you need to include questions in your test which show a similar result when the answers are genuine, but which people will guess differently.

if you move away from multiple choice questions to a more open format, you have the problem that most answers are likely to be unique, and thus subjective in their marking, but these questions work much better for creative people.

I suspect you might be right about the terminology being slightly different. I am back at the other location now, so I should be able to find my reference sources and find the terminology for you.

Yup, that's reliability. Actually, it's easy to get this wrong. I mean, the suggestions I have made about how to help are all very commonsensical, but quite difficult in practice.

Ah. Another person who is not a fan of multiple choice systems. You must be American. That said, though, I'm not sure that British exam culture is much better - where you have to learn the exam and get yourself in the 'insiders' group of those in the know to have a hope of passing well.

Anyway, I warn you, if the terminology is different, I'm not changing it

actually I am british.

I am not against multiple choice, only in favour of it being used appropriately, which it often isn't.

in a lot of cases people making these sorts of tests don't think of not using multiple choice, despite the diadvantages.

well, I haven't found the information I was looking for, but I did find something else that was relevent.

what I found was iso 9126, which is a set of measures for determining quality and which seems to be generally applicable.

it splits quality into six parameters which are then further splt to give a set of measurable quantities.

quility is split into functionality, reliability, usability, efficience, maintainability and portability.

these are further divided as follows:

functionality covers suitability, accuracy, interoperability, compliance and security.

reliability covers maturity, fault tollerance and recoverability.

usability covers understandabilty, learnability and operability.

efficiency covers how much time and other recources are required.

maintainability covers analysability, changeability, stability and testability.

and portability covers adaptability, installability, replaceability and conformance.

as this standard covers testing of computer programs, there might be the odd term in there which you are not familiar with.

if so as me to explain it.

so how does this sort of thing fit in with your thinking about the posibility of generalising the entry?

Nice entry - although it's doesn't really describe what language testing is, it just jumps straight in. I had an idea of what might be meant by "Language Testing", although I wasn't certain and reading the entry just confused me more until I'd got quite far into it and come to a conclusion on what was meant by language testing.

Then again, it could be that I'm just stupid as it now seems glaringly obvious.

Niwt

Are those the criteria for measuring the success or otherwise of the programmes, or the quality of the tests themselves. I must say, it's a lovely example of another type of test, outside the purely academic, to which the principles can be applied. But if they are the criteria for measuring the quality of the tests, then it does rather prove my point that although I could probably shove them (protesting) into my catagories for the principles, and probably there are a lot of similarities overall between fields, different fields have come up with different ways of listing and sorting the underlying principles.

Which would mean that in order to write about testing in general I'd have to rewrite the article completely, because I don't see why every other field should be held to the ways of grouping and talking about the principles that are used in langauge testing, although I'd bet there's a lot of cross pollination. I mean, I doubt the principles for language testing were derived in a vacuum. It's not about tweaking it a bit, is it?

And I don't really want to take on a project of that magnitude. Plus, I have no way of researching it properly. Sorry, xyroth. I'm enjoying this discussion, though.

Niwt: Now, that's interesting. If you could tell me what you were thinking it might be, perhaps that would help me to change it? Would changing the first sentence to 'There are as many tests of foreign language skills as there are reasons for testing them...' help, for example?

I am enjoying it as well, but from your last comment it has become ovious that it is peripheral to the entry.

so pop over to my userspace, and start a thread there about it, and we can keep peer review clear for comments which will help the entry.

good work so far, by the way.