wu :: forums - Print Page


    
      
        wu :: forums
        (http://www.ocf.berkeley.edu/~wwu/cgi-bin/yabb/YaBB.cgi)
      

        riddles >> medium >> Test Scores
        
(Message started by: Quetzycoatl on Feb 26^th, 2004, 9:51am)

Title: Test Scores
Post by Quetzycoatl on Feb 26^th, 2004, 9:51am

I am not sure how difficult this is so I am sticking it in Medium:

I work for a company that designs, administers and scores large-scale, standardized tests to students K-12. For a particular test and grade we can have up to as many as 100 different versions of a test, all testing the same subject matter. This is to increase security and reduce cheating among other things. Each of these versions are differentiated by a form number which each student must fill in on their answer sheet in order for it to be scored properly. Without it, there is no certain way to know which answer key to use.

Sometimes students will leave this form number blank and, for some tests, the procedure is then to check it against every answer key and use the one that would yield the highest score. In most cases this works fine and the student recieves the correct score. However, the worse a student performs on a test, the more likely their test will be scored with the wrong key.

My questions are:

What is the probability that a student who does not fill in the form number will receive a higher score than they deserve?

How badly does a student have to do on a test in order to gain an advantage by not filling in the form number?

DISCLAIMER: I should probably emphasize that this procedure is rarely used, and when it is, it is only on what we call low-stakes tests, i.e. tests that will NOT have a bearing on things like moving on to the next grade, getting into college etc.

Title: Re: Test Scores
Post by Sameer on Feb 26^th, 2004, 11:11am

Options:

- Computerise the tests!!!
- Give the answer sheet attached to the question sheet with pre-printed form number.

8)

Title: Re: Test Scores
Post by towr on Feb 26^th, 2004, 11:18am

Quote:

How badly does a student have to do on a test in order to gain an advantage by not filling in the form number?

He allways has an advantage, either he'll get the grade he deserves, or a higher one. It's a no-lose situation.
If he doesn't have all the questions correct there is a probability of (n-1)/a^q that there is a key that does fit all his answers (where n is the number of keys, a the number of choices per question and q the number of questions)
Only if the student has all the answers correct is there no expected gain, but it's doubtfull he can be sure of that.

Title: Re: Test Scores
Post by Quetzycoatl on Feb 26^th, 2004, 12:01pm

Good point Towr, I had not thought that through. That still leaves the first question though.

Sameer: we do, in some cases do computerized tests, but they are very expensive compared to paper and many school systems simply do not have enough computers to make this feasible. Also tests on computer are more difficult and less accurate than on paper. You would think in this day and age that the affect would be slight but it's not. If you are interested, do a search on Psychometrics and computerized testing.

We also do something similar to your other suggestion at low grade levels, which are less reliable in recording things like form numbers. However, this is also more expensive and when you consider how slight a problem this is (like I stated before, checking against all keys works most of the time), any additional cost is simply not worth it.

Title: Re: Test Scores
Post by Sameer on Feb 27^th, 2004, 7:25am

I suggested the computerized approach because you mentioned your company does "large-scale" standardized test. If this is the sole motive of the company then investing in a computerized test will be more beneficial down the road. Again I would disagree that computerized tests are erroneous than papers ones - in fact just he opposite. E.g. GRE and TOEFL exams.

I would assume we need correlation between each set of forms to compute whethe a student can score more than he deserved. Needs more thinking. I guess the answer might just lie in towr's individual probability.

Title: Re: Test Scores
Post by Quetzycoatl on Feb 27^th, 2004, 8:29am

on 02/27/04 at 07:25:03, Sameer wrote:

I am not sure what you mean by "sole motive," we do large-scale assessments, we also do computerized testing, but most clients, and in most cases clients are state level educational departments, do not have the money or equipment to test hundreds of thousands or millions of kids on computer.

In response to your disagreeing about the accuracy of computerized testing, I think you may be misunderstanding me. Answers get recorded accuratley nearly %100 of the time. However, the added task of having to know how to use a computer adds an unintended factor to the difficulty of the test questions. There are other factors as well. For example students read the same text slower and less accurately on a computer screen than on paper. There is a whole field of research dedicated to this sort of thing, it is called Psychometrics.

I will note that these issues become less of a factor with test takers of a certain age (18-30ish). Also, psychometricians (who are in essence half psychologist and half statistician) have various ways of adjusting scores to compensate for these things. Tests like the GRE, aimed at adults, are better suited for computers.

Title: Re: Test Scores
Post by towr on Mar 1^st, 2004, 7:48am

on 02/26/04 at 12:01:46, Quetzycoatl wrote:

That still leaves the first question though.

After giving it some though, I have to say that I can't answer it without knowing the distribution of a students scores..
If a student has all answers correct there is zero probability he will get a higher mark than he deserves, if he has every answer wrong there is a high probability one of the keys will give him at least one point.
Assuming all answers in all keys are independant, the chance of getting at least one point is, I think, 1- (a-1)/a ^q(n-1) (1 minus the chance of having them wrong in all keys)
Of course the key would probably be sufficiently different from each other, and thus not completely random (in which case they might in an extreme event all be the same)

Title: Re: Test Scores
Post by Sameer on Mar 1^st, 2004, 10:47am

Can the first question be answered by tackling this question?

Can you design a set of papers such that only the correct key to a paper will maximise your score?

One starting step. Normally all these tests have 5 choices.
Consider a paper with only 1 question. Then we can design 5 papers with each choice as an answer maximising the score only when the key is correct. Can someone expand on this?

Title: Re: Test Scores
Post by towr on Mar 1^st, 2004, 11:26am

on 03/01/04 at 10:47:25, Sameer wrote:

One starting step. Normally all these tests have 5 choices.

Must be regional, since here they are usually 4 (but that's hardly important)

Quote:

Consider a paper with only 1 question. Then we can design 5 papers with each choice as an answer maximising the score only when the key is correct.

?? there are 5 answers, and in each key a different answer (A,B,C,D or E) is correct, so given a random guess there is allways a key that gives the maximum score regardless of whether the student answered correctly.

Just consider this, a student may copy all the answer from a neighbouring student (with a different keycode), and supposing that student has all answers correct, then so will the cheater if he doesn't write down any code.

Title: Re: Test Scores
Post by Sameer on Mar 1^st, 2004, 12:50pm

Ahaa I get your point!!! so that means if all keys are same that should satisfy the condition?

Lets say you have five papers with 1 question each but different questions whose all answers are a). Given the student copies he will maximise it. But I guess you will moderate the test right? If student make a random choice and selects a wrong answer then any key will give him zero points. So I guess best way of designing question papers is have a different set all the time but such that there is only one key!!!

Title: Re: Test Scores
Post by towr on Mar 1^st, 2004, 1:27pm

If you have a good number of questions, you could make different sets where only the order of the questions are changed. (And the correct answer for each question at a certain position is the same letter)
Unfortunately once the students figure out this trick, they can still cheat by copying someone elses answers. (Which is what the key system is trying to prevent)