Author |
Topic: Test Scores (Read 556 times) |
|
Quetzycoatl
Newbie
Gender: 
Posts: 46
|
 |
Test Scores
« on: Feb 26th, 2004, 9:51am » |
 Quote  Modify
|
I am not sure how difficult this is so I am sticking it in Medium:
I work for a company that designs, administers and scores large-scale, standardized tests to students K-12. For a particular test and grade we can have up to as many as 100 different versions of a test, all testing the same subject matter. This is to increase security and reduce cheating among other things. Each of these versions are differentiated by a form number which each student must fill in on their answer sheet in order for it to be scored properly. Without it, there is no certain way to know which answer key to use.
Sometimes students will leave this form number blank and, for some tests, the procedure is then to check it against every answer key and use the one that would yield the highest score. In most cases this works fine and the student recieves the correct score. However, the worse a student performs on a test, the more likely their test will be scored with the wrong key.
My questions are:
What is the probability that a student who does not fill in the form number will receive a higher score than they deserve?
How badly does a student have to do on a test in order to gain an advantage by not filling in the form number?
DISCLAIMER: I should probably emphasize that this procedure is rarely used, and when it is, it is only on what we call low-stakes tests, i.e. tests that will NOT have a bearing on things like moving on to the next grade, getting into college etc.
|
| « Last Edit: Feb 26th, 2004, 10:11am by Quetzycoatl » |
 IP Logged |
|
|
|
Sameer
Uberpuzzler
Pie = pi * e
Gender:
Posts: 1261
|
|
Re: Test Scores
« Reply #1 on: Feb 26th, 2004, 11:11am » |
Quote Modify
|
Options:
- Computerise the tests!!!
- Give the answer sheet attached to the question sheet with pre-printed form number.
|
|
IP Logged |
"Obvious" is the most dangerous word in mathematics.
--Bell, Eric Temple
Proof is an idol before which the mathematician tortures himself.
Sir Arthur Eddington, quoted in Bridges to Infinity
|
|
|
towr
wu::riddles Moderator
Uberpuzzler
Some people are average, some are just mean.
Gender:
Posts: 13730
|
|
Re: Test Scores
« Reply #2 on: Feb 26th, 2004, 11:18am » |
Quote Modify
|
Quote:| How badly does a student have to do on a test in order to gain an advantage by not filling in the form number? |
|
He allways has an advantage, either he'll get the grade he deserves, or a higher one. It's a no-lose situation.
If he doesn't have all the questions correct there is a probability of (n-1)/aq that there is a key that does fit all his answers (where n is the number of keys, a the number of choices per question and q the number of questions)
Only if the student has all the answers correct is there no expected gain, but it's doubtfull he can be sure of that.
|
|
IP Logged |
Wikipedia, Google, Mathworld, Integer sequence DB
|
|
|
Quetzycoatl
Newbie
Gender:
Posts: 46
|
|
Re: Test Scores
« Reply #3 on: Feb 26th, 2004, 12:01pm » |
Quote Modify
|
Good point Towr, I had not thought that through. That still leaves the first question though.
Sameer: we do, in some cases do computerized tests, but they are very expensive compared to paper and many school systems simply do not have enough computers to make this feasible. Also tests on computer are more difficult and less accurate than on paper. You would think in this day and age that the affect would be slight but it's not. If you are interested, do a search on Psychometrics and computerized testing.
We also do something similar to your other suggestion at low grade levels, which are less reliable in recording things like form numbers. However, this is also more expensive and when you consider how slight a problem this is (like I stated before, checking against all keys works most of the time), any additional cost is simply not worth it.
|
|
IP Logged |
|
|
|
Sameer
Uberpuzzler
Pie = pi * e
Gender:
Posts: 1261
|
|
Re: Test Scores
« Reply #4 on: Feb 27th, 2004, 7:25am » |
Quote Modify
|
I suggested the computerized approach because you mentioned your company does "large-scale" standardized test. If this is the sole motive of the company then investing in a computerized test will be more beneficial down the road. Again I would disagree that computerized tests are erroneous than papers ones - in fact just he opposite. E.g. GRE and TOEFL exams.
I would assume we need correlation between each set of forms to compute whethe a student can score more than he deserved. Needs more thinking. I guess the answer might just lie in towr's individual probability.
|
|
IP Logged |
"Obvious" is the most dangerous word in mathematics.
--Bell, Eric Temple
Proof is an idol before which the mathematician tortures himself.
Sir Arthur Eddington, quoted in Bridges to Infinity
|
|
|
Quetzycoatl
Newbie
Gender:
Posts: 46
|
|
Re: Test Scores
« Reply #5 on: Feb 27th, 2004, 8:29am » |
Quote Modify
|
on Feb 27th, 2004, 7:25am, Sameer wrote:I suggested the computerized approach because you mentioned your company does "large-scale" standardized test. If this is the sole motive of the company then investing in a computerized test will be more beneficial down the road. Again I would disagree that computerized tests are erroneous than papers ones - in fact just he opposite. E.g. GRE and TOEFL exams.
|
|
I am not sure what you mean by "sole motive," we do large-scale assessments, we also do computerized testing, but most clients, and in most cases clients are state level educational departments, do not have the money or equipment to test hundreds of thousands or millions of kids on computer.
In response to your disagreeing about the accuracy of computerized testing, I think you may be misunderstanding me. Answers get recorded accuratley nearly %100 of the time. However, the added task of having to know how to use a computer adds an unintended factor to the difficulty of the test questions. There are other factors as well. For example students read the same text slower and less accurately on a computer screen than on paper. There is a whole field of research dedicated to this sort of thing, it is called Psychometrics.
I will note that these issues become less of a factor with test takers of a certain age (18-30ish). Also, psychometricians (who are in essence half psychologist and half statistician) have various ways of adjusting scores to compensate for these things. Tests like the GRE, aimed at adults, are better suited for computers.
|
|
IP Logged |
|
|
|
towr
wu::riddles Moderator
Uberpuzzler
Some people are average, some are just mean.
Gender:
Posts: 13730
|
|
Re: Test Scores
« Reply #6 on: Mar 1st, 2004, 7:48am » |
Quote Modify
|
on Feb 26th, 2004, 12:01pm, Quetzycoatl wrote:| That still leaves the first question though. |
|
After giving it some though, I have to say that I can't answer it without knowing the distribution of a students scores..
If a student has all answers correct there is zero probability he will get a higher mark than he deserves, if he has every answer wrong there is a high probability one of the keys will give him at least one point.
Assuming all answers in all keys are independant, the chance of getting at least one point is, I think, 1- (a-1)/a q(n-1) (1 minus the chance of having them wrong in all keys)
Of course the key would probably be sufficiently different from each other, and thus not completely random (in which case they might in an extreme event all be the same)
|
|
IP Logged |
Wikipedia, Google, Mathworld, Integer sequence DB
|
|
|
Sameer
Uberpuzzler
Pie = pi * e
Gender:
Posts: 1261
|
|
Re: Test Scores
« Reply #7 on: Mar 1st, 2004, 10:47am » |
Quote Modify
|
Can the first question be answered by tackling this question?
Can you design a set of papers such that only the correct key to a paper will maximise your score?
One starting step. Normally all these tests have 5 choices.
Consider a paper with only 1 question. Then we can design 5 papers with each choice as an answer maximising the score only when the key is correct. Can someone expand on this?
|
|
IP Logged |
"Obvious" is the most dangerous word in mathematics.
--Bell, Eric Temple
Proof is an idol before which the mathematician tortures himself.
Sir Arthur Eddington, quoted in Bridges to Infinity
|
|
|
towr
wu::riddles Moderator
Uberpuzzler
Some people are average, some are just mean.
Gender:
Posts: 13730
|
|
Re: Test Scores
« Reply #8 on: Mar 1st, 2004, 11:26am » |
Quote Modify
|
on Mar 1st, 2004, 10:47am, Sameer wrote:| One starting step. Normally all these tests have 5 choices. |
|
Must be regional, since here they are usually 4 (but that's hardly important)
Quote:| Consider a paper with only 1 question. Then we can design 5 papers with each choice as an answer maximising the score only when the key is correct. |
|
?? there are 5 answers, and in each key a different answer (A,B,C,D or E) is correct, so given a random guess there is allways a key that gives the maximum score regardless of whether the student answered correctly.
Just consider this, a student may copy all the answer from a neighbouring student (with a different keycode), and supposing that student has all answers correct, then so will the cheater if he doesn't write down any code.
|
| « Last Edit: Mar 1st, 2004, 11:29am by towr » |
IP Logged |
Wikipedia, Google, Mathworld, Integer sequence DB
|
|
|
Sameer
Uberpuzzler
Pie = pi * e
Gender:
Posts: 1261
|
|
Re: Test Scores
« Reply #9 on: Mar 1st, 2004, 12:50pm » |
Quote Modify
|
Ahaa I get your point!!! so that means if all keys are same that should satisfy the condition?
Lets say you have five papers with 1 question each but different questions whose all answers are a). Given the student copies he will maximise it. But I guess you will moderate the test right? If student make a random choice and selects a wrong answer then any key will give him zero points. So I guess best way of designing question papers is have a different set all the time but such that there is only one key!!!
|
|
IP Logged |
"Obvious" is the most dangerous word in mathematics.
--Bell, Eric Temple
Proof is an idol before which the mathematician tortures himself.
Sir Arthur Eddington, quoted in Bridges to Infinity
|
|
|
towr
wu::riddles Moderator
Uberpuzzler
Some people are average, some are just mean.
Gender:
Posts: 13730
|
|
Re: Test Scores
« Reply #10 on: Mar 1st, 2004, 1:27pm » |
Quote Modify
|
If you have a good number of questions, you could make different sets where only the order of the questions are changed. (And the correct answer for each question at a certain position is the same letter)
Unfortunately once the students figure out this trick, they can still cheat by copying someone elses answers. (Which is what the key system is trying to prevent)
|
|
IP Logged |
Wikipedia, Google, Mathworld, Integer sequence DB
|
|
|
|
RIDDLES SITE
WRITE MATH!
Home 
Help 
Search 
Members 
Login 
Register
Reply
Notify of replies
Send Topic
Print