In previous posts, I discussed why I use partial credit multiple choice in my courses and then presented instructions for automating this process using a little Python program. In this post, I’ll go through each input file that is used by the program and then discuss its output.
The raw MC output
The script expects an input file with the raw data scanned from the sheet. The file is structured with one student per line, something like this:
lzkpsdqh 213442311421343211440
- Characters 0 (“l”) through 7 (“h”) are the student’s unique ID tag. Here at Miami they correspond to the first 6 letters of the last name and then other initials and some random numbers as needed.
- Character 9 is the form number (“2”).
- Characters 10 onward are the responses (“13442311421343211440”). The numbers 0–4 represent answers “A”–”E”.
- Blank responses for the form number or question responses are listed as “.”.
In addition to all the students in the class, the program is expecting the answer key, given the unique ID “00000000”, at the top of the file.
The grading key
The key itself looks like this:
0 0 4 0 0 0 4 0 0 0 4 0 0 2 0 0 2 4 2 0 4 0 0 0 0 0 0 0 4 0 0 2 0 4 0 0 4 0 0 0 0 0 2 4 0 0 0 4 0 0 0 0 0 4 2 0 4 0 0 0 4 2 0 0 2 0 0 0 4 0 0 0 0 4 2 0 0 0 0 4 0 0 2 4 0 0 4 0 0 0 0 0 0 0 4 2 0 2 0 4
The format is pretty simple. One question per line with five columns, each corresponding to a response A–E. Each cell assigns a point value to one of the choices. I use a text editor that numbers each row, which makes it a bit easier to work with the file. I typically just use integers, but it can accept numbers with decimals just fine.
The test scramble
1 13 2 16 3 15 4 14 5 17 6 6 7 2 8 11 9 5 10 20 11 8 12 9 13 19 14 12 15 4 16 18 17 1 18 7 19 3 20 10
The test scramble is fairly straightforward. Two columns, the first is form 1, the second is the corresponding questions on form 2. In theory this should be able to handle more than two forms, but in all honesty I’ve never tested this out.
The output
The output of the program is a single text file that can be divided into three parts. If no filename is specified, the output is just dumped to stdout.
In the first section (after the title) we have some general statistics for the test as a whole. Should be pretty self-explanatory.
MC GRADER TEST RESULTS
======================
Overall average: 52.55 out of 80.00 points (65.69%)
Standard deviation: 12.53
High: 80.0
Low: 20.0
The second section goes through the exam question by question.
Average Scores by Question
--------------------------
1: 1.51 Key: - - 4.0 - -
Frequency: 37.7 10.0 37.7 6.9 6.9 (%)
Top 27%: 20.0 8.6 65.7 2.9 2.9
Bot 27%: 45.7 2.9 20.0 14.3 14.3
2: 2.49 Key: - 4.0 - - -
Frequency: 9.2 62.3 18.5 10.0 0.0 (%)
Top 27%: 0.0 91.4 5.7 2.9 0.0
Bot 27%: 17.1 34.3 28.6 20.0 0.0
- The question number is followed by the average points scored on that question (1.51/4 for Q1…yikes).
- The “Key” line gives the points awarded for each choice, including partial credit.
- The “Frequency” line gives the fraction of the entire class choosing each answer. For example, 62.3% of the class chose “B” for Q2, which is the correct answer.
- The “Top 27%” and “Bot 27%” lines give the fraction of the top and the bottom of the class choosing each answer. For example, for Q2, 91.4% of the top 27% of the students chose “B”, whereas only 34.3% of the bottom 27% of the class chose it.[1. No idea why it’s 27%, I got this number from the Scantron output that we get from Test Scoring. Must have some statistical significance. Feel free to comment if you know the significance of this number.] These numbers can be used to identify problem questions that may have been misleading.
Finally, we have the individual student scores. This is pretty self-explanatory. In future versions of the script, I’ll probably build in individualized output that can be returned to the students, but for now I use the printouts I get from Test Scoring and these numbers just get pasted into my gradebook.
Student Scores -------------- 00000000 80.0 adpxnkvx 42.0 aeaekfhv 60.0 apqaagpo 46.0 aqnioyxy 24.0 batmtdas 56.0 bmehvlxa 42.0 bxhjvmcr 50.0 cduorntq 36.0 ceidimzy 54.0 cxoylump 52.0 dhjfifud 46.0 dllsvqcm 58.0
So, that’s the input and output of the script. If your school follows the same format as mine, enjoy! In the next post, I’ll go through the logic of the script itself so that it can be customized to whatever data you have.