1. In the weeks leading up to the exam, I look for data sets. (They must be both real and new, where "new" means not in the book, and not on any previous exams or quizzes ever given here.)
  2. Next, I write questions tied to the data sets, based on what I consider a reasonable analysis given what we've done so far in the course.
  3. Next, I look duplications (I won't ask for the same thing to be done to two different data sets) and omissions. (Sometimes I have to find another data set to make sure that everything gets covered.)
  4. Then I break the questions into equal size parts, to help students plan their time.
  5. Finally, I check for length by taking the exam myself. However, since I try to allow students extra time, I don't worry a lot about getting the length exactly right.
I suppose you could call this a "data driven" approach to creating the exams. Steps 1 and 2 are the important ones; the rest is mainly grooming. I really do allow the data sets to suggest the questions, based on what I consider a reasonable analysis would look like. My hope is that this process helps keep the exams more like what statisticians actually do than would be the case with a different process. If, as I plan the exam, there are gaps in coverage, that tells me I need to find an additional data set of a different sort.

See all answers to this question

See all of George's answers