I ususally start by making a list of content areas that are covered by the exam, and then for each list the concepts that I would like to explore. For example, I might list linear regression as the content topic and then the meaning of least squares and how to identify potentially infouential observations as being among the concepts that I want to test. Then for each topic, I try to come up with a reasonable simple, but hopefully interesting context for the problem. As I mentioned in earlier responses, I am an advocate of using real data and strive to do that in in-class examples, homework problems, and project assignements. Even so, I often use realistic rather than real data on exams in order to be able to keep the contexts realively simple--given the time constraints of an exam I don't want students to have to spend an inordinate amount of time trying to understand the context. But, context is essential; if there is no context, you can't evaluate a students ability to interpret the results of statistical analyses.

Once I have a set of candidate questions, I start to worry about the length of the exam, and pare down as seems warranted. I work through the exam, and if I can complete it in 10 minutes or less, then I think it is OK as a 50 minute exam for students. I don't usually have another instructor review the exam, mainly because I am not always working far enough ahead of time, but I probably ought to do this more often.

As far as deciding if, after all is said and done, it was a good exam or not, I guess I don't really do any kind of formal analysis like some of the others have suggested they do. Mainly, if most students are able to finish the exam in the allotted time, I get a reasonable distribution of scores, and I wasn't toooo depressed by the collection of responses to any of the questions, I'm happy.

See all answers to this question

See all of Roxy's answers