In my last post I wrote about student course evaluations, with the suggestion that we’d glean more useful information from them if they occurred well after a course ended. With a delay, students could reflect on whether the course impacted their views of the world or provided skills that proved to be useful. Responses might be hard to gather, but I’d guess that the smaller number of responses would provide greater insights into the quality of the course (or lack thereof).
A few people (hi Julie and Professor X!) pointed out that there has been a bit of work on seeing whether student course evaluations are stable over time, tracking the same students and asking about their opinions on a course both immediately after it ends and later (i.e. a longitudinal study), and that there’s a claim that the later responses are quite similar to the immediate ones. I spent a bit of time looking into this. Though there are a good number of claims and comments about this, I can find remarkably few actual studies. The most relevant:
I can find remarkably few studies that report “longitudinal” student course evaluations. The only “real” study I dug up is:
J. U. Overall, H. W. March, “Students’ evaluations of instruction: A longitudinal study of their stability.” J. Educ. Psychol. 72, 321–325 (1980). [Abstract link]
The authors followed about 1400 students in California, “undergraduate and graduate business administration majors from 100 different classes,” comparing their responses to course evaluation questions immediately after and at least 1 year after they completed their degree program. The authors do find strong correlation between the initial and later evaluations by the same students, arguing against my suggestion that students’ thoughts on the worth of the course may change. However, there are some important caveats:
(1) The study is old (1973-1977), and I would guess that the present demographics, goals, and expectations of students may be quite different. (In 2016, about 33% and 34% of men and women in the U.S., respectively, had completed four or more years of college; in 1977 it was 19% and 12%; source.)
(2) More importantly, though correlation of mean scores for the various evaluation items is similar immediately at the end of class and later, these items are generally quite superficial — “Class presentations prepared and organized,” and “Instructor involved you in discussions,” for example. Only one deals with learning: “You learned something of value,” and this is quite vague (“something”? Really?!) As such, I’m not surprised that the overall correlation over time is strong: if I’m asked for my impressions of how the experience of a course is, then, as long as my memory is good, those impressions should be stable. I am unlikely to change my view of whether the class presentations were organized — it’s not like I’d later get new information about their organization. On the other hand, I may well get new insights into whether the course was meaningful or not. These surveys don’t address that, and so there’s not a lot to gather from their longitudinal consistency.
It’s possible there are more studies out there that I haven’t been able to find. If so, feel free to let me know!
On the issue of getting few responses from students long after a course ends: this would certainly be the case, but I’d maintain that the quality of responses would make up for this. (Of course, the sample would be biased, which would also need to be dealt with.) As for how big the attrition would be, there’s a neat longitudinal study of student assesment of an overall program (not specific courses) from the Univ. of Delaware, http://cms-content.bates.edu/prebuilt/bauer.pdf, following students from 1993-1997. They find a “cohort mortality” of about ~50% after year 1, then ~75% per year following. Overall, the number of responses dropped from 1700 to 270 after 4 years.
I find it interesting that this topic isn’t studied more. It’s not clear who would have the incentive to investigate it, though.
Today’s illustration
Since this isn’t a “real” post, but just a brief addendum, today’s illustration isn’t a “real” one, but just the beginning of a drawing of a potassium channel.
The Eighteenth Elephant 