The perils of online learning, in one graph

I’ll touch on the important question of how students learn from in-person vs. remote courses by showing just one graph based on the class I taught last term. Adding a bit of commentary, I note a conundrum that I think universities have avoided addressing..

Plotted below are the mean and standard deviations of the overall course grade and final exam grade for subsets of the 120 students who, as self-reported…

(1) attended class in person almost always (>90% of the time);

(2) attended class in person more than 50% but less than 90% of the time;

(3) attended class in person less than 50% of the time, and attended remotely but synchronously via Zoom; or

(4) attended class in person less than 50% of the time, and watched recordings of the majority of class sessions.

Summary: On average, students who “attended” by viewing classes on Zoom or by watching recordings more than half of the time scored an entire letter grade lower than those who attended in person more than half of the time. (Note that there is a lot of variation in the scores for each category.)

The class was the Physics of Energy and the Environment for non-science-major undergraduates at the University of Oregon that I’ve taught several times before, and written about before (2021, 2017, 2016).

The unintended experiment. As the Covid-19 pandemic continued but with much less severity than before, thanks to exceptional vaccines and a campus vaccination rate of 95%, we held classes in person but were required to make class content accessible to those who couldn’t be present. An easy way to do this, especially for large classes, was to turn on Zoom and broadcast it “live,” and / or to record the Zoom sessions and post the recordings. I did both. In principle, only ill/quarantining students would make use of this, but in practice throughout the university, the result was lots of students not showing up, typically around 50%. Not only is this far higher than the fraction actually affected by Covid, it was clear from conversations that many students simply made use of the convenience of not attending. Zero people contacted me about long-term health issues, having to care for others, etc. (Based on people asking for accommodations for all sorts of things, some reasonable and some not, I’m sure I would know if any sizeable fraction of students was actually prevented from being in person for Covid/care-related reasons.) Note that I’m using as the threshold for my categories being remote/online for >50% of classes. Plenty of people tested positive and had to miss ~10% of classes (a week).

Problems with the data. I didn’t keep track of attendance, so the plotted attendance categories are self-reported from a survey question I asked with the final exam. The numbers may be inaccurate — from a similar question earlier in the term, I think students over-state their in-person attendance. This could be due to the difficulty of remembering, or a desire to please me with the “right” answer. I might be able to validate some of the numbers by checking Zoom session participants, but this would take some work.

Problems with the class. It’s very hard to do a hybrid (in person + online) class well, and I certainly didn’t do this one well. It would have taken far more time, energy, technical skill, and institutional support than I could muster to make a high-quality class for both audiences, and I focused on those present in front of me. The remote course experience was therefore mediocre. (Though one might think its obvious low quality would spur greater in-class attendance.)

Correlation and Causation. The data show that not attending class is, on average, correlated with doing poorly. It’s not obvious which direction causation goes, however. It could be that the weaker students are the ones who choose not to attend, or that not attending in person makes it harder to learn, or that both are true. I suspect that both are true, but I don’t have a good justification for this. I think the class as a whole learned less — some questions I asked that were identical to previous years’ got fewer correct responses this term — suggesting that not being present in person caused, at least in part, worse performance, but I haven’t systematically looked at the data.

Should we care? Even assuming that it is the case that not attending class in makes it harder, on average, for students to learn, should we care? In an aggregate sense, yes: I think it’s good for society to have as many people as possible with an understanding of the physics of energy and the environment. In an individual sense, it’s less clear: students may legitimately assign more value to spending time on their sofa than to getting a “B” versus a “C”, and it would be presumptuous and condescending to say this is bad. (Academics in general have a hard time accepting this, or accepting that being a good person is uncorrelated with pursuing academic success.) A grade is, or should be, simply an assessment of how much one learned, and there’s nothing good or bad, noble or contemptible, beyond that. In addition, universities are already too paternalistic, and granting students the autonomy to fail is important.

Moreover, one certainly can do well in online courses — many of the remote attendees got high grades, and I did very well with a completely disembodied course I took a few years ago. In fact, one can do very well with just a book, motivation, and self-discipline, as any of us who routinely teach ourselves topics can attest. (This is an important skill for success in all sorts of things!) Given this, should we be bothered by my graph?

I would argue that the answer to “Should we care?” is still “yes,” though not as strong a yes as one might initially think, because I don’t trust that many students understand that motivating oneself outside the structure of an in-person classroom is difficult. We’re social animals — we pay much more attention to a live person talking to us than to a recording, or even a simultaneous video. It’s hard to resist the temptation to crank the speed up to 2X and delude oneself into thinking that one is absorbing knowledge. The issue, then is one of informed consent: we should allow people to fail, making whatever choices they like, but we should try to ensure that they understand the risks associated with their choices.

The conundrum. This term we’re more or less back to normal, so the problem illustrated by my graph has disappeared. But it resurfaces in other, larger forms. Online education isn’t going anywhere (and on balance, this is good), and my university has decided that we should aim for 20% of our credit hours to be from online classes. What will students learn from these courses? What will students understand, about themselves and about how learning works, when they sign up?

The more fundamental puzzle is this: our role as educators at a university involves some combination of presenting material to students, coaching them in how to learn, and assessing how much they’ve learned. We’ve never been very clear on the relative weights of these three, either to ourselves or to students. Changes in technology make our confusion even more consequential.

Obligatory links to my book, for those who haven’t yet taken a look: My description; publisher’s page; Amazon page. Many people have told me they like it, which is very uplifting!

Today’s illustration A quick painting of a lime.

— Raghuveer Parthasarathy; March 26, 2022