What have I got in my pocket?

6 thoughts on “What have I got in my pocket?”

  1. Really interesting idea to try to evaluate your qualifying exam!

    The part-whole correlation is a useful index only if you accept an underlying assumption, which is that people who do well on one question should be doing well on the others. That works out in undergrad classes, where there is lots of individual-difference variation in things that should affect performance across the board — ability, motivation, previous experience with the subject, etc.

    That may also be true of Ph.D. students taking a qualifying exam, but there are a couple of reasons I would wonder about it. One is statistical: the students are already heavily selected on being good at physics, and that can distort correlations among the things that go into that. (I wrote a blog post about a related issue not knowing that it was already an established principle in statistics called Berkson’s Paradox, which itself turns out to be a special case of something Judea Pearl calls “conditioning on a collider. The commenters clued me in: https://hardsci.wordpress.com/2014/08/04/the-selection-distortion-effect-how-selection-changes-correlations-in-surprising-ways/)

    The second reason I’d wonder is that Ph.D. students are in a process of specializing, and so some of them may simply be better at some areas of the field than others. I don’t know about physics, but in psychology we don’t even do comprehensive exams. It’s a broad field, and by the time they’re ready to advance to candidacy, students need to have invested so much time specializing. So they do papers or tailored exams in their sub-sub-field. Given a hard enough comprehensive exam, I would expect even our best third-years’ knowledge to look uneven.

    Like I said, I don’t know enough about physics to know whether that applies. Either way, it would be neat if you were also able to collect some kind of criterion information down the road — job market success, quality or impact of papers published in grad school, or whatever you think is appropriate.

    1. Hi Sanjay,

      These are very good points. I think it is likely, though, that the assumption “that people who do well on one question should be doing well on the others” is a fairly good one even for our Ph.D. exam. Students take it in their first year (and sometimes in their second) — before they specialize in any particular area. Also, it is essentially an undergraduate-level exam, and covers “standard” topics that everyone has (at least on paper) seen before, so one would expect some good amount of correlation based on the existence of the sort-of-standard “canon” of physics education. There are four main subject areas covered by the exam (classical mechanics, quantum mechanics, electromagnetism, and statistical mechanics), and it would be useful to see if correlations within subjects is, as one would hope, better than between subjects. We’ve never done this. It might be easier to do in the future — not that it should be hard now — since some of us are proposing an explicit partitioning of the exam into four sub-exams.

      About your first point, that we’re already selecting for students who are good at physics, so correlations become weak or meaningless: this is tougher to think about. (I remember your ‘burgers and fries’ post, by the way — it was neat!) Mean scores on our exam tend to be low (~60%), and I would expect that this Berkson’s Paradox is more pronounced if ceilings are important — i.e. without much dynamic range to discriminate with. I’m not sure of this, though. I’m also not sure if Berkson’s Paradox applies to the sort of intra-exam correlations I’m thinking about here. It seems more relevant to the question of correlation between the exam performance and student “quality,” since we’ve selected students based on some \*combination\* of exam-taking abilities plus other characteristics, and then we give them an exam.

      I strongly agree with your last point, that it would be great if we looked at some sort of data on student “success,” by whatever metric, even subjective faculty opinions. I’ve brought this up before. We don’t do it.

      Random comment: I met someone a month or two ago who seemed quite sure I was you. I restrained myself from pointing out that I’m the other Indian…


      1. Well that makes a lot of sense. It sounds like you do qualifying exams very differently than we do (I should have known better than to assume!). And I agree that the Berkson’s issue may not apply. It would be relevant if there were distinct physics competencies that affected selection *and* that mapped onto different exam questions. Which sounds like it isn’t the case. (That’s also probably a discipline difference – in psych, many students start specializing before they even get to grad school.)

        BTW 3 out of 8 social/personality psychology faculty at UO are men of South Asian descent (which is a funny little cluster, it is not at all representative of the field). As you might imagine we get mixed up from time to time. Even more opportunities for you to get mistaken for a psychologist!

  2. I had a student that did poorly on any question that required memorization (this student would write gibberish words in place of Drosophila gene names, for example) but had amazingly clear explanations of any kind of mechanistic question. Other students show proficiency at certain kinds of learning and do poorly in others. I think any class has too few questions to develop a good profile, but it would be cool (and time-consuming) to tag every question in every class with the kinds of thinking needed to answer it and develop individual training plans to work on weak areas.

      1. Hmm, I was just trying out a sequence assembler that uses a Bloom filter to reduce memory usage, but it looks like Bloom was not an exceptional polymath but instead 2 people.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s