How is it that computers can drive cars, interpret MRI scans, and recognize our children in photographs? For decades, we hand-crafted image analysis algorithms for computers to follow. Now, we set computers loose with more general algorithms and mountains of data, and the machines learn on their own. Both of these approaches are fascinating, incorporating threads from not just computer science, but physics, biology, and other fields.
Despite the enormous interest in artificial intelligence these days, there’s strikingly little in the pop-science literature about computation, whether the workings of algorithms or the context behind their emergence. Computational image analysis in particular abounds with elegant solutions to challenging problems. I decided, therefore, that computer vision would be the theme of my second pop-science book. My first book, So Simple a Beginning: How Four Physical Principles Shape Our Living World (2022), was about biophysics, i.e. how physical principles help us make sense of the living world – see here for a blog post, and here for the publisher site.
Beyond raising general awareness, I have two other goals for my new book. One is to document a lost art. Conjuring algorithms by hand is becoming ever rarer, and those of us who have meditated on how to translate abstractions of shapes and forms into computer code may be the last of our sort.
The second goal is, perhaps paradoxically, to explain modern artificial intelligence, the very framework that is making classical methods obsolete. Despite its revolutionary importance, there are remarkably few good explanations of how today’s AI works. This isn’t too surprising; it’s a daunting topic. Image analysis provides a path to understanding. Many of the key developments in artificial intelligence emerged from computer vision approaches, so understanding the latter makes the former seem natural. Moreover, image analysis lends itself to visual explanations. In this book, like my other one, I’m drawing or painting the illustrations myself.
How We Taught Computers to See starts with classical computer vision and ends with the transformer architecture that underpins modern “large language models” like Claude and ChatGPT. We’ll see how we taught a robot to roll through rooms in the 1960s and how we figured out how protein molecules walk inside cells in the early 2000s, examples of the vision tasks of identifying edges and points, respectively. We’ll see contests won by machines and controversies about whether neural networks can learn. (Spoiler: they can.).
The book will be published by Princeton University Press, who also published So Simple A Beginning and who have been wonderful to work with. My completed manuscript is due in February, 2027, so the book will hopefully be published later that year. I’ve written about three quarters of it so far, and I expect to meet the deadline!
If you’d like to get email updates about the book, the publication date for example, you can enter your address here. If you have comments, feel free to email me or write them on this post.
I’ve enjoyed all the writing I’ve done for this so far — all by myself, with none of the text written by AI. I firmly believe that “writing = thinking,” and that much of the value of written work comes from it being a form of personal expression. I’m happy to offload much of my computer programming to machines, but not my prose.
Today’s illustration…
One of my illustrations for the book! (Here in color; the published version will be grayscale.)
— Raghuveer Parthasarathy, November 6, 2026
