Leibniz's Dream and Dijkstra's nightmare

I was inspired to browse some of Edsger Dijkstra’s essays today, and came across his speech, “Under the spell of Leibniz’s Dream”.

It’s the sort of personal history I love to read, which gives one person’s sense of the world over a period of change. One quote that immediately struck me was his view of his country’s spirit after WWII:

An important side-effect of the hard times was the creation of a spiritual climate in which the distinction between pure and applied science had vanished: of all the things you could do, you just did the most urgent one, and the development of some urgently needed theory was often the most practical thing to do. It was my privilege to grow up in that spiritual climate, and the conviction that what is theoretically beautiful tends to be eminently useful has been an article of my faith ever since: in the design of sophisticated digital systems, elegance is not a dispensable luxury but a matter of life and death, being a major factor that decides between success and failure.

The idea of “urgently needed theory” driven by practical demands jives with where my mind has been trawling in the last five years. But I find it fascinating how little he credits applications for creating the climate. Instead, he praises beautiful theories for their practicalities, as if their roles were reversed. He spends the rest of the speech largely disparaging the industry outside of academia, and basically calling most of the students and professors in every modern university idiots unworthy of the gift of higher education. If I were feeling generous, I’d say his attitue is a reasonable reaction to the pendulum of history swinging away from rigor and quality for political reasons.

I am also likely biased because I view myself as someone who would have not likely been allowed into the exclusive club of elite academics he wished to preserve. I nevertheless gained enough education within the system he derides to contribute something elegant and useful to the discipline of programming. Though by the way he writes, he would likely consider my output, “chatty instead of crisp,” and akin to “junk science.”

In the speech, Dijkstra discusses the topic of programming languages, praising ALGOL 60 for its elegance and innovations. ALGOL 60 invented things we take for granted today, like lexical scope, an official boolean type, and function definitions, and BNF notation.

Altogether Dijkstra paints a picture of society’s decline into stupidity. Programming is pulled along with it, led by the corrupt idea that programming should be easier for more people. The irony that this is exactly what his beloved ALGOL 60 features provided, especially in their descendants (i.e., all modern programming languages) appears to be lost on him. Instead of designing languages in novel ways to make entire classes of errors unrepresentable, Dijkstra puts personal discipline as the primary requirement and solitary solution for success in programming. In his telling, we should accept that programming is extremely difficult, and write mathematical proofs of program correctness before trying to implement a program. Going in the other direction, what he calls “trial and error” and what we might today call “iterative development” is intellectually bankrupt.

Academia has suffered, for programming methodology could not flourish as a research topic in a world in which it was considered irrelevant. For many a university, this has been a great loss; here and there the situation might be improving, but it is a slow process.

The way I see it, what has actually played out is that the expansion of programming as a discipline, even to the “average programmer,” which Dijkstra calls “fictitious morons,” is the primary reason the pendulum is currently swinging back toward rigor. Programmers repeatedly firing the same footguns has itself become the practical problem requiring the development of an “urgently needed theory.”

Rust is a good example of this. As far as I can tell, it’s the first major programming language designed for memory safety. And while Rust had many historical inspirations, as far as I can tell none even attempted to ensure memory safety. Note: while garbage collection came with the first LISPs, I always think of garbage collection as an implementation detail rather than part of the language’s design. Maybe that is naive.

But far beyond Rust and even the many other modern languages with interesting theory, the 2010s saw a messy explosion of new software tools and libraries. Rewriting, refining, and ad hoc theory building became the norm, leading people to rediscover and synthesize old and new ideas to create a more cohesive software ideology. I remember learning React and Redux in 2016 and having it dawn upon me that React’s statelessness, itself driven by the miasma of global state management that characterized frontend development, drove Redux to reinvent the monad. Examples like this abound.

As all these flowers bloomed, the 2020’s dawned with complexity burnout. What I see as the beginning of a return to elegance and rigor, with new theories to back it, isn’t driven by the leadership of academic purity. Instead, it’s a means to fend off the chaos that encroaches when a community reaches the limits of its analytic capabilities.

In other words, we were so baffled by the jungle we grew that we invented languages like Go, which, in the apocryphal words of its creators, is “so simple even a Google engineer can’t screw it up.” I take this to mean both that Google had hired inexperienced new graduates like crazy, and also that Google engineers are the sorts of crafty bastards to use complicated language features where they are not necessary. While Go’s charter explicitly discourages programming language research, its mandate for simplicity has nevertheless led it to design new elegant theories for version control and improve upon earlier theories of generics.

This could not have happened in the ivory tower by people who, as Rob Pike somewhat clumsily put it, “can appreciate brilliant languages” (implying the rest of us cannot). It could only have happened in Dijkstra’s nightmare. One full of “fat, yellow books titled ‘Such and such for dummies’.” This brings me back to my philosophy, which Dijkstra must have missed in his youth, is that theory builders need a world full of urgent problems. When the dam breaks, structural engineers respond with renewed purpose. When the legacy software jungle is cut and the fibers composted, the new theories and tools planted in the loam move the entire field forward. In time, that same forward leap will deliver humanity a new jungle, unrecognizable from the first in the same way that the jungle of punch card programming is unrecognizable to my contemporaries. Simply put, progress cannot exist in a vacuum.