People who care about such things are talking about an article by Tom DeMarco in IEEE Software entitled Software Engineering: An Idea Whose Time Has Come and Gone? (PDF file). Well, that's what I said 13 years ago. At the time I was taking, and hating, a second-year university software engineering course; I'd just come off a co-op term working in the real world making actual software people would use, and I wasn't too thrilled to be told a pack of lies about how the ricidulous "methodologies" of software engineering were not only helpful, but absolutely necessary, when I'd just seen validation of my long experience as an amateur, writing software and learning what actually worked. I wrote the article below, and it stayed on my Web site for a long time, and then I eventually took it down because I was remodelling, it had never attracted the attention I'd hoped for, and I didn't want to continue maintaining it and a bunch of other old material. This January (2009) one of my friends who actually works in the industry - I did a few more co-op terms and then headed for the academic path instead - was telling me how he thought I should put the "bromides" article back up. He said other people were catching on to similar ideas, he'd even discussed it with some famous software engineering writer whose name I don't remember (could even have been Tom DeMarco, could have been anyone) who had agreed substantially with my friend's description of my ideas, and the only problem was I'd been ahead of the curve writing about it in 1996. So, maybe the time for this article has finally come.

Do bear in mind that this was written in 1996; both I and the business were younger and less experienced at the time. Not everything I say below is still expressed in exactly the terms I would use today. I think the central idea, though, was and still is right. Software isn't and shouldn't be engineering. If we must have a paradigm to understand it we'd be better off using some other, more creative, field of human endeavour. I said "journalism." DeMarco likens it to raising a teenager - and good on him for mentioning that "The very idea of controlling your child ought to make you at least a little bit queasy," which is something else I like to talk about but won't discuss further right here. Now, my old article.

Software Engineering Bromides (June 29, 1996)

I started writing computer software around 1983. I learned BASIC out of a book, and nobody told me anything about software engineering. Back then BASIC was real BASIC - not "Quick", not "Visual", not even "GW". Real BASIC with variable names consisting of either one letter, or one letter followed by one numeral. Real BASIC with line numbers on every line. And LET was not optional. And THEN could only be followed by a line number for the implicit GOTO.

Now, I'm sure that there was software engineering occurring even back then but it was no big deal. I mean, software engineers in 1983 would be the slide-rule boys in an IBM dungeon someplace feeding punch cards into a hopper or punching COBOL on a tape. And IBM's always been a little starched anyway. Welcome to 1996 - everyone's coming out of the closet and software engineers are no exception.

I've been in the industry. Four months of paid full-time employment as a software development co-op student may not seem like much, but as a cap to 13 years of hobby interest/addiction, I think it's enough to qualify me as a commentator. I've written real commercial software, and done it well, and now I'm back in school and the academics are trying to tell me what the real world is like.

Academics teaching software engineering are like virgins teaching sex.

Don't take this as a rant against the University of Victoria's Computer Science 265 class, which I'm currently taking. In fact, please don't - I don't want to get in trouble! Rather, this rant is a rant against the tired, commonplace statements ("bromides") that plague software engineering in general.

Software == hardware

The whole term "software engineering" is suspect. It suggests that software engineering has something to do with engineering. Believe it or not, some people even think that software engineering actually is a kind of engineering. This myth is surprisingly pervasive - lots of programmers graduate with B.Eng. degrees and go off calling themselves engineers, and they get hired as such.

Why do I say that software engineering, isn't? There are several critical differences. For one thing, all other kinds of engineering deal with physical objects of some kind. Bridges, if we have to use a stereotype. Software engineers don't produce anything tangible. Sure you can pretend that a computer file is a real object and you can get philosophical and raise issues about the fundamental nature of reality and the blurred line between hardware and software, but the fact remains: you can jump off a bridge but you can't climb a B-tree.

That may seem irrelevent, but how about this one: physical engineering disciplines are very straightforward. There may be 207 different kinds of bridge, but those are the only reasonable kinds of bridge. You can't think of a new kind of bridge different from those and expect it to work. Engineers will use innovative approaches but it's not their place to invent them from scratch. It is engineering's job to apply the techniques discovered and created by other people.

A junior civil engineer who gets creative with a bridge design will be shown the door in short order; vastly different the situation of a creative hacker. Good software usually involves hacking actual new ideas from the untamed wilderness of the mind. Other forms of engineering are seldom like that. No value judgement either way, I'm just saying the two are different.

If we have to go looking for an established paradigm for creating software, I think writing is the appropriate one. We can write software the way we write text. I think that's a good description of the programming style I use in my personal (not paid) projects. Don't specify, outline. Don't inspect, proofread. When I write code for other people at work or in class, I write it in the style they want. But if they ask me how I think things should be done, I tell them.

Writers have a whole set of established techniques for controlling the creative process, and I think a set of journalism classes would do the average programmer a lot more good than trying to make us build software the way we build bridges. Software isn't like bridges. It's not like magazine articles either, but it's a lot closer. We can, and should, apply the existing procedures for magazine articles to software programming. They'll work a lot better than the procedures we use for bridges.

Like most bromides there is a grain of truth in this one. It's certainly true that a wizardly hacker can go off and write code and then drop dead or leave the company and nobody will be able to work on the code again because it wasn't done in any organized way. I've seen code like that and, I must admit, I've even written code like that. We do need organized approaches to software and it's a good idea to look at other fields for inspiration. I just claim that journalism is a better model for software than is any kind of engineering.

Structured programming

You know they tried to put one over on us before. I'm thinking of the bromide of structured programming. You remember that one? The idea was that we could solve all our software problems by following a certain process. I'm not sure I can summarize the philosophy of structured programming in a few sentences, but it's a very specific and organized way of building code. "Divide and conquer" are watchwords of structured programming - you split the major task of the program into smaller subtasks, split those into even smaller jobs, and eventually you've reduced it to thousands or millions of instructions which constitute your program. This is called "top-down design".

Structured programming also involves code structures which help organize the instructions along the lines of your design. These include loops, if-then-else constructs, subroutines, and so on. A structured program is like a set of nesting boxes, all one inside another, except some of the boxes are more like Klein bottles and contain themselves (that's called "recursion"). When you arrange the little boxes inside the big boxes you aren't allowed to think about what's inside the little boxes because it's not supposed to matter. That's why structured programming discourages the GOTO statement - it's like a hyperspace wormhole that digs into the inside of a box and connects it to the inside of some other box, which means that you can't consider either of them in isolation anymore.

Structured programming is good because software is really complicated stuff. When faced with 200K of source, you absolutely have to have some kind of structuring device if you want to be able to understand the code or explain it to others. In fact, I would say that a lot of the ideas of structured programming are inevitable, because there is no other way to handle big conceptual structures. Funnily enough, however, really great software is often written bottom-up or middle-first or in some other unauthorized way. Why?

I claim that humans normally think in a holistic, all-levels-at-once manner. When we think about a program we want to write we do not, by nature, first think of its major task and then split it down and then split those down and so on. Rather, we jump around and consider several levels at once. "Hmm. I want to write an install program for this server and I could represent the .INI file as a single-linked list and I wonder if the windowing toolkit has an outline control and how many bits is a socket address?"

Good programmers are people who can concentrate on one level at a time. Great programmers are people who can concentrate on one level at a time and still not lose track of the other levels that their minds inevitably generate. You can join high-level systems together better if you have a good idea of how they work on the lowest nuts-and-bolts level. Other (not software) engineers seem to understand that, so maybe it's one thing we should embrace from that tradition.

So, in addition to learning from journalism, we should learn from nuts-and-bolts engineering. These days software is too big for one person to design alone, so we have to have teams and specialization, but that doesn't mean we have to bury ourselves in specialization. You want one person to work on assembly language subroutines and one on combining high-level OOP modules? Fine. But make sure the OOP expert knows what a register is, and give the assembly hacker some abstract algebra courses. They will work better together if they have some notion of what each other are doing.

Good programmers can separate the different abstraction levels of a project. That is, for instance, what the OSI networking model is all about. I've learned that that's an unrealistic ideal. In real life, you can't totally separate the levels. You can't design a transport layer protocol without knowing a little about the network and session layers. You need organization and boundaries between the parts of a program, or everything will fall apart. But those boundaries had better be written in pencil.

Design specs and code inspections

As far as I can tell, the idea of a specification is that it's a good idea to know what something is supposed to do before you build it. This is presented as if it were a profound insight, and taught in programming and management classes at a cost of thousands of dollars per person.

Code inspection, on the other hand, involves the concept that once you've built something, it's a good idea to look at it and make sure you built it right before you try to use it. If you've got the people power, it's a good idea to have someone look at it other than the person who built it in the first place, because everyone's biased about their own creations. Like design specifications, this is a very good idea.

My question is, why are we bothering to talk about these things? Shouldn't it be obvious that specification and inspection are good? By making a big deal of them and writing textbooks and having classes, we're making it too complicated. The academic establishment is perpetuating the concept of programming as a difficult, black art, that you need years of training and hard work to learn.

That's elitism. What ever happened to systems with a programming language in ROM, where everyone who used one learned to program as a matter of course? How come we don't see programming books written for seven-year-olds anymore? A loop is a loop is a loop, and will be while(1). The concept isn't any more difficult now. Modern C with a good compiler isn't really any harder than the cryptic old BASIC interpreters. But now we've bogged ourselves down so much with common sense dressed up as real progress, that we've convinced ourselves programming is hard.

I don't really mind being part of a technological elite. It's kinda fun. But there is real magic in programming, and I'd rather spend my time learning that instead of bromides about work habits. If we're going to call ourselves elite, let's deserve it. Let's study the new and subtle stuff, not the obvious conclusions we should already know.

Conclusions

All the ideas I've attacked here have some validity. That's why so many people believe them unquestioningly. I say our faith in the tenets of software engineering must be tempered with reason. It's not enough to just follow a checklist of top-down design and specification, implementation, inspection, testing, or whatever other methodologies you're going to use. Even the word "methodology" is excessively dignified. You should use a given technique because it makes sense, not because it's next on the checklist. Stop mooing and start thinking, people!

In the highly competitive world of software, I think that sensible approaches must eventually evolve. I would like to see the academic discipline of Computer Science lead the way by discovering some more intelligent ways of thinking about software, ways that don't involve regurgitating checklists of obvious facts disguised as deep wisdom. Isn't discovery what science is all about? But it's pretty comfortable to wear guru robes and sit at the top of the mountain telling novices things they already know. I'm afraid we will see the academic establishment doing that for some time to come.