I am sometimes asked how one goes about writing an article for Overload. I usually rattle off an email with a few random thoughts about getting the text down and editing it into shape. This editorial is my attempt to properly address the topic.

But, why write?

In The Elements of Style [1], Strunk and White describe the rewards of writing as:

“[The writer] will find it increasingly easy to break through the barriers that separate him from other minds, other hearts – which is, of course, the purpose of writing, as well as its principle reward.”

Well quite, but people write either because they enjoy it, or because they know that it’s good for them. I’m definitely in the latter group. Words do not flow from my fingertips in the same way that code does, but I know that I have benefited from improving the quality of my writing. Strunk and White continue:

“…the act of composition, or creation, disciplines the mind; writing is one way to go about thinking, and the practice and habit of writing not only drain the mind but supply it, too.”

The distinguishing quality of a senior engineer is their ability to communicate ideas clearly and efficiently. I’ve seen talented engineers limit their progression, unable to project themselves beyond their immediate workgroup, because they do not take up opportunities to write or conduct presentations.

Selecting a topic

Selecting a topic can be the hardest part of writing an article. We all have a vast array of thoughts spinning around inside our heads, but it can be hard to pin down one that we think is interesting enough to present to others.

Don’t judge your ideas too harshly. Something you think of as simplistic and well known will turn out to be multi-faceted and interesting under further examination.

Don’t try to cover too much. It can be overwhelming to write about the architecture of an entire system, or even a hundred lines of code. Some of the best Overload articles are those that carefully examine a pattern, an idiom, or even a single phrase or keyword.

People naturally want to write about their successes, but we actually learn more from our failures. Learning to examine and share our failures is an important developmental milestone for us all, both as individuals and as professionals.

‘Write about what you know’, is the most common advise given to prospective authors. Indeed, draw from your own work experiences. Base your writing on problems you are trying to solve, and the solutions that could be deployed.

Conversely, I find that ‘write about what you don’t know’ to also be true. A difficulty with writing about what you know, is that the topic material can be so ingrained in your being that it is no longer at the forefront of your mind. The process of researching and documenting a new topic can be easier than the introspection required to dredge up the reasoning for the everyday assumptions under which you operate.

Overall I’d say that the journey is more interesting than the destination. For me, the process of solving a problem is more interesting than a statement of the solution. For example, when I interview engineers I look for people who know how to go about solving a problem, rather than people who know the solutions to problems.

Audience

In all writing it is important to consider your audience. This is simple for Overload; your readers are people just like you. I have in mind professional software engineers who are self-educating but busy people. They are seeking a forum of peers in which to share their thoughts, learn from others, and discuss ideas.

Planning

Inexperienced writers often skip this important stage. Would we start coding before designing? Perhaps that’s not the best analogy, but a simple plan can keep an article on track. Without a plan, the tendency is to produce a wayward collection of random paragraphs. The editing process then retrospectively imposes a plan, which rarely works out well.

Developing

This is the process of collecting the material that will make up your article. I personally have most trouble with this stage. I’m unable to capture all my thoughts on a topic in a single session. I read background material and build up a sheaf of hand-written notes before I reach for the keyboard. Jotting down potential sub-topics, key phrases, and supportive material like examples, references, and quotations helps me a lot.

Avoid starting by writing code. The text is more important than the code. The article should be about the writing the code, not what the completed code looks like.

Organizing your material

Given a plan and a collection of notes you can now develop an outline of the article. The outline pulls together related material, showing how ideas are grouped and related to each other. This process helps bring balance to the article by ensuring sufficient coverage for each point.

Rough draft

At this stage it is important for you to adopt a state of mind where getting any text down is more important than getting the perfect text down. Forget grammar, punctuation, and spelling, just get started. Possible ways to approach a draft are to:

Start at the beginning – Writing the introduction is a natural place to start and planning the route will ease the journey. I often start with the introduction, but with the explicit assumption that I’ll throw most of it away in the first revision. This seems to help me get going.

Start at the end – Writing the conclusion first makes a clear statement about the destination.

Start in the middle – Start with the part of the document that you feel most confident about.

Throw one draft away – Just assuming that the first draft is to be thrown away can help grease the writing wheels. When the draft is done you may decide it’s good enough not to bother starting from a blank page again.

Develop alternative drafts – Writing multiple drafts from alternative perspectives can help you to find the best way to approach a topic.

The difficulty with getting started is psychological; you must be in the right frame of mind to write. Any number of factors can contribute: time pressure, the location, or distractions. Write where and when you feel comfortable. Schedule time specifically for writing. Turn down the ringer on the phone. Close your office door (if you are so blessed). Go home. Go to work. Go to the library. Shut down your email client. Above all just focus on the present and set yourself achievable goals.

My favorite technique is to send myself an email. I feel totally uninhibited writing email messages. I can crank out a paragraph in about thirty seconds, a paragraph that might take me an hour with a word processor.

Editing the rough draft

Having completed the rough draft it’s best to take a break from writing, so that you can return to the text with a fresh mind. I find a good night’s sleep works for me, others may prefer a couple of days, or even a week. The rough draft should be edited for substance rather than language. Don’t waste time fixing up the text for publication, concentrate on which points should remain, and which should be eliminated.

Now is a good time to think about the length of the article. Overload magazine has no minimum or maximum article length restrictions, but we usually serialize articles longer than five pages. We rarely receive short submissions, which is unfortunate as they are very useful when composing an issue.

Revising the first draft

Check the draft against the plan. Has the objective has been achieved? Is the message clear, and has the main point been adequately addressed?

Test the draft against the outline, revise the organization to group ideas together and put them in the proper order. Balance the main points of the article so that they get equal attention, making sure that there is enough supporting material for each, not too much, and not too little. A hard, but important, step is removing content that does not contribute to the main point. For example, Allan Kelly makes excellent use of sidebars to further develop material that is surplus, yet still supportive to the main point.

Revising the second draft

The second revision of the draft focuses on the text of the article; the paragraphs, sentences, and words.

Revise paragraphs – Each paragraph should express one point. They should vary in length, typically between 30 and 150 words, not all long, and not all short. Use short paragraphs for emphasis, and long paragraphs for descriptive or discursive text.

Revise flow – Ensure that the narrative of the article flows smoothly from paragraph to paragraph.

Revise sentences and words. – Use a variety of sentence constructions. Vary sentence length; short sentences for impact, long sentences for descriptions. Rely on your inner ear to find sentences that don’t scan well. Rework any that sound awkward, imprecise or wordy. Spike Milligan said, “Clichés are the handrail of the mind”. Substitute them for fresh, interesting descriptions. Prefer the active voice to the passive voice.

Revise for tone – Everyone writes in their own voice, but articles written for Overload should be a mixture of instructive, confident, explorative, and friendly; you should avoid preachy, arrogant, fancy, or overly academic and formal tones.

Revise the introduction – Check that it introduces the topic of the article. The introduction can also be used to grab the reader’s attention, possibly by asking a question, or making a blunt statement.

Check the conclusion – The conclusion can be used to pull together all the parts of the article, it can rephrase the main argument, or it can reaffirm the importance of the topic. Above all it should convey a sense of completion.

There are three excellent books that I often refer to during this stage of the writing process. The Elements of Style [1] is a short classic that I mean to reread more often than I do. Essential English Grammar [2] is a short guide that makes up for my not having paid any attention to my English teacher. And, Bugs in Writing [3] is an instructional and beautifully presented book written specifically for engineers who write.

Revising the third draft

The final revision is for overall style, grammar, punctuation, and spelling. Read the text slowly, carefully, and aloud. Let your ear guide you. If some text doesn’t sound quite right, rework it. You can also ask someone else to read and comment on your draft. Select a reader who will provide you with constructive and affirmative feedback.

Introduce titles and sub-headings to clearly identify topics. They can also pull the reader into the article as they browse through the magazine.

Use font effects sparingly; reserve them for emphasis to clarify meaning.

Submission to Overload

Articles should be submitted to the Overload editor after the second or third revision. An editorial board, comprised of the editor and a number of readers, manages the content of the magazine. The editor distributes the article to readers, who read the article for technical correctness and relevance. The readers return their comments to the author, who revises the article once again and resubmits the final draft to the editor. The editor performs the final proof reading before passing the article on to the production editor for inclusion in the next issue.

You’d write, only…

You don’t have time – You should regard writing as an investment in yourself. You take time to educate yourself to keep your technical skills fresh and relevant. Writing is another important skill you should nurture. Try asking for some work time to write an article. Enlightened management will recognize your increased value to the organization. In order words, writing an article for Overload will get you a promotion!

You’re not so good at writing – The hardest part is getting started, but by developing a regular writing habit you will improve with practice. We’ll help you get through the writing process by discussing topics, approaches and editing drafts.

People won’t want to read what you write – To publish seems to be calling for attention, inviting others to judge you. But the Overload audience is friendly and supportive and I’ve only ever received positive comments for my writing in Overload.

Bjarne is smarter than you – Probably, but we publish articles at all levels. An issue full of language extension proposals would not be fun for any of us.

You just don’t feel good about this – Writing is much like public speaking, it takes time to gain confidence. Start with a small audience and build from there. Challenge yourself.

You mean to, but you can’t get started – Take each stage at a time. It’s hard to sit down at your desk with the goal of writing a book. A more manageable goal to set is to write a page, or even a single paragraph. As each milestone is achieved you build success upon success until you reach the greater goal.

You don’t have anything to say – Trust me, you do.

You have no choice

I hope this editorial has persuaded you that there are real benefits from writing for publication, and that writing an article for Overload or CVu is something that you can and should strive to do.

References

[1] The Elements of Style, Struck and White, Allyn and Bacon.

[2] Essential English Grammar, Gucker, Dover.

[3] Bugs in Writing, Lyn Dupré, Addision-Wesley.

I’m dreadful at testing code, as I rarely bother to do it. Yeah, yeah, I know that I should. It’s just a mental blind spot that I have.

Have you ever experienced the phenomenon of reading what you intended to write instead of what you have actually written? It’s the same situation when you have a piece of broken code that you stare and stare at but can’t see any errors. You then show it to a peer, claiming that the compiler is busted, the microprocessor is messing up, and the laws of physics no longer hold true. They look at it for ten seconds, go ‘ugh’, and point at ‘if(x=y)…’. I’ve extrapolated this mental assumption upwards from the level of syntactic correctness to the level of semantic correctness. I assume that because I intended the code to work, it must actually work. Thankfully, due to some introspection, and a few people politely, and not so politely, pointing out that it would be nice if I tested my code properly, I am aware of my short-comings in this area.

But, I’m not the only person who is bad at testing; the majority of development organizations are bad at it too. I have never discovered a development organization that conducts excellent testing. Most organizations are content to test to the ‘good enough’ level. I don’t believe the individuals involved intend to build a shoddy product, there’s just something wrong with the way most people conduct testing.

Testing has always been a hard problem that few software engineering organizations put much intellectual effort into solving. In evidence, we have the fact that most test groups spend much of their time executing test cases by hand. I’ve experienced developer groups where the prevailing attitude was that all responsibility for testing lay with the Quality Assurance group. The developers throw the release candidate over a wall; the testers kick it about for a while, and then throw it back if they find something wrong with it. The cycle then continues, seemingly endlessly, until the QA manager and the engineering manager resolve their differences over a drunken fistfight in the carpark. [1]

A common solution to poor product quality is to throw more testers at the problem. Microsoft demonstrates that this doesn’t work. They have a very high ratio of testers to developers, yet still produce ‘good enough’ quality products. Are they unwilling to produce quality products? I don’t think so.

Another problem is that the role of test engineer is not well regarded by the software engineering profession at large. Most testers I’ve worked with don’t want to be test engineers. They want to be developers. For them the test group is a stepping-stone into the sustaining engineering, or development engineering groups. I’m in favour of career progression, but when everybody wants out and nobody wants in there’s a problem.

Is the software quality problem due to testers just not being very good at testing? I don’t think so. I’ve encountered some very smart and highly productive, dedicated and motivated testers.

I’ve talked to colleagues with an Electrical Engineering background about their experience of testing in hardware development organizations. They have told me that test engineering is a well-respected role that is staffed by engineers who are just as qualified as engineers in the development organization, and that testers are involved in all phases of the development process to ensure that the product can be efficiently tested for correctness.

The point to learn here is that testability has to be designed into the system, and the engineering group bears the responsibility for doing that. Typically the development engineering team doesn’t think much further than unit testing. More upfront thought needs to be put into integration and system testing at the specification and design stages of development.

I’ve tried to reflect this lesson into my current project. I’ve thought about testing right from the beginning of the development process. There’s a line item on the product requirements that states that the product must be easy to test. The design and implementation follow through against this by exposing interfaces specifically for testing. In this case our user interface is a C++ API, so the test interface we chose was a scripting language. The scripting API includes methods that expose some of the internals as text strings, so that the test cases can make assertions about the internal state of the system.

I don’t know the solution to the software quality problem, and I’m not sure if anyone really does, but there are a few small things that we could be doing to improve matters. In summary, designing testability into the system allows us to fully leverage the QA group and therefore improve software quality.

Apologies

In Overload 50 I mistakenly published a draft version of Pete Goodliffe’s STL-style Circular Buffers article. There were a few minor errors in the text, which have been corrected in the online version, which can be found on Pete’s website. [2]

Notes

[1] That’s a slight exaggeration.

[2] http://cthree.org/pete/circularbuffer.1.pdf

Lawrence Lessig received a standing ovation for his keynote presentation at Usenix 2002 in Monterey California. Lessig is a professor of law at Stanford Law School, and a founder of the Stanford Center for Internet and Society. He is a cyberspace lawyer. [1] Usenix is a venerable and aging conference for operating systems academics and open source contributors. They are beardy Unix hackers. [2] I am beardless, and unixless, but I attended anyway as part of the yearly company beano.

Lessig has a gift for oration. His legal training and his many courtroom appearances have provided him with a highly polished presentation style. The pace of his delivery is measured, with a poetic meter that is hard to identify. The slides that accompanied his presentation were the most un-powerpoint-ey powerpoint slides I’ve ever seen, and were so numerous, and so smoothly transitioned, that they took on a filmic quality. His voice, his presence, and the visuals, combined to form a compelling platform for his message.

His talk, entitled ‘The Internet’s Coming Silent Spring’, presented the argument that the Internet is being undermined by those that are threatened by the neutral and unrestrained innovation fostered by its network architecture.

Stories of the allowed

He introduced his topic by referring to a number of historical examples.

Firstly, he described the invention of FM radio by Edwin Armstrong in 1934. AM radio was plagued with static. FM offered higher fidelity, and greater range, but FM threatened the established broadcasting industry, in the form of RCA. RCA used the Federal Communicatoins Commision (FCC) against Armstong, by harassing him with patent infringement suits, and forcing him to switch frequencies. Twenty-three years to the day after patenting FM, Armstrong put on his hat, coat, scarf, and gloves, and walked out his apartment window, falling13 floors to his death. FM radio was not allowed.

Secondly, Lessig referred to the invention of Packet Switching by Paul Baran of Rand in 1964. Packet Switching was designed as a decentralized communications network that could survive nuclear attack. But, AT&T said: It “will never possibly work” and we’ll be “damed if we’ll allow the creation of a competitor to ourselves.” The Internet was not allowed.

Thirdly, Lessig described the website that gave instructions on how to teach a Sony Aibo to dance Jazz. Sony invoked the Digital Millennium Copyright Act to have the code forcibly removed. Jazz was not allowed.

Lastly, Lessig told the tale of the Brothers Grimm and Walt Disney. The Grimm’s unpleasant stories had passed into common property by the time Walt started making animated films. He quite legally borrowed from the stories liberally. In 1790 an author of a work owned rights over that work for 14 years. In 1832 that term was increased to 42 years, in 1909 to 56, in 1962 to 59, and then extended often until 1976 when it stood at 75 years. In 1998 the Sonny Bono Copyright Term Extension Act (aka the Mickey Mouse Protection Act) increased it to 95 years. Thus, no one shall do to Disney what Disney did to the Brothers Grimm.

Societies and Architecture

Lessig went on to describe the architecture of the Internet as a simple network, with smart applications. The benefit being that network users don’t have to ask AT&T’s permission in order to exchange information. The end-to-end nature of the network architecture allows anyone to do anything with anyone. Furthermore, the number of innovators increases from one to many, and the kind of innovations change from those that benefit the network owner, to those that benefit the network users. For Example, the Internet was innovated by Vint Cerf and Robert Kahn. The World Wide Web was innovated by CERN of Switzerland. ICQ was innovated by Israelis. HotMail was innovated by Indians. What do they have in common? They were kids and non-Americans, and most importantly they were not the owners of the network. These innovations were the consequence of the architecture.

Within society policies are agreed upon, and then codified within a body of law. By analogy the policies of the Internet are its architecture, and the law of the Internet are the protocols and the code. The architectural policies are implemented within the design of the protocols, and within the actual implementations of those protocols.

The Internet is the content that is delivered, the code that implements the protocols, and the physical media it runs over; the cables and the spectrum. The code at the core of the network is being corrupted by the influence of the corporations that control the content and the physical media. The core is being corrupted whilst policy makers do nothing. They do nothing because of the framing of the debate.

Stolen Property

The debate is being framed in terms of property rights. Corporations own the property, and their property is being stolen.

For example, innovators are broadcasting cable content over the Internet. Just as the cable industry broadcasts network television content. That’s “blood sucked from our veins” say the cable bosses. It’s their property, and it’s being stolen.

Another example is the Recording Industry Association of America (RIAA). They would rather have all consumer devices modified to prevent copying, than provide a digital distribution channel. Napster is theft!

Some historical context helps to highlight the shift in policy that has occurred. At the start of the 20th century there was a thriving sheet music business. Sheet music could not be copied and resold, as it was protected by copyright. An innovator created piano rolls, which were not a copy of the music, but a new form of distribution, so therefore the rolls were not infringing the right of any copyright holder. Piano rolls ‘napsterized’ sheet music. Similarly cable napsterized broadcast television, and the VCR napsterized the film industry.

In the past the law was fitted to the technology. Today the technology must fit to the law of the past. The shift is clear from the Napster case. The judge ruled that Napster must prove that their system was 100% free of copyrighted material. By comparison the innovators of the VCR only had to show that there were legitimate legal uses for their device. Would the VCR exist if the manufacturers had to prove that no one would ever use the device to infringe copyright holders rights? No.

This policy change has occurred because corporations wish to minimize competition. They want to be in the position of dictating which new technologies should be allowed.
The corporations want policy put into the code to protect their property.

Lessig believes that policy makers must be persuaded to acknowledge this policy change. The barrier is that everyone supports the protection of property owner’s rights. Theft is after all wrong. But, reframing the debate can break down the barrier.

Is it property, and should it be owned? Should the Ford Motor company own the highways? How well would General Motors cars work on Ford Highways? The architecture of the Internet should be common property and should not be owned, as neutral platforms build innovation.

Is the property being stolen? Creativity has always built upon the past. The rights of the owner must be balanced with the rights of society. This is the precept upon which the patent system and the copyright system are based. Creativity breeds innovation.

The Internet should be free, free in the way Richard Stallman means free. We, the innovators of the Internet, should have the freedom to innovate, to tinker, to copy.

Silent Spring

Finally Lessig explained the meaning of the title for his talk. The book ‘Silent Spring’ was written by Rachel Carson in the early sixties. She single-handedly took on the chemical industry to stop the uncontrolled use of pesticides. A compelling chapter of the book describes a town where all life has been ‘silenced’ by the insidious effects of DDT. Lessig’s ominous parallel is that if the attacks upon the core of the Internet are not defended, then the Internet will be silenced.

Code and Other Laws of Cyberspace

In closing if you’d like to pursue more of the thoughts and writings of Professor Lessig you’ll find information on his website [4] and in his recent book [5], an excellent review of which was written by Mike Godwin [6].

[1] http://cyberlaw.stanford.edu/lessig/

[2] http://www.usenix.org/

[3] http://aibopet.com/

[4] http://cyberlaw.stanford.edu/lessig/

[5] ‘Code and Other Laws of Cyberspace’, Lawrence Lessig, Basic Books.

[6] http://www.oreilly.com/news/lessig_0100.html Review of ‘Code and Other Laws of Cyberspace’, by Mike Godwin.

In a recent issue of CVu, Pete Goodliffe wrote an excellent article surveying the history of popular methodologies. In this editorial I would like to chart my own personal journey through that history to discover where I now stand.

At the start of my computing experience, before any formal computing education, my methodology was to just hack everything into existence. I was programming, rather than engineering. I would assign myself a project, perhaps a simple line drawing program, and then set about coding. My methodology was to start with the part of the program that I knew least about. That way, if that piece defeated me, I would have wasted the least amount of time on the project. But, once the puzzle was solved, I would decide that the rest of the program was pretty obvious, and therefore there wasn’t much point in completing it, so I might as well just move onto the next project.

Much later, at university, I learned about SSADM [1] and various lecturer-devised methodologies that seemed very suitable for building large payroll applications. They didn’t seem to offer much for my day-to-day software engineering projects. My experience was with very small teams (a team of one), short term (midnight-6am), with a well-defined customer (me and my friends). My products were games. I spent four years porting and enhancing a variety of bulletin boards, chat systems, and multi-user dungeons. [2]

After graduation, I was straight off onto the games industry proper, at that time predominantly the vocation of small groups of underpaid youths in cramped quarters having a whale of a time making up crazy stuff all day. My chums and I were ensconced in a one room, fourth flour, lower Chelsea, studio that was reputed to have once been the office of a popular beat combo named the Rolling Stones. [3] Our methodology? Write a proposal, tart it about some publishers, code like crazy, beg for more money, code like crazy, beg for more money, etc.

I had to grow up a bit after that, and went to work on some terminal emulation software. It was all OO, so I dusted off my Booch and OMT books. OOP had always seemed pretty natural for me, so the methodologies made sense, but the process didn’t do much for me. I just went through the motions. The diagramming notations were great though; a common language for communicating ideas efficiently and effectively.

Then I joined a small research and development company that was staffing a project. They were an independent start-up that used the RAD [4] approach to produce a demonstration product. A dozen engineers spent a year building a working demo to show off to investors and customers. This was a great success, Mr Gates said he thought it was ‘very cool’ and wanted to be a customer, so the company was sold to an American corporation.

They were a largish, established, telecommunications supplier, so the system was to deliver telecom level reliability. The project was blue sky; a paradigm shift in voice messaging. It was to be the next generation system to replace two aging product lines. There was plenty of money to invest, and plenty of time to do it right. This was an ideal opportunity to try a more formal engineering approach.

The project was already staffed with three teams of seven when I joined. We proceeded to write documents for six months. Yes, six months. We wrote functional specifications for the user facing components and detailed design documents for all the systems’ components. Each document was based on a standard template to ensure that every engineer considered issues such as performance, testing, and internationalization.

As each document was completed it went to a five-person review team, each person having a distinct role in the review: author, two readers, scribe, and a moderator. The readers would raise defects against the document, which were recorded by the scribe. The moderator ensured that the correct process was followed, and that nobody got too upset. The review process would repeat until all defects had been dealt with to the satisfaction of the readers.

This was very formal, and more than some could take (me included). When no one was looking I would Alt-Tab out of Microsoft Word/Rational Rose and hack some code until I was happy my design was going to work. But this alternating between the abstract and the concrete was a valuable way to ground myself and prevent me from designing a castle in the air.

This is all very well, but did it work? Yes, and no. Yes, the software was developed as designed, and pretty much on time, without any serious unforeseen complications arising. And the software has provided a very solid foundation for the past eight years of functional enhancements. To my continued amazement my original code is still largely intact.

But the project did fail in a couple of respects. Firstly, for political expediency, management, against the advice of engineering, signed off on a poor quality design for one component. None of the engineers wanted to build the component, so a contractor was brought in. He did his best, but at the end of the project we had to junk his work and redesign and rebuild it from scratch. Secondly, the business has never been much of a success. The product still doesn’t have many customers. But, it did help the company sell itself, again, to an even bigger company.

That was a natural point for me to move on, and, having had a taste of America, I decided to move to Silicon Valley to do the Internet thing. I thought I had missed the big boom of software development, and didn’t see another one coming down the pike. I can recall thinking that $35 was far too much to register a domain name, and, in any case, that would be a pollution of the namespace. How would the inter-network be paid for now that the US government had backed out? By attaching an advert to each packet, or message, or something, we joked. Chuckle. Sigh.

I ended up at Netscape working on a server product. I had never experienced such contrasts between projects. That was where I realised the inverse relationship between code quality and profitability. I went from a four year project with a formally designed object-oriented architecture written in the most advanced C++, which made very little money, to a project with twelve month cycles with little design or expressible architecture written in the most awful C I’d ever witnessed which, of course, made more money than you could imagine. There was little regularity between anything, there was no data encapsulation, everything messed with everything else’s privates; it was an orgy of code.

Their success came from moving quickly. If it worked it shipped, it didn’t need to look pretty. We had internal customers who were very demanding and vocal. We had a very small and tightly nit team that intimately understood the problem domain. We had a program of continuous build and test cycles. We had no formal process of any kind, just gut feel and rule of thumb. If a concept was too hard to explain on a white board in a couple of minutes, then an email was needed, if that didn’t suffice then a document was needed. All team communication, within and between teams, was via mailing lists, and internal web sites. When Kent Beck’s XP book appeared the content seemed very familiar to us. [5]

Of course, as the project matured, and grew larger, it became exponentially harder to add new features to the code base. We spent increasing amounts of time rewriting swathes of code to componentise the system, and to try to cram the jinni back into the bottle.

I’ve now come full circle, and am back to the team of one, working at home, often in the evenings, on distributed semi-structured database systems. Not MUDs this time, but XML databases. Oddly enough, they have quite a lot in common.

Summary

So, what I’ve learned on this journey is to solve the hardest problems first, and that CASE tools draw nice diagrams. That writing documents works, if you have lots of time, and that having a good architecture in place provides a long-term development foundation. And, finally, that solving real problems is better than writing nice code, and that team communication is critical to project success.

New Reader

Richard Blundell has been dabbling with computers for a quarter century, but has been developing software professionally for half that time. His beginnings were in Basic and 6502/Z80 machine code on Commodore Pets and TRS-80s. He progressed through the likes of Pascal, Fortran, C and even things like PostScript, on PCs, Vaxen and Sparc systems. Today he is Principal Developer for a management consultancy company in Surrey, developing business visualisation software in C++ for the Windows platform. Some of his interests include: security/cryptography, DIY, extreme programming, gardening, algorithms, Linux.

Notes

[1] Structured Systems Analysis and Design Method, a methodology favoured by the public sector in the UK.

[2] Essex Mud, Abermud, TinyMud, LPMud, Hunt, Sun of Bullet.

[3] To my knowledge the phrase ‘popular beat combo’ was coined by a barrister in response to a judge’s query as to who ‘The Beatles’ were. ‘I believe they are a popular beat combo m’lud’.

[4] Rapid Application Development.

[5] Extreme Programming Explained, Kent Beck, Addision Wesley.

It was years before I started keeping notes. Even in my college days I rarely took notes, relying on handouts and memory instead. In my first few professional jobs I just did my day-to-day thing. I had the one project, and I always knew what state it was in. I felt no need of notes. I was foot-loose, fancy free, and unencumbered.

I read an article in 1994 espousing the wonders of keeping a work jotter. I tried it for a week or two, but like diary keeping after new years day, it was a good idea that just wouldn’t take hold.

Then I started working with a compulsive work diarist. He’d trained as a scientist, and consequently knew more about the ear canal of the common cricket then your average software engineer. Thankfully he also knew more about software engineering then your average software engineer too. He had an array of A4 books on his shelf, neatly labelled and dated. Every meeting, decision, conversation, thought, was written up, in the style of the scientific method, with a green tortoise shell fountain pen. Consequentially his knowledge of our product, its history, and who said what, where, and when was encyclopaedic.

Spurred by this inspiration I resolved to start an engineering notebook at the start of my very next major project. Two years later I bought myself the thickest A4 notebook available and set out to document my journey through a major piece of software development. I approvingly noticed that my VP of engineering carried a notebook to keep track of the many intertwined projects within our division. But, her notebook was A10. Size matters. My notebook was too big. It couldn’t be with me always. My entries petered out.

Three years later, at the inception of my next project, I went for a midsize, thin, A5 college notebook. Success! I have now been a compulsive note taker for two years. And, I’m not going back to my unenlightened self.
I find it serves multiple purposes for me, over different time frames. In the short-term, day-to-day and week-to-week, I mostly maintain to-do lists, and project status information. Interspersed with this is some commentary, and description of problems that arise, the possible directions to take, and the currently preferred solution.

In the mid-term, month-to-month, I scan back through the pages seeking threads of thought that went un-concluded. Or, for a reminder of the problems that were hastily swept under the carpet in the interests of unhindered forward progress. [The code written in the rush of pub lunch confidence.]

In the long-term, year-to-year, I get some historical perspective over how the project progressed, both of the team and of myself. Writing my end of year appraisal is no longer the soul-searching agony of: ‘what the hell have I been doing all year, all that time and only a few thousand lines of code to show for it.’ I now have a day-by-day, blow-by-blow record of what I was doing.
Reading your own forgotten words a year later can be quite enlightening. From how incredibly insightful you can be about the final form of the project, to how incredibly naïve and deluded you were on how much effort it would actually take.

Professional advancement comes from considering feedback about your performance, from your management, from your peers, and from your own introspection. There’s none as devout as the redeemed, so I fervently recommend the maintenance of an engineering notebook. Start yours today.

Apologies

Apologies to Björn Karlsson for two mistakes made in the development and presentation of his Boost article published in Overload 47. Firstly, a draft of the article was published in place of a more polished final revision, and more noticeably the header and byline of the article were omitted. Graciously Björn has accepted our apologies and has offered to write a follow up article covering one of the boost libraries in more detail. I hope that other boost library authors will follow suit in discussing their fine work within the covers of Overload.

Notes

http://www.meadweb.com – My brand of notebook. I suggest Green for work, and blue for your Overload articles ;-)

I find myself lacking a burning software engineering issue to bang on about in this editorial. So I shall subject you to a random collection of thoughts and comments until I fill the requisite space.

Computer Literacy: A chain of technical bookshops in the bay area named Computer Literacy has just closed its doors. This is a great shame as they were a great place to pick up weird and eclectic engineering textbooks. It’s doubly sad as they were kindly hosting the ACCU-USA monthly meeting, and they carried our newsletter in all their stores. [1] Their stumble came when they set up a website called Fatbrain to sell books online. [2] Their costs soured competing against the online book-selling giants and in the end they were forced to sell out to Barns and Noble to survive.

Printers Ink: Another independent purveyor of strange books and magazines has finally folded. They limped along for a couple of years after benefactors stepped in to prop them up a little longer. They finally capitulated and closed all but the Mountain View store, which Books Inc took over with a renegotiated lease, and an inventory limited to the mainstream. [5]

Liar’s Poker: Michael Lewis, an American journalist and author, wrote the book ‘Liar’s Poker’ from first hand experience of 1980’s Wall Street. It’s an amusing account of his failure to become a big-swinging-dick of the bond trading world. [1] His most recent book, ‘The New New Thing’, is a biography of Jim Clark, the Stanford Professor who went on to found Silicon Graphics, Netscape, and Healtheon. Lewis became obsessed with what drives Clark to invent, build and abandon his creations. After losing control of SGI to its financial backers Clark set out to redress the balance of power from the money men to the engineers. [2]

Economics: The economy of Silicon Valley was bad, and now it’s getting worse. New investment from both venture funds and big companies has dried up. American enterprises live and die by their quarterly results, and with every dismal earnings announcement comes another round of layoffs. The flabby overcapacity built up in the booming internet years has long since been cast aside, now they’re down to making hard choices about what’s a core business and what’s a non-essential accessory. The balance of power has flipped back from the engineers to the money managers.

Death of a dot com: In the new harsh environment of ‘reality’ my own new new thing just hit the wall. The third round money was raised, but the founders refused to sign the deal. They chose to own a big chuck of nothing instead of a small piece of something. Sigh. Anyone want to buy some intellectual property?

Interviewing: And so I’ve been out there interviewing, and it hasn’t been much fun. A year ago the modus operandi was to contemplate what you’d like to work on, find companies that were doing that, pick the market leader, email the VP of engineering, negotiate a deal, reject the counter offer from your current employer, and start next Monday morning. Now job openings are scarce, the recruitment process is excruciating, and the competition is stiff. Managers are looking for people with very specific skill sets and background experience. They want people who have done exactly the same job before. There’s not much learning opportunity in that.

Puzzles: I’ve had to relearn white board programming, and how to answer dumb logic puzzles. I find that my day job exercises neither of these skills. Ask me about using a pair of scales to identify the odd ball out from a bag of twelve using only three comparisons, and a say ‘who cares’. Ask me about how to enthuse and empower a troublesome QA department and I’ll wax lyrical about the wonders of automated testing. Do logic puzzles make people who can define, architect, design, implement, test, and deliver a product? Do crossword puzzles make writers? Nope. (Is this just me being crusty? I actually managed to answer the dumb logic puzzle correctly, but rejected the company on the grounds of giving me a lame and unimaginative interview. If you’d like to study and memorise the most commonly trotted out puzzles you’ll find them listed on the web. [3])

Homework: I’ve been working from home the past month, but I’m being troubled by demons. They’re tormenting me with all the alternative fun things I could be doing instead of work. I’m happily working away, deep in thought, and next thing I know I’m under a piece of furniture playing with the cat, or standing in front of an open refrigerator about to place a spoonful of peanut butter into my mouth.

Library: They can’t get to me at the local city library, as they have no cats or refrigerators there. (In the USA the definition of a city is somewhat less stringent than in the UK. Mountain View has a population of only 75,000 people. Back home you’d need a cathedral and a note from the Queen.) Phrases that come to mind when I think library are: large print Mills and Boon romance novels, a strange musty smell, radiators leaking rusty water, books about Ashton Tate dBase 4, VisiCalc, and WordStar. But these do not apply to Mountain View City Library, oh no, for it has: A network port for every seat! Superb air-conditioning! Conference rooms for hire! And I think they have books too. [7]

References

[1] www.accu-usa.org

[2] www.fatbrian.com

[3] ‘Liar’s Poker’, Michael Lewis.

[4] ‘The New New Thing: A Silicon Valley Story’, Michael Lewis.

[5] www.acetheinterview.com

[6] Overload 30, Editorial - Books, John Merrells.

[7] www.ci.mtnview.ca.us

The notion of the identity of an individual on the Internet is an amorphous and almost nonexistent concept. The most commonly exchanged token of identity is the email address. I’m sure you have many. Perhaps one for each of your roles in society: @work.com, @home.com, @society.org, @lastname.com, @hobby.com. Of course email addresses are easy to come by and become disused just as quickly. They provide no authentication as to the identity of the owner. But, this is a powerful thing for breaking down social barriers and for preserving anonymity.

I can make a choice of the appropriate identity for each communication I initiate. Sometimes I represent myself, or some other organisation with which I am affiliated. Within a community there may be an established level of trust for a particular domain. An email from accu.org may carry more weight for you than one from kingospam.com.

In the real physical universe the notion of identity is very real, and again I can identify myself in many ways depending upon the communication I which to initiate. A credit card for a financial transaction, a library card to borrow a book, or a passport when I enter another county. These are the credentials that I carry to authenticate myself. Each issuer of identity, in this case a bank, library, or government, have different levels of trust ascribed to them by society. Each body of trust also has trust relationships with other bodies of trust. Governments have reciprocal trust agreements between each other such that citizens may or perhaps may not travel between them. Governments license banks to operate within their boundaries of control. Banks act as trustees for the owners of assets.

In the virtual world of the Internet the identity providers are numerous. Your internet service provider issues you with an identity so that you may gain access to its network point of presence, and perhaps also to its email sending and receiving servers. You may make use of a free email service at a portal site, so that you may roam freely of your ISP. Your work may provide you with a work email address. Your bank may have provided you with online access to your bank account. Your landline and wireless telephone provider may also offer online account checking. You may also have a paypal account that allows you to pay money over the Internet. You may be running a business on the side via eBay. Perhaps you use PGP and have registered your public key in a directory. You may even have bought a client side certificate from verisign. There are so many providers of identity on the Internet. Yet how many have reciprocal trust relationships? None I can think off.

Identity, authentication, privacy, and trust are all complex issues. I don’t claim to be particularly well educated in any of these topics, but I sense that another apocalyptic battle of the Internet age looms on the horizon. Well, yes that may be a little strong, but as time goes on doesn’t the browser war start looking like the thin edge of a very big wedge?

Microsoft has launched as part of its .NET platform and XP Operating System two new technologies: Passport and Hailstorm.
Passport is a centralised directory of user identity made openly available over the Internet for any system for user authentication. Envisage a world where you just have one user name and password for every device, system, and account. That’s actually very cool, but also very scary.

Hailstorm is an extension of Passport whereby collections of attributes are stored with the user identity, again openly available for any system to retrieve once authenticated. Envisage a world where you never ever again have to type your home address into a website form. Just provide your passport identity and password and the website can fetch the attributes you have approved it to read. Again, very cool, very scary.

Why is this scary? Well, in the real world I have many identities provided by many identity providers. I can choice from them freely. What happens if devices, systems, and account providers only accept identity issued from one provider? I have many user attributes scattered over the Internet, each with different levels of sensitivity. Travelocity knows my seating preference for MD80’s (by the exit), and JCrew knows my trouser measurements (34x34). Do I really want all this information centralized?

In the real world the ultimate provider of identity are governments, who are held accountable to the society that elects them into governance. On the Internet the ultimate provider of identity could become a corporation held accountable only by its shareholders.

Why need the user directory be centralized, in order to gain the benefits of portable identity and sharable attributes? Why must the schema of the data be controlled by a corporation in order to benefit from the standardization of the syntax, semantics, and naming of information?

Of course the cynical might suggest that this might be necessary for leveraging the monopoly of a browser market into a monopoly of a server platform, and from there to service provision, and perhaps even from there to an assault on the ownership of the primary customer relationship, currently the province of banks and telephone companies. A common strategy for them attacking a market has been to adopt a standard API so as to turn the implementers of that API into a commodity. Then the API is extended to favour the favourite provider (themselves) and that implementation is offered at a very competitive price point (ie. free).

So, how might these benefits be derived without compromising control of our identities and attributes to a single corporate identity that has not proved to be particularly trustworthy in the past? After all there’s little point in complaining without proffering some solution.

The user directory in the sky could be administered by an entity that is trusted by the Internet community at large. An analogous system with which this could be compared is the Domain Name System. Admittedly this is a much simpler technical problem, yet has much of the same social issues to address. The DNS system is administered by commercial entities appointed contracts by ICANN, which is itself run by a board of directors elected by the Internet community. All the technical details are dealt with by a peer organisation named ‘The Internet Society’ (ISOC), whose sub-organisations; the IETF, IAB, IESG, and IRTF do all the real work.

Another solution would be to give up the centralized model and allow multiple issuers of identity within a federated model. The user identity would include the identity of the issuer. I might then be able to log into a website with the identity ‘yahoo.merrells’, ‘aol.L00z3R’, or ‘msft.4CB26C03-FF93-11d0-817E-0000F87557DB’. The website then authenticates my credentials with the issuer of the identifier. I can now choice how I identify myself for each communication I initiate.

Within this federated model user attributes become fractionally distributed amongst many identity issuers, and perhaps also non-issuers. Yahoo (an issuer) owns my calendar information, Travelocity (a non-issuer) owns my seating preferences. But, any service or user trying to locate a particular set of attributes will find it very hard. There would need to be some extra information that describes the distribution of the information. One solution might be for non-issuers to register the location of a set of user attributes with the issuer of the identity associated with the data. But then I might have many identities and I don’t want to have to remember which attributes are associated with which and be limited to using just one for certain things. A solution might be to allow the user to select a primary identity to which all the secondary identities refer. Now any service can navigate from any identity to any set of attributes. And, of course the user should be allowed to switch primary identities, just as I can switch banks, or citizenships.

This is just the start of a solution. Other issues to be resolved are: the standardisation of schema elements that describe the attribute sets, achieving adequate performance when chasing referrals to other directories, allowing the user to specify whether attribute sets should be stored with non-issuers at all, allowing users to ascribe access control to their attribute sets. And of course the practicability of the solution decreases as the complexity of the solution increases.

I hope this topic hasn’t been a surprise to you. I hope that the software engineering community at large is aware of these issues and is discussing the implications of what may play out here.

References

Microsoft’s own description of Hailstorm.

Historical discussion of the DNS system

A rather abstract discussion of identity on the Internet.

Open P2P: community discussions

Mono: An open source response to .NET.

The Internet Corporation for Assigned Names and Numbers.

The Internet Society.

Economics is hard to define, beyond that which economists think about. Most succinctly it might be expressed as the allocation of scarce resources, and the distribution of their product. Engineers mostly address the allocation of scarce resources. Marketing, sales, and business development specialists deal with the distribution of the product of those resources.

Engineers make economic decisions every day. There are alternative solutions to every problem, and many factors in deciding between them. Some technical: correctness, extensibility, reliability, performance, and scalability. Some economic: build or buy, sub-contract, or re-factor existing code. Typical questions from project management are: How much risk does each solution introduce to the project? Are there complementary actions that would mitigate that risk? Should we just make it work for now, and try to get it redone later? Should we build a generic solution that can be applied to variants of this problem? What other tasks are dependent upon the successful completion of this task? Which engineers could easily complete this task, and which engineers are actually available to work on it?

An engineering team enters into a contract with the product definition team to deliver a specified product, consuming a certain amount of resource. The resources consumed by a software engineering project are mostly intangible, being the time of professionals with a certain breadth and depth of skill. Tangible resources, such as a comfortable office space, machines, and tools, are less significant. In economic theory these resources are known as capital. The engineering brains are intellectual capital, and their environment is physical capital. Both of which can be bought with financial capital.

The reason that my thoughts recently have turned to economics, rather than software engineering, is that the economic climate in Silicon Valley has changed. For the past few years the scarce resource has been intellectual capital, now it has returned to being financial capital.

The technology stock market boom of the past five years made huge amounts of financial capital available incredibly cheaply. Venture capitalists, the gatekeepers of financial capital for startup companies, were fighting over each other to invest larger and larger sums. As the cost of capital tended towards zero the large infrastructure vendors extended cheap credit with lenient terms to the startups in return for stock. For example: Lucent extended $10bn in credit to telecom dotcoms. Intel took equity stakes in 450 of its customers, which at its peak was worth $10bn. Now its books don’t look quite so good. BT overextended itself to win 3G licenses and have had to oust their CEO, cancel their dividend, sell their valuable foreign holdings, and ask their shareholders to stump up five billon pounds to pay down their debt.

The collapse of the technology stock sector has snapped the cost of capital back to traditional levels. The existing venture capital funds have been crushed, and the new funds are being held on the sidelines until the debris settles.

Large companies are refocusing their businesses, taking the opportunity to clear out some of the dead wood and fat that they'd taken on in the boom.

Small companies, seeking their seed or first rounds of funding, are having some success. A couple of million dollars is still small change to the VCs, and the chance to be in at the ground floor of the next big thing is too tempting for them to miss out on.

The beginning of the downturn was generally regarded with disappointment, but also with some relief that the madness was finally over. The daily commute in and around San Francisco and San Jose has eased significantly. The proliferation of unviable businesses had soaked up valuable resources causing a scarcity of office space, and engineering and management talent. But the downturn always cuts deeper than you’d like. Some perfectly reasonable businesses are struggling through capital starvation.

Medium sized companies that were raising their capital by spinning a visionary yarn were brought up short when the game changed. Those funded in 1999 and 2000 are having a very hard time raising their 2nd or 3rd rounds. Up until December last year company officers were being pushed to spend heavily to gain first mover advantage and to grow employee headcount. They've burnt through $30m shooting for the moon and have few real customers to show for it. Asking the venture capitalists for another $20m on blind faith doesn't fly any more. The only thing that counts now is real revenue from signed, deployed, and referencable customers.

As engineers we love the invention, design, and implementation of new things, but what it takes to get those things built requires an appreciation of economics. Spare a thought for the economics of engineering. From the microeconomics of your engineering team; to the macroeconomics of who is paying for your project and why.

Project Website

The focal point for the team is our internal team website. The site is a web of content rooted at the main project page that serves as a portal into the whole project. The content comprises of documents that describe our requirements, functional specifications, architecture, design documents, project plans, current status, build instructions, and testing results. This static content is stored in a revision control system, (we are using perforce), which provides us with backup, versioning, and an interface for developers to manipulate the site contents from their desk or home machines. There’s also some dynamic content in the form of queries into the bug database, and reports from the continuous build and test machine.

We never email office documents as attachments, we always forward an http URL. This prevents anyone from having to read out of date documents, and ensures that everything worth writing into a document is under revision control, and is visible to anyone accessing the web pages, and that the documents actually live within a contextual web of information.

Team Mailing List

Within the team we use many means of communication: email, telephone, video conferencing, instant messaging, ad-hoc whiteboard discussions, and formal meetings for communicating. But, the most frequently used medium is the team mailing list. We use this list for collaborative problem solving at all stages of the project, and for tight co-ordination of our individual activities. From a basis of effective and efficient communication we have built a team that is self-organising and self-directed. We have a team lead rather than an engineering manager.

Most of the team members, although at a new company, have been working together for a number of years. For two of us this is our third company together, and for five of us this is our second. This familiarity with each other, and the trust that we have built with each other, makes open and honest communication much easier.
It might seem strange to be explaining the use of a mailing list, but our team has developed a way of working that may seem odd at first sight. There is a lot of traffic on the list, perhaps two hundred messages a day, and some of it is terse, and not always relevant to everyone on the list. But, this is all intentional, resulting from our need to work around a number of organisational problems.

Firstly, people tend to work the hours that suit them best. At Netscape the QA team tended to work late afternoon until midnight. We needed a way of communicating information between teams across time when not all participants were present. In the morning the engineers would pick up their test results messages, and would have a fresh build ready by the end of the day.

Secondly, as the team expanded we started having more people who worked from home some days, people who worked out of state, and who even worked in different time zones. Again the only way of communicating efficiently across time and space was email.
Both these factors contribute to the team becoming a virtual team, which may only physically congregate at the same time and place as infrequently as once every couple of months.

An empirical study I once read suggested that a significant proportion of an engineer’s day is spent communicating information to other team members. Where a virtual team breaks down is if the majority of that communication is taking place as face-to-face chatting between cubes. In a packed cube farm office you overhear an awful lot of other people's conversations, sometimes you tune in, or join in, or tune out, and sometimes you tell them in no uncertain terms to go to the other end of the building. But, the point here is that it's important to recreate the same effect on the mailing list. The virtual team members must have the opportunity to overhear a conversation between other people, and for their own conversations to be overheard by others too.
Two researchers, Heath and Luff, named this behaviour “talking-out-loud” and “overhearing”. In an ethnographic study they performed in 1991 they found that workers in the London Underground control rooms would often seemingly talk to themselves as they worked. This allowed their colleagues to have peripheral awareness of their actions, who would sometimes take supportive, collaborative action to aid them.

Talking-out-loud with email is easy, just cc the mailing list. Some of our messages are addressed to an individual, or group of individuals, or just to the list at large. There are specific topics of discussion, cry’s for help, and moaning about difficulties. People have no obligation to respond, but they become attuned to the activities of everyone around them, even though they are not physically present.

Taken to an extreme, you might see a conversation someone is having with themselves. Nobody pitches in with a comment, and they resolve the problem themselves and follow-up their own message for completeness and to let others know that the issue is closed.
The feeling of virtual presence is enhanced by the use of Instant Messaging clients, like Yahoo Messenger. Beyond enabling speedy peer-to-peer interactive communication, you also get a list of your team members currently logged into the network. You get a warm fuzzy team membership feeling that there's help and advice there if you need it.

All this results in a very chatty mailing list. Single sentence messages being exchanged between two parties and echoed to the list. Instant Messenger services don't have the useful properties of automatically archiving the conversations, or allowing others to listen in to what's going on. So, we chat by email instead.
The volume of email messages can be hard to deal with. But, it also encourages people to close issues, and not to revive them. An issue can never be left dangling; otherwise the email just keeps on backing up. The issue must be closed out with a decision, a bug, a work item, a document, whatever ever it takes to end the thread. Then the thread can be deleted or archived. Archiving decisions is important as it prevents wasted time when an issue resurfaces from another direction. Archived messages can also form the basis of design documents, bug reports, and FAQ web pages. The only messages in my inbox are those that I have read and need to take some action on, or am expecting someone else to take an action upon. So, at any one time I have about fifty messages to deal with, and I pay more attention to some topics than others, as we each have our specialized areas within the team.

Another aid in dealing with the constant message flow is that all messages must be written clearly and effectively. The author must take into account the knowledge of the listener, and make the effort to write briefly and unambiguously, so that the reader can discern exactly the point being made without a game of twenty questions. The number of issues in a message should be low, and the subject line of the message should be kept pertinent to the content, even if this means changing the subject of the message when replying.

And, finally, as in all things, good team communication requires intellectually honesty, openness, and a good sense of humour. There are always extra points for jokes!

PS: Many of the thoughts here are echoed by an article that appeared in the January 2000 issue of Communications of the ACM by Mike Robinson, Mikko Kovalainen and Esa Auramäki. It’s title, ‘Diary as Dialogue in Papermill Process Control’, may not seem particularly relevant. But, it describes how a team of paper mill workers make use of an electronic diary as a means of inter and intra team communications.

I've been playing about with Java, on and off, since it appeared in the 'HotJava' browser about five years ago. The language was simple and elegant with built in multi-threading, the virtual machine provided cross platform support, the environment supported dynamic loading of classes, and the libraries included powerful networking capabilities. The Java enabled internet browser held the promise of becoming a great platform for building network aware client applications.

Flash-forward to the present day and it seems that Java has not taken hold of the client side as expected, but of the server side. Application servers do the hard work of managing the operating system resources efficiently, leaving the application programmers to concentrate on solving the business problem at hand.

This has been my situation for the past four months, and it's been oddly frustrating. Mostly, I would have to admit, from me assuming too much about these technologies and not taking the time to study the details more carefully.

This editorial is a write up of my experiences so far. I'll structure them into the two broad categories.

Language Whining

Multiple Inheritance: I thought I'd miss multiple inheritance a lot, but it turns out that I mostly used this in C++ for defining interfaces anyway. I got burnt by the Symantec compiler many years ago, and learnt my lesson about relying on recently introduced features. That particular compiler could create some very strange v-tables. What I miss is being able to provide default implementations for those interface methods. I'm forced to write more code than I know I need to. Mix-in classes are a great way of factoring out common implementation.

Generics: I didn't realise quite how essential templates, and the STL are. The Java collection classes hark back to the Smalltalk libraries, where every object has to have a common base class - Object. I'd forgotten those early C++ days when the result of every 'get' call on a collection had to be cast to its real type, and sometimes you'd get it wrong, and very bad things would happen. At least Java does a type check for you, throwing an exception, but you don't get to find out about the problem until your server trips gaily through that piece of code. I'm having to relearn the old ways, the pre-template ways of programming in C++.

Destructors: I used to use destructors a lot, not just to clean up member variables, but as a callback point for when a static variable goes out of scope. It's so handy to wrap a dynamic object with a static handle object. You just can't forget to release the resource you've acquired. It just goes away when you reach the end of the scope. Finalizers in Java don't work the same way, they don't get called at the end of the scope, they get called when the garbage collector gets around to it, and from what I read, that may never happen. This would be alright if all resources had the same characteristics as dynamic memory objects, but they don't, some resources are far more scarce. For example: references to singletons, database transactions, file and network activity, operating system handles, timers, and synchronization primitives.

This last example is worth considering more closely. A typical technique for implementing threads and synchronization in C++ is to wrap the native system primitives in classes. You inherit from a thread class to create a threaded class, and you create a static object on the stack to have a lock held for the extent of the current scope. The Java implementation of threads works just the same way, but mutual exclusion does not. The synchronize keyword was introduced instead. True, it provides additional semantics, but I do wonder if it didn't come about initially because of the lack of destructors.

So, I have to remember to explicitly release everything I acquire. I can't add a keyword to have my transaction committed, or my timer stopped. I have to remember to balance my open's and commit's, and my start's and stop's.

Abstraction: The Java programming system provides a high level of abstraction from the underlying machine. In part this makes simple Java programs very quick to implement and get right the first time. But, hard programs are just as hard to write in Java as they are in any other programming system. In fact I seem to find it harder. Threading, synchronization, and memory management are all there, but there's a man behind the curtain not doing quite what you'd like.

Layers of abstraction are essential for solving complex problems, but without the intimate knowledge of what lies beneath it's very hard to reason about the performance characteristics of a system. An argument against my case here is that I may just be an old curmudgeon stuck in my own time when programmers started out hacking about on their Z80 and 6502 based home computers. Perhaps fifteen years ago the editor of Fortran-Fanciers-Monthly was bemoaning these new fangled C programmers who didn't know how to implement floating point arithmetic in assembler, or how to add new instructions to the processor by extending the microcode.

Strictness: The language provides positive reinforcement of good object oriented programming practices. That's great for a teaching language, but when you know what you're doing, and you want to bend the rules slightly, well, you just can't.

Environment Moaning

Virtual Machine - Performance: I don't think that Java programs run slowly, in fact it's quite impressive how fast they run, considering all the low level indirection that is being performed. I'm thinking about server code here. GUI apps still seem quite sluggish, but I'm told this is mostly architectural and implementation problems with those libraries. In theory Java code should run just as swiftly as C++ code. There's enough information there to translate it down to equivalent machine code. In fact, the latest version of the java virtual machine from Sun includes 'HotSpot' technology that translates commonly executed bytecode into native machine code at runtime. Very cool, but (and here's my moan) the development tools seem unable to deal with these complexities. Half my tools break every time I upgrade my virtual machine.

Virtual Machine - Cross Platform: The point of the virtual machine is to allow a single piece of compiled object code to run on many different platforms. This solves the cross platform problems found with lower level languages like C and C++. This is clearly a big win for client software developers, but not much gain for server implementers. Perhaps this creates a level playing field for server hardware vendors, but in my experience there are so many versions, variations, and ports of virtual machine, that you have little choice but to run on Solaris. All the porting problems have been transposed from the application domain to the configuration of the runtime environment. Getting the right mix of virtual machines for your tools and deployment platforms can be quite a challenge.

Memory: Where does it all go? It could be that Java objects have more associated meta-data than C++ objects, or that old unused objects are hanging about waiting for the garbage collector to zap them. The strict reference counting built into the language means that it's impossible to leak memory, but it is possible to leak references. And, it's very easy to end up referencing far more memory then you ever thought your server would ever need. Debugging these memory problems can be very painful. I have found no API that will retrieve the current reference count of an object. I'd like to make assertions about reference counts so that I can catch these problems early on.

Garbage Collection: This is a debate often visited. My view is that garbage collection is great if it works. However, different programs need different memory management strategies to achieve efficient use of resources to meet the performance requirements. Java was initially designed for short-lived client applications, an ideal environment for a garbage collector. Do I want my 24x7 server to hunker down every half hour to bloat up and page like crazy as it walks over all allocated memory? No thanks. This is how the garbage collector in the client side virtual machine appears to behave. Thankfully the server side virtual machine has an improved garbage collection algorithm that is well behaved for server applications.

CLASSPATH: Agh, this just drives me nuts, don't get me started.

Of course I realise I may be talking complete rot here, and it's just my Java-newbie-ness showing through, in which case I'd gladly entertain your repost to all this whining and moaning.

Patents

A quick update to follow on from my comments in Overload 40: In December the members of the European Patent Convention met to vote on whether software patents would be admissible. They decided to not admit software patents for the time being, but will reconvene in one year's time.

New Authors

I want to point out that two of the authors of this issue are first time contributors to Overload. Oliver Schoenborn, and Antony Pinchbeck have presented a pair of interesting and well written articles. I hope this serves as inspiration for those thinking of writing up an idea, idiom, pattern, or tale of the software professional.

There were 161,000 patent applications filed with the United States Patent Office last year. Thankfully only a few thousand were software patents, or were submitted by viewers of Mr. Springer’s televisual program.

The first recorded patent was granted in 1421 to an Italian architect who invented a novel, yet easily duplicated, lifting barge. From these origins patent systems were instituted in many European states in the 17th, 18th, and 19th centuries. Their purpose is to grant a temporary monopoly to inventors in return for releasing their inventions into the public domain. The inventor prospers from their skill, and society benefits through derivative and complementary innovations. The term of the monopoly is usually a not so temporary sixteen to twenty years.

The original system was devised to protect engineering designs, but was easily extended to cover chemical engineering, pharmaceutical development, and industrial processes. In the early 80’s a number of high profile cases opened up the US patent system to the information sciences. Biotechnology in 1980, software in 1981, and in 1998 the much vilified business-method patents were permitted. The most infamous being Amazon’s One-Click-Shopping ‘invention’, for which Jeff Bezos, Amazon CEO, received considerable personal flak.

As the majority of readers of this magazine are residents of the United Kingdom this topic may seem a little remote for an editorial. There is a Europe wide patent convention that does not allow software or business-method patents. However, at a conference later this year it is expected that the convention will be extended to cover software patents, and undoubtedly business-methods will follow shortly.

A rough poll of my colleagues suggests that the engineering community regard software patents as a bad thing. Rather than fostering innovation, they stifle it. Software development tends to involve the creative combination of already existing ideas. Twenty years ago the software patent land-grab began, staking out huge tracts of the software engineering landscape. He who files first wins! The patent gatekeepers, the examiners, are hindered by limited databases of prior art in the field, by the high turnover rate of their staff, and confusion around the novelty and innovation in the software industry.

The more severe black mark against software patents is an economic one. They actually serve as a tax on software development. A tax that is levied whether you file or don’t file. If you file, a patent attorney, who knows the secret language, will charge a healthy fee to file the description, diagram, and flow chart. Also, your senior engineers will spend time articulating the nature of the invention. If you don’t file, there are fees to defend against aggressor patent holders claiming infringement. And, if the case is lost then there are licensing fees to those patent holders. A firm of aggressive lawyers could buy up some patents and go fishing for infringement cases. Sadly this has actually happened.

A company may use software patents, as above, for defense against aggressive patent holders. Cross-licensing deals are played out like a hand of wist. Each company plays their patents off against each other attempting to cancel each other out by countering with equivalent infringement claims. This is classic cold war game theory – always try to have one more than the other guy, or agree to keep things even.

Beyond defense patents can be used for building the capital of a company. Industrial age firms have their plant and buildings as capital from which their market value is partly derived. Information age firms have some rented space, some office furniture, and some rapidly depreciating desktop computers. As a manifestation of intellectual capital information companies have something more tangible to show off to the investors.

The final use of patents is for generation of revenue. IBM, for example, generated $1.5bn in licensing fees last year, 20% of its profits.

Companies are increasingly finding that the old barriers to market entries are being eroded: Capital? Cheap. Labour? Mobile. First-mover advantage? Transient. Brand? Ephemeral. One of the few defensive measures remaining to them is to assert their strength through the patent system.

I think that software patents are an evil thing, but they exist, and they’re not going to go away. It can be argued that a patent portfolio is a manifestation of the intellectual capital of a company. Yet the real value is the fact that there are employees talented enough to invent new technology, and will do so again. Software patents have no technical value, just business value. As a pragmatist, I’m not going to whine about it, I’ll just try to make the most of them. I’d be very interested to hear your comments on this issue.

PS: One piece of irony is that penalties for infringement whilst knowing about the existence of some intellectual property are more severe than for accidental infringement. Ignorance is an excuse, and any patent layer you meet will encourage you to remain this way.

PSS: As penance Jeff Bezos is now funding a prior art database to combat frivolous patenting.

References

The IBM patent database

My favorite business journal

Britannica Encyclopedia

Electronic Freedom Frontier

ACM

Tim O’Rielly takes on Jeff Bezos and Jay Walker.

Campaign against software patents in Europe.

Month One: Depression

The company I have joined is both small and young; one hundred people and one year old. They have a rather dull business problem to solve; from which they hope to reap buckets of money. But while problem may be mundane, the ultimate solution is not.

They have succeeded in a small way by following the usual software development methodology: Analyse the business knowledge, map it into a model, and implement the code. Well, to be honest, they skipped the first and second bits and just wrote the code, but that's only where the trouble begins. As each new requirement of the system arrives we iterate through the whole process again. This is what you'd expect for major software releases with a year between revisions, but in our case the deployment of each and every customer introduces new requirements into the system, which we hope to be a weekly event. The implemented solution meets neither the scalability nor flexability requirements of the business.

This failure stems from the short sightedness of the original system designers in not focusing on the fundamentals and on their reliance upon 'tools'.

One component of the extreme methodology that I disagree with is 'build for today, not for tomorrow'. The majority of my work has been on large server systems that had to be built today for tomorrow and serve as a platform for the future. In some cases designing functionality two versions ahead, and three years into the future. To me architectural vision is of key importance in any software project.

My approach to software engineering has always been from first principles. Step 1: Identify and document the problem. This is often hard, as the people closest to the problem tend to only know a small piece if it well. This is reminiscent of the five blindfolded men asked to identify an object in a room. One man believes it is a snake, another believes it to be a tree. I forget what the other three were up to, but the thing is actually an elephant. Step 2: Devise a solution from first principles. Ignore implementation; just solve the problem. Step 3: Search for appropriate off the shelf systems, components, tools and technologies that could form an implementation. Step 4: Buy what can be bought, and build what can't. Step 5: Integrate.

I have rarely come across so many people able only to think in terms of tools and technologies. Given a problem they'll cast about for any vaguely relevant tool hoping it'll serve as a solution. They'll then combine it with some other randomly selected component. If that doesn't work out they'll blame the tool, and decide that the next version of the tool will actually solve the problem. Or, that some other brand of tool within the same category of tools is in fact a better solution. Never questioning whether the category of tool is even appropriate. Actually studying the problem from a fundamental viewpoint and developing a language and model of the problem domain seem to be alien concepts to these engineers. [Conceptual Modelling]

My Depression: The body of a startup company is a host for dysfunctional behaviour.

Month Two: Excitment

Having determined the sanity bisector, I settled down to think about the problem from scratch.

The fundamental problem is that the soft code of the system has been hard coded. The system needs to be able to adjust itself dynamically to a changing environment at runtime. Czarnecki and Eisenecker describe this type of system as being adaptive in their superb book 'Generative Programming'. [GP]

Having identified the real problem, I set about to document the problem, and to define the requirements of a solution. I then cast about in the software engineering literature for examples of similar problems and solutions. For this kind of exploration I find the digital libraries of the ACM and the IEEE to be invaluable.

Unfortunately I can't say too much about the details of the problem or solution, as the lawyer guy in the office next door will have a fit. I can however discuss the topics I've been investigating recently, which I hope might be of interest to some of you.

XMI: Rational Inc. and the Object Management Group (OMG) are working on a standard interchange format for modelling tools. XMI, the XML Metadata Interchange, is an XML encoding of a UML model. This has the potential to allow the seamless sharing of models between development tools. [XMI]

XML Schema: The XML specification includes 'document type declarations' (DTDs) that provide a grammar for describing the contents of a class of document. Commerce One Inc. and Microsoft Inc. have developed more advanced schema languages, SOX and XDR respectively. The W3C accepted these as proposals for standardization, but elected to develop their own representation, named 'XML Schema'.

XML for UML: Rational Inc. and Commerce One Inc. have developed a specification suggesting how UML may be used to design XML Schemas. [UML-XML]

OCL: The Object Constraint Language is a component of the UML used for writing model constraints. It can be used for writing invariants on classes, and for specifying the pre and post conditions of operations. It has come out of the work of the formal methods community, so has a very strong formal mathematic background. I'm not aware of any mainstream modelling tools that support the use of OCL, but hopefully there will be support soon. [OCL]

MOF: The Meta Object Facility is an OMG standard that describes the UML metamodel. A metamodel is a model of a model. In the case of UML it's a model of UML, and of course that model is itself written using UML. The OCL was in fact added to the UML specifically for writing the UML metamodel UML model. [No the room is not spinning. At the first sitting this specification made me feel quite ill. We should leave the discussion of the meta-metamodel for another time.]

CWM: The Common Warehouse Metamodel is a specification for modelling metadata for relational, non-relational, multidimensional systems, and most other objects found in a data-warehousing environment. In addition, CWM models enable users to trace the lineage of data as CWM provides objects that describe where the data came from and when and how the data was created. Essentially, the CWM is a specification for a repository of UML models.

Until this month all of these efforts were totally unknown to me. It's easy to get yourself into the comfortable position of thinking that you've pretty much seen a bit of everything in software engineering. My team leader gave me some advice during an exit interview some years ago. 'You're bright, but never stop learning. If you get comfortable, you're doomed.'

If you're feeling comfortable, then you're probably not an Overload subscriber. But, if your mind is parked in front of the television with a beer in one fist and the remote in the other: then I highly recommend 'Generative Programming'. It opens up a whole new horizon of thought.

My excitement: I am no longer a mere-programmer. I am a meta-programmer.

Kevlin

I picked up a copy of Java Report the other day in Printers Inc., ordered a coffee, flicked through the magazine, and there was Kevlin Henney staring out of page 92. It made me feel so proud: our Kevlin, international superstar of the object world! [Printers Inc]

Over the years Kevlin has made a huge contribution to Overload, and I hope his work between these covers has contributed to his success.

The subtext of these paragraphs is that each of you should stop for a moment and think about your contribution to Overload, and what Overload could do for you. Without Kevlin stepping in at the last moment with an article previously printed in C++ Report this issue would not have been completed.

Each of us has a worthy software tale to tell, perhaps not with the eloquence of Mr. Henny, but the editorial board are experienced at helping authors polish up their texts for publication.

Software is just another medium for knowledge. [ACM Article] If you're a programmer, then you're a writer.