Lessons From Other Industries

The software industries need to follow in the footsteps of other more mature industries and move towards product development models that are highly flexible, where critical knowledge is embedded in highly automated processes, and where the switching costs involved in moving to different suppliers of commodity inputs are minimized.

A modern, industrial production line leverages domain-specific tools and processes designed to produce a range of similar products. In contrast, the software industry still depends to a large degree on individual craftsmanship and generic tools. Where a modern production line would have difficulties producing anything else than what it was specifically designed for, the traditional software developer's toolkit is used to develop any type of software, be it embedded software for real-time applications or distributed enterprise-wide information systems. The evolution from individual craftsmen to industrial, specialized production lines render unprecedented levels of scale, economy, quality and predictability - qualities the software industry still struggles with.

In "traditional" industries, models, prototypes and crash test dummies are widely used to study different aspects of a new product before, during and after production. Numerous tests verify expected capabilities and safety limits; what is the maximum load, minimum operating temperature, or average fuel consumption? There are well defined standards that a product must meet, or technical specifications that must be listed; radio interference, power consumption, power output, signal-to-noise ratio and distortion are a few examples from the audio domain. Comparing the technical specifications of an electronic component with its software counterpart leaves an ocean of unanswered questions. Software standards are primarily about interoperability, declaring the "physical" runtime environment (hardware and operating system platform), and what you can expect the software to do. However, we lack standards and conventions for specifying runtime characteristics the way we do with electronic components. Consequently, most software products list features and platform requirements (what can I do with it, where can I run it?), but precious few specify reliable and comparable performance specifications (max concurrent users, latency, throughput, fault tolerance, …). Except for products in mature domains (like transaction monitors or RDBMS), few software components list technical specifications in a uniform way that let consumers judge and compare qualities. This makes it very hard to effectively source (or outsource) components for a software product line in a way that a more mature industry can. As an example, compare the technical specifications of some kind of audio device (like the Ipod) to the way software components are presented at major component repositories like Apache or Eclipse.

Furthermore, mature industries study and learn from problems and disasters. In mission-critical systems we have monitors and "black boxes" that record system performance and operator actions. Interesting and/or critical failures are thoroughly analyzed, and lessons learned are published for the benefit of peers and followers - in some cases, this is mandated by law. In software, there is precious little published; we tend to hide our problems rather than learn from them, and doing a meaningful post-mortem analysis is becoming very hard with ever-increasing system complexity. In mature engineering disciplines, technology evolves by studying existing designs, both the successful and the unsuccessful. In software, this is very hard, since architecture is rarely documented in a comprehensible way. It is generally hard to understand why some solutions worked and other didn't, and we have a hard time learning from history the way mature disciplines and industries do. Grady Booch is pioneering what appears to become the first ever handbook of software architecture, a long overdue counterpart to traditional engineering and architectural handbooks.

Mature industries have worked with components and patterns at least since Christopher Polhem's "mechanical alphabet". This is an area where the software industry did pick up some good ideas; inspired by Christopher Alexander's architectural patterns, the "Gang Of Four" (GOF) Design Patterns book was a milestone, and design patterns are now mainstream in the software industry. We can now enjoy the ease and speed of communication and designs similar to mature industries. Patterns are now "everywhere" (Hillside, Portland Pattern Repository, …). However, although patterns give us a language and a way to design robust solutions quickly, we still lack ready-to-use components; we can talk about things, and decide which pattern to apply, but we cannot readily go out and get a suitable component the way the electronic or mechanical industry can. We tend to build these components ourselves. The reason is not necessarily that there are no components, but that the components are either hard to find, hard to specify, hard to evaluate, or hard to plug into the existing architecture - or all of the above. We don't have to look far for inspiration of how to do it - our "siblings" in computer hardware and our "cousins" in the electronic industry have some interesting success stories to tell about components, standards, design patterns, and a lean production line with significant flexibility that responds well to constant evolution and ever-changing requirements.

However, there are some significant differences between hardware and software components. Where a hardware component is fairly hard to modify after being built, a software component can be updated and patched over the internet, even automatically. If you've shipped a million RS-232 connectors over the world, you don't easily modify the pin configuration if you discover a mistake, or if the standard evolves. With a software component it is quite possible to handle this. Similarly, if you build and ship thousands of computers with PS/2 mouse/keyboard connector, and the market changes direction towards USB ports, the machines already shipped with the PS/2 connectors are not easily modified to handle USB devices - you depend on external, after-market adapters or add-on components. With a software component, the story is different; you may be able to update the component over the net, or you may be able to update the specifications and re-generate the interface altogether. Hence, the lesson to be learned from the hardware industry is not necessarily about the solutions. We share the desire for reusable components, but although the problem is the same, the solutions may be different - it is easier to re-generate an adapter for a software component than to reconfigure a hardware port. We are not necessarily bound to the interface specs "forever", the way the hardware industry is.

In more recent time (after "the Component era" in the mid and late nineties), the software industry has started looking into principles for lean manufacturing, a concept made famous by Toyota (also known as the Toyota Production System). One of the most influential books in this area is Mary and Tom Poppendieck's book Lean Software Development: An Agile Toolkit. Like Toyota, we strive to avoid waste - a concept that rhymes well with the Agile Manifesto. This includes a number of principles, where perhaps the most important one is not do things that don't contribute to production, and the runner-up being to do things the most effective and efficient way possible - the latter principle including a strive to minimize tedious workarounds and recovery from sloppy groundwork and poor quality of upstream components. We can learn a lot from Toyota's attention to component quality and frenzy to figure out why things go wrong, focusing on total cost of production rather than cutting corners trying to sub-optimize individual production steps. Ken Schwaber, a co-founder of the Scrum methodology, argues strongly for maintaining software quality much like Toyota: As professionals we should not accept business requests to sacrifice quality in order to meet timelines, and if quality does need to be sacrificed such a decision should be made by executive management and reflected in the financial statements of the company. Not compromising on quality is not only your professional obligation but it is also important for your own joy of work and is critical for the company.

Pursuing the comparison with the automotive industry, which today has a significant ingredient of software and computer technology, the computer hardware and software industry is looking into qualities that have been core values in transportation for a long time. The computer hardware industry has a long history of measuring power, speed, uptime and MTBF (Mean Time Between Failure), and has more recently begun looking into matters like energy performance and environmental aspects. The software industry is also looking into similar matters, but where embedded, real-time systems often have sophisticated systems for tracking, identifying and analyzing problems, errors in traditional information systems are often hard to fix. Although the SLA (Service Level Agreement) is commonly used, it has modest or no impact on information systems design. Log messages and error dialogs are - as it seems - inherently hard to understand, and second or third line support consequently spend expensive time finding critical problems at customer sites. Where mature industries (including computer hardware) consider repair- and maintenance cost a core value and invest heavily in sophisticated diagnostic systems, a traditional information system is almost never designed or built with repair- and maintenance cost as a prime objective. The lesson being learned from traditional industries is that dependability and uptime is not "just" a question of preventing errors, but also about fixing them swiftly when they occur.

work in progress

Next Chapter » Software technology fashion parade

Add a New Comment
Unless otherwise stated, the content of this page is licensed under Creative Commons Attribution-ShareAlike 3.0 License