8 Reflection on Theory

In chapter 2, we said that the Software Engineering Method and Theory (SEMAT) initiative called for “a process to re-found software engineering based on a solid theory, proven principles and best practices that:

  1. Include a kernel of widely-agreed elements, extensible for specific uses
  2. Addresses both technology and people issues
  3. Are supported by industry, academia, researchers and users
  4. Support extension in the face of changing requirements and technology.”

Our work on Essence has provided a kernel, that is able to represent any software engineering method, which we will demonstrate in later parts of this book. Essence is gaining traction by both industry and academia. But what about theory? What about the theoretical foundation behind software engineering?

In the paper “Where’s the Theory for Software Engineering?” by Pontus Johnson, Mathias Ekstedt and Ivar Jacobson [8], these important questions are discussed and with the permission of the authors, we publish in the next section that entire paper.

8.1 Where’s the Theory for Software Engineering?

Most academic disciplines are very concerned with their theories. Standard textbooks in subjects ranging from optics to circuit theory to psychology to organizational theory to international relations either present one single theory as the subject’s core or discuss a limited set of alternative theories to better explain the discipline’s essence to its students. A prime example is the central role of Maxwell’s equations in the subject of electrical engineering. It’s difficult to fathom what electrical engineering would be today without those four concise equations. A quite different example is the contested Domino theory, which heavily influenced American foreign policy in the 1950s to 1980s by speculating that one nation’s embrace of communism would entail the conversion of surrounding countries in a domino effect. Even though electrical engineering and political science are different in almost all respects, they’re both highly interested and invested in their theories.

8.1.1 What is Software Engineering Theory?

Software engineering, however, isn’t concerned with the topic of theory—and if asked, the community surely couldn’t give a coherent answer about which is the most important one. Candidates might include theories with significant scope, such as formal systems theory, decision theory, organization theory, or theory of cognition. Collections of propositions might also be suggested, such as Alan Davis’s 201 Principles of Software Development (McGraw-Hill, 1995), Frederick P. Brooks’s propositions in The Mythical Man-Month (Addison-Wesley, 1975), or SWEBOK (A.Abran et al., eds., IEEE, 2004). Specialized models such as Cocomo might also be candidates. We suspect that you’ll disagree with most of these proposals, but that just proves our point about the lack of consensus. Still, why are so many other fields explicit with their theories while software engineering is not?

Before discussing this question, we need to—in impossibly few words—describe what we mean by that multifaceted word “theory.” The best definition comes from a thoughtful article published in Management Information Systems Quarterly (S. Gregor, “The Nature of Theory in Information Systems,” vol. 30, no 3, 2006). According to author Shirley Gregor, there are many definitions for the term, but most theories share three characteristics: they attempt to generalize local observations and data into more abstract and universal knowledge; they generally have an interest in causality (cause and effect); and they often aim to explain or predict a phenomenon. Considering the purpose of theory, Gregor proposes four goals. The first is to simply describe the studied phenomenon; SWEBOK could serve as an example. The second goal is to explain the how, why, and when of the topic; theory of cognition, for example, is aimed at explaining the human mind’s limitations. The third goal is to not only explain what has already happened but also to predict what will happen next; in software engineering, Cocomo attempts to predict the cost of software projects. The fourth goal of theory according to Gregor is to prescribe how to act based on predictions; Alan Davis’s 201 principles exemplify this goal.

8.1.2 Three Arguments

Returning to the main question—why the software engineering community seems so uninterested in discussing its theories—we can imagine three arguments: software engineering doesn’t need theory, software engineering already has all the theory it needs, and software engineering can’t have any significant, defining theories. We don’t believe that these arguments are valid, but let’s consider them individually. Engineering Doesn’t Need Theory

Software engineering is doing fine without explicit theories, so why change a winning formula? First, software engineering isn’t doing fine. Numerous reports about failed IT projects have surfaced for decades now. Second, all engineering fields need theory. To build something good, you must understand the how, why, and when of building materials and structures. Indeed, you have to predict in the design stage the qualities of the end product if you want to avoid the painstaking labor of trial and error. In the words of Kurt Lewin, “There is nothing so practical as a good theory.” (Field Theory in Social Science, Harper & Row, 1951). Third, for the many software engineering researchers employed at universities around the world, a researcher without a theory is like a gardener without a garden. According to philosopher Thomas Kuhn, the maturity of scientific disciplines can be measured by the unity of their theories (The Structure & Scientific Revolution, Univ. of Chicago Press, 1962). In the most established disciplines engaged in what Kuhn calls normal science, a paradigmatic theory defines a whole field (for example, Maxwell’s equations, Einstein’s theory of relativity, and Darwin’s theory of natural selection). In a less mature phase, called pre-paradigm, a small number of theories, typically with ambitious explanatory scopes, compete for academic hegemony. This is the case in psychology, where cognitive theories challenge psychodynamic theories, and in international relations, where realist and liberalist theories battle for dominance. Kuhn doesn’t offer a name for the phase before the pre-paradigm, in which there exists a large number of unrelated theories, because he considers this something less than science. Software Engineering Already Has Its Theory

A discipline’s significant theories should be able to provide answers to that discipline’s significant questions. Considering software engineering, one of the most hotly debated questions concerns the choice of software development method. Although there are many opinions on the subject, we can name very few theories that attempt to answer the question. And to the extent that such theories exist, they aren’t, as in other disciplines, given names, presented in textbooks, or debated at conferences. The same goes for other significant questions of software engineering, such as which programming language to use, how to specify system requirements, and so on. Note that many proposed software development methods, programming languages, and requirements specification languages exist, but very few theories explain why or predict that one method or language would be preferable to another, under given conditions. Software Engineering Can’t Have a Theory

Software engineering is a practical engineering discipline without scientific ambitions where rules of thumb and guidelines assume the role of theory. We can counter this argument by reiterating the tight connection between engineering and science. A typical definition of engineering is the one found in Encyclopedia Britannica: “the application of science to […] the uses of humankind.” Thus, no engineering without science. Second, it isn’t true that there is no theory in the software engineering community. In a sense, theory is abundant. To the previously mentioned propositions, we could add Kent Beck’s suggestion that the change cost curve could be logarithmic rather than exponential (Extreme Programming Explained, Addison-Wesley, 1999), Parnas’s principle of information hiding (On the Criteria to Be Used in Decomposing Systems into Modules), Conway’s law, Dijkstra’s theory of cognitive limits as presented in the classical article “Go to statement considered harmful” (Comm. ACM, 1968), stepwise refinement, and so on. But all of these theories are small and most are casual, proposed by the author but rarely subjected to extended studies, and they explain only a limited set of phenomena. Furthermore, most of these theories aren’t subject to serious academic discussion; they aren’t evaluated or compared with respect to traditional criteria of theoretical quality such as consistency, correctness, comprehensiveness, and preciseness.

As should be evident by now, we don’t believe that there’s any rational reason for the lack of theoretical focus in software engineering. It’s surely historical; born in the hurly burly of software practice, explanation and prediction were often merely glanced through the car window in the race between problem and solution. Today, tens of thousands of software engineering researchers are employed in the universities of the world, spending innumerable man-hours on software engineering research, but theory is still on the sidelines. To our knowledge, very few explicit attempts propose general theories of software engineering.

This has to change: without the predictive and prescriptive support of theory, software engineering would be relegated to the horribly costly design process of trial and error. With theory, we rise from the drudgery of random action into the sphere of intentional design. Software engineering is already full of implicit theory. We just need to bring it out into the open and subject it to the serious scientific treatment it deserves.

Here we end the quote of the paper “Where’s the Theory for Software Engineering?”.

8.2 Uses of theory

Theory is generally used to (i) describe a phenomenon of interest, and (ii) to explain and predict that phenomenon. Prediction is of importance for engineering, as prediction allows engineers to build artifacts with predictable features and properties. Without prediction, engineering becomes no more efficient than a random walk. Prediction and explanation, however, require the capability to describe the salient features of the phenomenon in question. Therefore, description precedes prediction.

To describe something, a language is needed. In the case of mechanics, that language includes primitives such as mass, force and acceleration. Such a language can then be used to express causal relationships; e.g. “if the force increases, then so will the acceleration”, or “if the mass increases, then the acceleration decreases”.

8.3 The Essence is a general, descriptive theory of software engineering

Unfortunately, as we have explained above, there is currently no widely accepted predictive general theory of software engineering. However, the Essence takes the first step by proposing a coherent, general, descriptive theory of software engineering(i.e. a language of software engineering). The concepts included in the Essence were designed to capture the most important features of the various phenomena of software engineering.

As a descriptive theory, the Essence can be used to describe and facilitate discussion of future predictive theories of software engineering. However, the Essence is not strictly limited to description; it does make a number of general claims regarding the relationships between concepts. For instance, the states of alphas have a certain order, excluding the movement between non-neighboring states. But a complete consideration of the causality between concepts and thus prediction is beyond the current version of the Essence.

8.4 Towards a general predictive theory of software engineering

As mentioned above, a general (predictive) theory of software engineering ought to be able to provide answers to the discipline’s significant questions. Considering software engineering, there are a number of such questions, such as which software development method to deploy under what circumstances. There are several questions of similar importance, and there are numerous sub-questions to each overarching one. Three more concrete examples of questions that a general theory of software engineering should be able to answer include (i) what is the explanation of Brookes Law? (ii) are domain-specific languages here to stay? and (iii) is continuous integration worth the effort? The first one is explanatory, and the remaining two are predictive.

There is currently no widely accepted general theory, but there are a few contenders. Here, we outline one such, called the Tarpit Theory. This theory is described in greater detail in [9]; here we can provide only a brief sketch.

The theory’s name stems from the communicative tragedy that, according to the theory, constitutes the core of the software engineering problem. On the one hand, people have a strong desire to explain to the evermore ubiquitous computers what we would like them to do. Computers, on the other hand, are obediently awaiting our instructions, and once received, those instructions will be performed with remarkable loyalty and zeal. But despite the eagerness of the two parties, the relationship is a frustrated one, because it has proved strangely difficult to establish efficient means of communication between people and computers. Currently, almost twenty million software developers are employed worldwide, with the sole task to explain to the computers what it is that we really want from them. In the proposed theory's interpretation, Alan Perlis's famous epigram "Beware of the Turing tar-pit in which everything is possible but nothing of interest is easy" refers precisely to the communicative intractability that separates the wondrous capabilities of the Turing machine from the many desires of humankind.

The theory is grounded in four sub-fields:

  1. the theory of software, on the structure and meaning of computer programs,
  2. the theory of design, on the process of decision-making,
  3. the theory of cognition, on the limitations of human (developers’) minds,
  4. the theory of organization, on the possibilities and limits of human cooperation.

In its most skeletal form, the Tarpit theory can be reduced to four propositions:

  1. The goal of software development is to create programs that, when executed by a computer, result in behavior that is of utility to some stakeholder.
  2. The two first challenges for developers are to (a) make appropriate design decisions, and to (b) translate between languages.
  3. These challenges appear because the human mind suffers from limited cognitive capabilities.
  4. The cognitive limitations of a single developer can be partially mitigated by adding more team members, but this generates a third challenge, namely that of team coordination.

The authors claim that the theory is able to answer many of the most pertinent questions in software engineering, including the three questions above, which are discussed further in [9].

The Tarpit and the Essence integrate nicely. Many concepts are identical or easily translated, such as endeavor, stakeholder and team. Other concepts constitute the extension of each theory with the concepts of the other. Jointly, the integrated theory gains the descriptive and practical power of the Essence and the predictive power of the Tarpit.

8.5 A Theoretical Foundation Helps You Grow

As a software engineering student, you may first learn the mechanics of software engineering, e.g. writing code, writing tests, planning a project. But as you grow in this software professional, you will need to gradually develop your understanding and interpretation of this discipline. You will try to make sense of this world. In other words, you start to form your own theory. As mentioned above, the Essence kernel provides a universe of discourse. It gives you a language to describe phenomena thus facilitating the discussion of future predictive theories of software engineering. We hope that as you read the subsequent pages of the book, you reflect on the observed phenomenon. If you are already a software engineering professional, we encourage you to compare described phenomenon, in Smith’s story, with what you experience in the real world, and determine the value of the kernel in presenting the facts. As a researcher, we certainly encourage that you would use the Essence kernel to frame your research hypothesis, findings, and conclusions. Regardless of who you are, the Essence kernel is a spring board towards more mature software engineering practices and a more mature software engineering discipline.


In this part of the book, we have introduced Essence, its motivation, its purpose, its principles, its language and its kernel. In the remaining parts of the book, we will demonstrate how Essence helps you and your teams collaborate more effectively.

Many of us know how to develop software, but we have different ideas about how to go about doing it. Some of us know more than others and can figure it out; some need more guidance, or just enough to get started. We make the distinction between explicit, knowledge and tacit (implicit) knowledge. Explicit knowledge is structured and expressed clearly, whereas tacit knowledge is unstructured and not written down – you just keep it in your head. It is knowledge emerging from experience and discussions. Software development needs a combination of explicit and tacit knowledge. It is not possible to be fully explicit and prescribe everything a team needs to do, because every development is different and involves creative problem solving, which is by nature emerging as more experience is obtained. Thus, we need to strike a balance between the use of explicit and tacit knowledge.

We take you on a journey in the rest of this book describing more and more advanced use cases of Essence.


Recommended additional reading

This book is about the essentials of software engineering through the lens of Essence, the standard. There is of course more to be learned than what is presented here. In the following you will find references that at the time of writing this book are well suited as additional reading. At the web site that accompanies the book you will find more references related to software engineering in general and also updates to the following references as such become available.


  1. Object Management Group, Essence - Kernel And Language For Software Engineering Methods (Essence), December 2015. (http://www.omg.org/spec/Essence/1.1). This is the source for the Essence standard, where you may want to go to find more details and a formal definition of the Essence Language. In this book we present only a subset of the standard, selected to be useful in everyday life. The standard is not set in stone; it undergoes continued improvements and it will usually be updated yearly.
  2. Ivar Jacobson, Pan-Wei Ng, Paul E. McMahon, Ian Spence and Svante Lidman, “The Essence of Software Engineering: Applying the SEMAT Kernel”, Addison-Wesley, 2013. This is the first book on Essence written primarily for software development professionals. It provides the underlying ideas behind the Essence kernel, an overview of the kernel and examples of how teams can use the kernel to run a software endeavor. This reference should be used when the reader is looking for details related to the ideas behind the kernel and more details in how the kernel can be used by teams.
  3. Ivar Jacobson, Pan-Wei Ng, Paul E. McMahon, Ian Spence and Svante Lidman, “The Essence of Software Engineering: The SEMAT Kernel,” Communications of the ACM, Volume 55, Issue 12, December 2012. (http://queue.acm.org/detail.cfm?id=2389616). This paper discusses the background, motivation and overview of the Essence kernel. This reference should be used when the reader is looking for a short synopsis of Essence.
  4. Ivar Jacobson, Ian Spence and Pan-Wei Ng. “Agile and SEMAT: Perfect Partners”, Communications of the ACM, Volume 11, Issue 9, Oct. 2013 (http://queue.acm.org/detail.cfm?id=2541674). This paper discusses how the Agile ideas and Essence are related. Essence is a tool to work with methods in an agile way. This reference should be used by readers that are using an Agile approach and want to understand more about how Essence can help their endeavor.
  5. Ng, Pan-Wei. "Theory based software engineering with the SEMAT kernel: preliminary investigation and experiences." Proceedings of the 3rd SEMAT Workshop on General Theories of Software Engineering. ACM, 2014.
  6. Ng, Pan-Wei. "Integrating software engineering theory and practice using essence: A case study." Science of Computer Programming 101 (2015): 66-78.
  7. McMahon, Paul E., “A Thinking Framework to Power Software Development Team Performance”, Crosstalk, The Journal of Defense Software Engineering, Jan/Feb 2015. http://www.crosstalkonline.org/. This paper discusses specific features of Essence that can help teams improve performance in ways previous frameworks have fallen short. This reference should be used by teams that are using other frameworks and are looking for ways to improve their performance.
  8. Pontus Johnson, Mathias Ekstedt, Ivar Jacobson. “Where’s the Theory for Software Engineering?”, IEEE Software, Volume 29, Issue 5, Sep 2012.
  9. Pontus Johnson, and Mathias Ekstedt. "The Tarpit–A general theory of software engineering." Information and Software Technology 70 (2016): 181-203. https://www.researchgate.net/profile/Pontus_Johnson/publication/278743539_The_Tarpit_-_A_General_Theory_of_Software_Engineering/links /55b4490008aed621de0114f5/The-Tarpit-A-General-Theory-of-Software-Engineering.pdf. This paper describes the Tarpit general theory of software engineering.