20 What it Means to Scale

The term scale can mean different things in different situations.  In this chapter, we clarify what we mean by “scaling” and how the kernel approach helps scaling in different situations. The kernel contains the essential elements of software engineering, but does not contain everything that a team needs to know or to do in specific situations.

20.1 The Journey Continued

Before we proceed further, let’s dive a little deeper into Smith’s development organization. Having gone this far into the book, you should now be familiar with Smith’s development team. Let’s now see how that team fits within their larger development context. Figure 149 shows Smith’s development organization chart.

Figure 149     TravelEssence’s Organization Chart

Right at the top of the IT organization is Cheryl the CIO (Chief Information Officer). Cheryl reported to the CEO (Chief Executive Officer) and was responsible for digitalization efforts within TravelEssence. Assisting her is Lim, the development vice president (VP), who is responsible for developing IT systems, and Tan, the operations VP, who is responsible for running the IT systems in their data centers. These IT systems included the Recommendation Engine which Smith’s team is responsible for. Jones is the development program manager responsible for the legacy Core Hotel Management System (CHMS).

A program is a larger endeavor, which usually comprises several smaller, still potentially large related endeavors. Different teams may work on different endeavors within the program. Sometimes, the program also refers to the teams working on the program.   This program comprises several teams, one of which is led by Jane. In such a large organization, with many systems being developed and operated in parallel, many challenges arise. Software engineering is not just programming; it involves analysis and design, collaboration between teams and stakeholders, etc. In Figure 149, we provided some names for individuals that will appear in our story in subsequent chapters.

Before we continue, we want to stress that large-scale complex development is not just about having more people. The software system itself usually has more complex and more stringent requirements on for example performance, reliability, security, compliance. For example, during peak season, the CHMS need to handle many user requests at the same time. Users do not like having anything broken, lost or tampered. Just imagine if someone were to hacked into the CHMS and steal user data. Indeed, we do hear of such things happening in the news. However, how to satisfy such architecture qualities of performance, reliability, security, etc. are advanced topics which we will not cover in this book. But suffice to say, there can practices (built upon Essence) to address these qualities. As an example, the microservice practice is an example of a technical practice to enable teams to evolve software systems faster and handle performance (i.e. handling more requests at the same time) and reliability (i.e. reducing downtime) requirements.

20.2 The Three Dimensions of Scaling

Looking at the large development organization exemplified by Figure 149, you should be able to easily recognize several major challenges:

  1. Members will often have diverse background and experiences.  They may or may not possess the needed competencies.  Some may need more explicit guidance than others to do their job, and to collaborate with other team members.
  2. With large-scale development, coordination of the work between members becomes a challenge. This was particularly the case for Jones’s program. Smith’s team did not experience this issue because his team was small. But when team size grows beyond about 7 people as was the case for Jones, coordination and communication often starts to break down.  One of the greatest challenges faced with large-scale development endeavors is how to go about ensuring that members discuss, share and agree on their plans, progress and the results of their work.
  3. Different kinds of software endeavors have different risks and constraints, and therefore face different challenges. For example, Smith’s team and Jones Core Hotel Management System will definitely have different challenges. Smith’s team’s major challenge was how to deliver quickly, whereas Jones’ challenge was about maintaining quality while functionality was being added to the system.

The above are three distinct challenges to scaling. Accordingly, when considering what it means to scale, we need to address three corresponding dimensions of scaling (see Figure 150), all of which may happen at the same time in specific situations.

Figure 150     Important dimensions of scaling.

Zooming In – We call the first dimension “zooming in” which is about providing guidance beyond what the kernel provides. This is necessary in several cases. First, inexperienced members need more guidance. Second, for larger teams with members having different backgrounds, they need further details in order to ensure that they have a common understanding of how to conduct development. This additional guidance comes in the form of what we refer to as practices, which are extensions to the kernel. Essence provides a way to describe and assemble practices together in a systematic manner. For team members it is a practical way to understand how to apply practices, and how different practices work together. Essence allows teams to zoom in through essentialization, which is about making practices explicit and also making them available through a practice architecture. We will elaborate these two concepts in chapter 21. In the previous part, we have given quite a number of examples of how Smith’s team received explicit guidance through practices like Scrum, User Stories, Use Cases and Microservices. By making the alphas, alpha states, work products, activities and patterns explicit, Smith’s team received clear explicit guidance on how to proceed and continually improve. Chapter 21 will continue this story, but taking a larger view in the context of TravelEssence’s organization chart.

Scaling Up – We call the second dimension “scaling up”.  Scaling up is about expanding the use of the kernel from a team with a small number of members and low complexity to endeavors involving large numbers of people and systems with high complexity. “Scaling up” happens in large development. This is the case with Jones’ Core Hotel Management System (CHMS) development program. Jones had to manage his development teams and also interact with other development teams, such as Smith’s team. This kind of large-scale development is known as enterprise systems development. There are other kinds of large-scale development such as building product lines (e.g. a smartphone series with various screen sizes and configurations) and large-scale systems and software engineering (e.g. building an aircraft). These are all extremely challenging development scenarios and are beyond the scope of this book. The Essence kernel can be extended with practices for such complex development. What we focus on is providing a glimpse into large-scale development, and how explicit practices can help, by diving into Jones’ development program in chapter 22 and comparing it with Smith’s development team (which we saw earlier in part 3).

Reaching Out – The third dimension is “reaching out”, which is about extending the kernel with appropriate practices to meet the specific needs of different kinds of development. Lim’s development organization, not only developed the hotel management system, but also other IT systems; some small, some large. Small development includes applications, such as automating spreadsheets to produce management reports. Large development includes applications, such as the hotel management system, human resource systems, and finance systems. Some of their development was conducted in-house while other development was outsourced to another company. Outsourcing refers to contracting work to an external organization. Cheryl was responsible for leading the digitalization efforts, which included applying facial and emotion recognition to authenticate users and capturing user feedback, and the results of data analytics, and artificial intelligence. These required new technologies involving much research and exploration as well as collaboration with expert organizations (i.e. organizations and service providers who specialized in these technologies).  Each development endeavor had its own set of challenges, and often required its own set of practices.

Reaching out is about selecting appropriate practices and composing an appropriate method to address the risks and challenges for a particular software endeavor.  A simple approach is to empower the team to build their own method. For example, in part 3, we saw how Smith’s team selected practices to address their challenges. Once it has been shown that a particular set of practices is useful for a particular kind of endeavor, organizations may choose to make known pre-composed practices to their teams. Thus, instead of composing their own methods, they can choose from pre-composed methods. For example, Smith’s development organization had evolved a number of such pre-composed methods e.g. for small development, large enterprise enhancements, exploratory development. Each pre-composed method comprises mandatory practices on top of the kernel to jump start teams through explicit guidance. These practices can be about collaboration, or architecture and technical aspects of development. Teams are still permitted to add additional practices as needed and they are encouraged to improve their practices as they continually learn through their development effort. Reaching out to different kinds of development is not just about making practices available, but also training and coaching teams about those practices and transforming them into a learning organization.  This is the focus of chapter 24 in this part of the book.

In these more complex cases, it is important to provide tools and mechanisms for teams to self-manage.  Self-management means taking responsibility for one’s own behavior.  Self-management has been shown to be an essential management approach for successful large-scale development. The kernel, with its alphas and structured approach to explicit practices provides the tools for team members to design the practices needed. Examples of such practices include the way teams collaborate with other teams (referred to as Team of Teams in chapter 22), and the way teams ensure their work is aligned with other teams (referred to as Periodic Alignment in chapter 22).  Traditional approaches to scaling often rely on strengthening the command hierarchy (e.g. creating more managers that direct team members rather than encouraging increased self-management) and adding more checks, which can lead to bottlenecks, and unnecessary conflicts. Modern approaches emphasize self-management, self-organization, and even self-governance. Essence supports whatever approach your case requires. This is elaborated further in chapter 4.