22 Scaling Up to Large and Complex Development

When we here talk about scaling to large-scale development, we are talking about development involving 30 people or more. The number 30 is not a magic number. Some consider 50 or more to be large-scale, some even 100 or more.

In large-scale development, the team members are often distributed across geographical locations. They may be in the same building, but we frequently encounter large development being spread across different cities and even different time zones. These development teams will likely have multiple requirement sources coming from different customers at any point in time.  They may also be serving different opportunities from different stakeholder groups. Stakeholder groups might be related or independent of one another. Moreover, each team might have different skill sets and competency levels.

One of the authors have applied Essence practices to a large development endeavor with several hundred people. The trick is about reducing and managing dependencies between across subsystems and teams. If this is not dealt with, bottlenecks appear quickly and collaboration becomes difficult.

22.1.1 Large Scale Methods

There are many approaches to large-scale development. In the past, computers were indeed very expensive, and software development endeavors took a long time. There wasn’t any environment for rapid prototyping. The Capability Maturity Model Integration (CMMI) helps teams improve processes and the Rational Unified Process (RUP) is an iterative software development method. Both were used mostly for large-scale development.  In today’s modern world, computing is becoming cheaper, and technology is not so much the bottleneck when compared with human interactions. Thus, we see the rise of the already mentioned large-scale agile development methods like DAD, LeSS, SAFe and so on. It is outside the scope of this book to introduce these methods.

Lim, the development VP, organized his practices into layers according to their scope of influence (see Figure 152).

  • The team layer comprises practices useful for teams to work effectively together to deliver great quality software. You have seen a great example of how a team can work effectively through the story of Smith’s team discussed in part 2 and part 3.
  • The program layer comprises practices to help coordinate the work across teams and across departments or IT systems. So, for example, this might involve the collaboration between Smith, Jones and Jane to evolve the Core Hotel Management System (CHMS).        

Figure 152     Practice Architecture Organized into Layers

Note that all the practices above are similar to those we showed earlier in chapter 21, Figure 151. The practice architecture above had been customized by Lim and his colleagues.

Before continuing our discussion, we would like to highlight that the team layer practices in Figure 152 are a decomposition of Scrum. Scrum is what we term as a composite practice (see Figure 153). In part 3 we presented Scrum Lite as a single practice but here we want to show how Scrum can be composed. It is in fact a composition of product ownership (what to develop), backlog management (managing the work), team sprint (working iteratively), and retrospectives (continual development), all at the team level. 


Figure 153     Scrum as a Composite Practice

The upper layer, the program practices, takes the same notions, but applies them at a larger scale. At the program layer, Jones was not working directly with individual team members, but through representatives such as Jane. Other than that, the concepts at the program layer are very similar to that at the team level, except that they operate at a larger granularity, with a longer planning horizon.

The practice architecture was organized to show how each practice contributes to helping the organization work effectively at each level as seen in Figure 151. Even though organizations have different problems at different organizational levels, they can reuse the same principles to solve them.

In a similar way, the lines linking practices in show a simple relationship to indicate that these practices should be applied together to achieve synergy. For example, take the example of Team Sprint, and Program Sprint. They both shared the same notion of sprints, (i.e. to work within periodic cycles). This is about having teams working in a cyclic manner with clear and periodic schedules for when to conduct planning, reviews and retrospectives. This periodic alignment worked well for Smith’s team and Jane’s development team, who both worked on weekly sprints. In particular, it helped coordinate dependencies between them, such as when Smith’s use case slices depended on Jane’s use case slices, Jones the development manager for the Core Hotel Management System (CHMS) was responsible for several development teams working collaboratively to evolve and enhance the CHMS. The periodic cycles at the program level were longer due to cross-team dependencies. Jones, and his colleagues concluded that a monthly cycle would work well for his program. Lim, the development VP, would review the progress of CHMS at a quarterly cycle to determine if budgets should be extended or changed.  These practices all apply the same principle of repeating certain practices iteratively.   It is natural therefore that their cycles should somehow match one another. For example, it makes sense for team sprints to fit within program sprints. This also means that these practices are best applied together to give the greatest benefit.

The same applies for each column in Figure 152, which you will see in the next section. However, to keep within the scope of this book, we will restrict our discussion to the program and team layers.

22.2 Large Scale Development

TravelEssence was in fact operating on a large-scale manner. In part 3, we saw how Smith’s team applied several practices to deliver effectively. As mentioned, Smith’s team was part of a larger development organization. In particular, Smith’s recommendation engine (as we have seen in part 2 and part 3) drew data from TravelEssence’s Core Hotel Management System (CHMS), discussed in the previous section.  

Recall that CHMS is an existing legacy system that provides critical functionality for many customers within the hotel management business area.  The users of this software were constantly requesting enhancements and it was Jones’ primary responsibility to assess those requests and manage the agreed upon CHMS enhancements.  Because of the number of enhancements that were constantly being approved the work required multiple teams to be making changes to the CHMS core product at the same time.  This obviously added a level of complexity to the management of the work.   Jane’s team was one of those teams and we shall use her team to explain how the multiple development teams worked together as part of Jones’ overall program.  Because of the need for multiple software development teams to be working in parallel some additional practices were needed to help ensure the work was coordinated properly. Figure 152 depicts some of the practices that Jones’ and Jane’ team used.  These practices are discussed further below.   

Let’s start by discussing the two layers of practices identified in Figure 152, referred to as Jones’ Program layer and Jane’s Team layer. The columns in each layer highlight practices that are similar, but applied to a different target audience.  Each column represents a distinct concern of software development, including agreeing on what to develop, managing work (through backlogs), working iteratively, and continual improvement (through retrospectives). It should be noted that these four concerns are perspectives that Lim came up with, and not general concerns from the Essence perspective.  We will elaborate on each column, the practices therein, and how Jones and Jane benefit from the practices they applied to help them in their large-scale development challenge.

Large-scale development practices are not simple practices.  They span beyond the realm of the small team that we encountered in part 3.  It is not an easy task to replicate what happens in the real world of large-scale development in a classroom setting. It is also not easy for new students who work mostly alone to visualize how large teams operate. Hence, for the rest of this chapter, we will intersperse the explanation of the above practices with the TravelEssence story. This will help you appreciate how large-scale teams actually collaborate, and the dynamics involved.

22.3 Kick Starting Large Scale Development

Running large-scale development is not just about selecting and applying practices as we have seen above. There are also steps to kick-start the process, similar to that for small-scale development. We have explained in part 3, chapter 14, the steps to kick start development for small-scale development. The steps for large-scale development are the same, except that we have more practices (as discussed above). The steps are:

  1. Understanding the context through the lens of Essence
  2. Agree on development scope and checkpoints (where it begins and where it ends)
  3. Agree on practices to apply
  4. Agree on the important things to watch

We will review each step in the following sections.

22.3.1 Understanding the context through the lens of Essence

Understanding who is involved in the development and the current state of the endeavor can easily be found by walking through each kernel alpha and assessing its state, along with providing rationale for each assessment.  Refer to Table 15. In the interest of simplicity, we just look at the alpha states achieved for Jones’ Program team. 

Table 15        Development Context through the lens of Essence


State achieved by Jones’ program

Rationale for achieving the state



In part 3 when our TravelEssence team was smaller the team identified just Cheryl, Dave, and Angela as key stakeholders that needed to be involved. But now that the endeavour has grown Jones has to be concerned about involving many more stakeholders including the representatives of the users who have enhancement requests for CHMS. At the start of the endeavor. Jones knows he has not yet received the full commitment and involvement of all of the needed stakeholder representatives.


Value Established

The value of the system has been established because the users of the CHMS are already gaining great value, which is why they keep asking for more enhancements, and the successful demo of the Recommendation Engine will just increase the value more.



Jones recognizes that many of the user groups have been making requests for enhancements beyond the original scope of CHMS.  

Software System

Architecture Selected

The legacy CHMS system has an existing architecture and the Mobile App, through the use of microservices has been proven to support the needs of the recommendation engine.



Work is underway as parts of Jones’ teams are already successfully working on enhancements to the CHMS system

Way of Working

Foundation Established

Jones realizes that the collaboration between  teams is sometimes more important than collaboration within teams.  Teams will need to understand what other teams are doing and give the necessary support. They will need help to do this.



Part of Jones’ team is already working on enhancements to the CHMS systems


22.3.2 Agree on development scope and checkpoints

The scope of development context can be defined easily by agreeing what should be achieved before the beginning of each cycle (or sprint), so that the team or program can start effectively, and by agreeing what the team or program should achieve by the end of each cycle. Again, these can be defined easily through alpha states (see Table 16).

Table 16        Development Scope for Jones’ Program


Before sprint starts

Before sprint ends



Satisfied for deployment


Value Established





Software System

Architecture Selected





Way of Working

Principles Established

Working Well





The criteria for the sprint to start served as a gentle reminder regarding the health of Jones’ program, In particular, the team had been lacking Stakeholder involvement. They also lacked the understanding of the value of the Opportunity in preceding sprints (Compare Table 15 and Table 16). Jones highlighted these challenges to Cheryl and the business departments. It wasn’t easy, but they knew that these issues had dragged for too long, and had to be fixed. To solve this issue, some roles were assigned. Jones played the scrum master at the program level while Jane accepted the role as team level scrum master. Working alongside them respectively, are Samantha and Seet, who played the role of a product manager and team product owner. This more explicit agreement of roles overcame the chaotic nature of previous interactions. Now, it became clearer to the busy stakeholders who should attend which meetings. As a result, stakeholder involvement improved.

Figure 154     Participants in the Jones’ Program

22.3.3 Agree on practices to apply

By having a clearer indication of roles, expectations became clearer and Jones’ program was able to reach the Stakeholders Involved state. But this was just the beginning.  The newly appointed Samantha and Seet still needed some explicit guidance regarding how to do their job. This was where practices became helpful.

The third step is about agreeing on which practice to apply at both the program level and the team level. We have discussed this in the preceding sections. Rather, we summarize the practices that Samantha, Jones, Seet, and Jane applied (see Table 17). The first column highlights the purpose of the practice selected, which is namely: Agreeing what to develop, managing the work, running iteratively, and continual improvement.  The practices on the next two columns have already been presented earlier. The last column indicates the kind of guidance that practices provide. For example, both the Product Management, and Product Ownership practices provide guidance on how to achieve Opportunity and Requirements alpha states, at the program and team level respectively.

Table 17        Practices Selected


Program Level Practices

Team Level Practices

Guidance to kernel alpha progression

Agreeing what to develop

Product Management


Product Ownership


Opportunity, Requirements

Managing the work

Program Backlog


Team Backlog


Requirements, Work

Running Iteratively

Program Sprint

Team Sprint



Continual Improvement

Program Retrospective

Team Retrospective

Way of Working


You might wonder why there are just four rows, one for each purpose. Is that all we need? This question can be easily answered by walking through the kernel alphas. For example, agreeing what to develop is guided by Opportunity and Requirements.  Similarly, managing the work is guided by Requirements and Work. 

From Table 17, it is clear that there are no explicit practices to address challenges with Stakeholders, Software System, and Team. Of course, we can add practices to deal with these alphas, but in the TravelEssence situation they do not present significant problems and challenges.  Thus, in this case we deal with these practices without explicit guidance in the form of a practice description; we deal with these practices with our tacit knowledge.  However, in other organizations, these might be major challenges and in that case defining practices to monitor and progress Stakeholders, Software System and Team may be appropriate.

Note that beyond the practices listed in Table 17, development teams also apply more technical practices like Use Cases and Microservices.  We do not discuss these practices here because we have done so in part 3. In this part, we place more emphasis on collaboration between teams and team members and other challenges that large scale complex development needs to address.

22.3.4 Agree on the important things to watch

This fourth and final step to kick start development is to agree on the important things to watch. The point of this Table 18 is that on each endeavor, regardless of size, there are alphas from the kernel and alphas from the practice level that teams agree need to be watched to assess the progress of the endeavor accurately.

Table 18        Things to watch




Alphas from the Kernel

Opportunity, Stakeholders, Requirements, Software System, Work


Software System, Work

Alphas from the practices

Program Backlog Item

Team Backlog Item, Program Backlog Item


It is without doubt that running a large-scale development is not easy. There are just so many things happening, and it is important for everyone to focus, and filter out the noise.  Everyone needs to do her/his job well and all in all make sure that nothing is missed. Jones would have his responsibilities and so would Jane. Figure 155 shows the things that Jones and Jane would be watchful of. It is without doubt that there would be some overlap, albeit from different perspectives. Jones, had to look at the bigger picture focusing on the Opportunity, whereas Jane had to focus on progressing the Team Backlog Items making sure the work her team was doing (Team Backlog Items) stayed consistent with overall program requirements and the agreed to program work (Program Backlog Items).  They both had to be concerned with the Software System, Requirements, Work and Program Backlog Items.

Figure 155     Things to watch in a particularly large-scale endeavor

Running Large Scale Development

To help understand what it means to run a large-scale development, we organize the story around the columns in Table 17 beginning with agreeing what to develop, managing the work, working iteratively and continual improvement.

22.3.5 Agreeing What to Develop

The first set of practices is about building the right product, and getting the teams to focus on a common vision, whilst being able to respond to customer change requests. This is about product ownership and product management.

We have seen in part 3, that Angela was responsible for setting the priority of development. She was playing the role of a Product Owner. Her team was small and the scope of development equally small. Hence, a single person playing the role of a product owner sufficed.

However, this is not the case for Jones’ program. Recall that Jones had many requirements sources, and many enhancement requests coming from the many customers of TravelEssence within the hotel management business. In this case, due to the increased complexity of his requirements sources, he needed to make sure that team members are aligned towards a common vision at the program level. This is achieved through a product vision work product which clarified the value of what they are developing, and the way requirements were to be prioritized. Jones also established a new product manager role that was filled by Samantha. The responsibilities included evolving the product vision and achieve acceptance at the program level.   Second, he established a team product owner role led by Seet, which included the corresponding responsibilities at the team level (see Figure 156.)

Figure 156     Product Ownership and Product Management

With the new responsibilities, Samantha and Seet, along with other product owners started to systematically agree what to develop, and they started to Evolve Product Vision, both at the program and team level.

They all agreed that the overall theme should be about providing great customer experience over a robust software system. At the program level, Samantha and Jones updated the vision. For great customer and user experience, this would be achieved as follows.

They agreed to reducing the number of user actions to complete a use case.  Some CHMS functionality required users to traverse through complicated form filling steps. Because of that CHMS lost some customers. Thus, an important theme for the next release was to explore options for reducing the number of steps in every customer facing use cases.

They also agreed to provide a waiting list functionality to deal with situations when hotel rooms are fully booked. This provides a way for TravelEssence to keep in contact with potential customers even when the rooms were full. In this way, they could follow-up and notify customers when rooms were available.

For robust Software System, this would be achieved as follows.

They agreed to split the monolithic database.  Over several years of operation, the CHMS database had grown large and difficult to maintain, and slower. It was time to streamline the database. Jones suggested that it would be useful to separate the database into smaller parts, and archive historical or unused data out of the database.

Seet and Jane were responsible for that part of the CHMS that had to do with reservations. There were other parts of the CHMS that dealt with customer check in and check out, customer relationship, sales, etc. which we will not discuss in detail because they are not within the scope of this part of the book.

Seet and Jane’s team updated their vision to include:

  1. Reduce the number of steps needed to complete a reservation, through the use of default data or past user data entries.
  2. Provide waiting functionality as required by program level.
  3. Move old and unused reservation records out of the existing database to improve database performance.

22.3.6 Managing the Work

With a good understanding of what needs to be done, the next step is to work towards this goal effectively and efficiently. The teams agree how they would achieve the vision by identifying pieces of work they need to achieve, (i.e. product backlog items and prioritize them). They made sure that these product backlog items were work open and transparent to everyone, so that teams can understand each other’s priorities and workload. This also helped in managing dependencies across teams. For this reason, Jones made sure that his teams had a shared and transparent backlog as depicted at the program level in Figure 157.  When scaling to large and complex endeavors individual smaller teams within the overall program often have their own more detailed product backlogs.  However, these more detailed backlogs need to align properly and be coordinated with the overall program backlog.  This is an example why we need additional practices when software endeavors scale up.  In this case, a practice to help align and coordinate multiple detailed team backlogs with the overall program backlog may be needed. This program backlog management practice had a Maintain Backlog periodic activity which the program manager conducted and attended by all of the team product owners with the purpose of ensuring each individual team backlog has the proper priority given the overall program backlog priority. 

Figure 157     Program and Team Backlog Management

Based on the agreed vision at both the program and at the team level, as discussed in the previous section, Samantha, Seet, Jones, Jane, and the rest of the CHMS members collaborated to identify the backlog items (see Table 19).

Table 19        Program and Team Backlog Items


Program Backlog Item

Team Backlog Item (Seet’s and Jane’s Team only)


Improve user experience by reducing number of steps

  1. Reduce the number of steps during reservation by using cached data, and prediction based on user’s location, and other information.


Provide waiting list functionality

  1. Provide waiting list functionality when making reservation
  2. Notify customers when rooms are available
  3. Handle the scenario when room becomes available when customer checks out early.
  4. Handling the case for VIP customers


Improve database performance

  1. Move old reservation records out of existing database (i.e. archive).
  2. Provide administrative functionality to archive records.
  3. Update report generation functionality to use archived records.


The Program Backlog Items were translated from the Vision of providing great customer experience over a robust Software System, discussed earlier in Section 22.3.5. These Program Backlog items were then translated to Team Backlog items that Jane’s team could work on.

For example, the Program Backlog Item for reducing the number of steps to carry out a use case.  Seet studied the reservation use case and found that it indeed had too many steps and found ways to simplify the use case. He put the improvements into Jane’s team product backlog.

Through discussions, Seet and Jane agreed the Product Backlog Items for Jane’s team, as shown in Table 19.

22.3.7 Working Iteratively

Having a good understanding of what needs to be done, as prioritized in the overall program backlog, Jones and Jane had to break that work down and determine what they could complete in each sprint. To achieve this, they worked iteratively in a synchronized way as shown in Figure 158.  This is really the application of scrum lite (as discussed in part 3) at both the team level and at the program level.

Figure 158     Running Program and Team Sprints

Jane’s team, as well as other teams in Jones’ program, agreed to run two-week sprints. At the program level, Jones decided to run monthly sprints. It is not uncommon in large-scale development that you might have multiple development teams running shorter sprints than an overall program level team.  Usually the program sprints are multiples of team sprints. For example, if a team sprint is two weeks, the program sprint can be 4 weeks, 6 weeks or 8 weeks. This allows the development team to reach internal checkpoints and verify their work prior to integrating that work with other teams at critical points agreed to at the overall program level.

Often, in these cases, on large complex efforts, individual smaller teams find that they cannot complete all the work that is part of the overall vision within the proposed time frame.  This is one reason why additional practices are needed in complex efforts to communicate related issues and coordinate any changes needed across development teams working in parallel. 

A common practice to help coordinate such issues across multiple Scrum teams is referred to as a Scrum of Scrums (See Figure 158). A Scrum of Scrum can be understood as a form of Daily Scrum, which we have seen in part 3, that is applied across teams rather than amongst members within a single team. At a typical Scrum of Scrums representatives from the individual teams highlight issues that are hindering their progress and creatively look for solutions. Just like the Daly Scrum, the Scrum of Scrum meeting should be short to make efficient use of time. The idea is to get teams moving as smoothly as possible. If there are any severe issues, affected representatives can organize a separate problem-solving session. 

Jones and his fellow colleagues agreed that they would run sprints according to the rhythm and duration as summarized in Table 20. With this settled, they were able to mark down their calendars, book meeting facilities and avail themselves for the activities.

Table 20        Jones’ Program Activity




Program Sprint Planning

Last Friday afternoon of each month

3 hours

Team Sprint Planning

Each Monday morning

2 hours

Daily Scrum

Every morning at 9am

15 minutes

Scrum of Scrums

Every Tuesday and Thursday after Daily scrum at 9:15 am

15 minutes

Team Sprint Review

Every Friday morning

2 hours

Program Sprint Review

Last Wednesday afternoon of each month

3 hours


Team members did not see the value of having these meeting rituals at first, but that changed soon. In the past the team members has always resorted to ad hoc meetings, which often disrupted the work of other team members. Having meetings at regular times allowed the team members to maintain their work momentum.  Moreover, over a period of time, the meeting agenda items evolved and became more focused and productive.

We now dive deeper to see how Jones and Jane benefitted from this way of working continuing the discussion from the preceding sections.

Jane’s team was responsible for providing the waiting list functionality. This functionality was related to room availability. Room availability in turn was affected by guest check-in/out, which was the responsibility of another team. Moreover, since Jane was also responsible for room reservations, she was also responsible for moving old reservation records to archive as part of improving database performance.

In the past, Jane had to persuade other teams, which her team depended upon to conform to her deadlines, and priorities.   But of course, other teams had their own deadlines and priorities, which were quite different from hers. This was resolved through the Program Sprint Planning activity, which got all team representatives together to align their deadlines and priorities, and reach a healthy compromise together with the Program Product Manager and Team Product Owners.

Both Jones and Jane sorted out issues with dependencies across teams through the scrum of scrum meetings. In particular, through the scrum of scrum meetings, Jane could get other teams to collaborate with her team, which allowed her team to complete integration earlier.

Finally, the review meetings were a highlight for everyone. Samantha the Program Product Manager participated and saw the value of having other stakeholders involved. She gradually invited folks from sales and marketing, customer service to attend the review meetings. This allowed people like Jane to understand business needs. It also helped these so-called business people understand what was required to communicate requirements clearly to development teams.

22.3.8 Continual Improvement

Continual improvement is essential on all software development endeavors, regardless of scale.  Jones knew that there would always be ideas his teams would come up with that could lead to better ways of working.  This is true regardless of the size and complexity of the software development effort. Therefore, Jones made sure that retrospectives were held at both the team level and the program level as depicted in Figure 159.

Figure 159     Conducting Program and Team Retrospectives

At the team level, the Scrum Master facilitates the team retrospective, in a similar manner as discussed in part 3, chapter 15 on Scrum Lite. The main difference when you scale in size and complexity is that there will be issues and impediments that the individual teams cannot fix on their own. This is an example of why we need additional practices when we scale up in order to make it clear to individual developers how they can raise to the program level the issues they see for discussion and proper attention.  At the program level, there needs to be a program scrum master, just like at the individual development team level there needs to be a team scrum master.  The program scrum master is responsible to conduct a retrospective that is appropriate for the overall program.  Experience has shown that program level retrospectives need to have representatives from each development team to properly express each team’s concerns.

In Jones’ program, these retrospective meetings were held in conjunction with the review meetings both at the team level and at the program level, at weekly and monthly cycles respectively. Smaller improvements that teams could quickly put into action got implemented quickly, while larger improvements that impact several teams, or even beyond the CHMS were deliberated in the program retrospective. Retrospectives were an important part of Jones’ program. It was the avenue where team members saw that management took an active part in understanding the problems they faced and cared enough to fix the problems. These problems ranged from having the right computing resources, having a good working environment, and policies. They could be as mundane as moving the printer to a more convenient location. 

22.4 Value of Essence to Large Scale Development

What we have just described above is the way of working of Jones’ program and Jane’s team. If you compare what Jane’s team did here in this chapter and what Smith’s team did in part 2 and 3, you would notice that there were significant differences. Jane’s team was using a decomposed Scrum in collaboration with the program practices. Moreover, she was working on part of the legacy CHMS, whereas Smith’s team was working on the new recommendation engine and using relatively new technology (at least at the time when this book was written) such as microservices. This means that the practices that are needed by one team are often different from another team even on the same endeavor when you are working at large scale development. This is because the issues the different teams have to face can be very different. For example, when you are dealing with a legacy system often the issues are related to understanding the underlying design and legacy requirements.  Whereas, when you are working on new functionality, often the issues are related to what the real need of the customer is that is driving the new functionality. 

Even Jane’s peers, and other teams in Jones’ CHMS program, used somewhat different practices. They have practices for testing legacy software, code reviews and so on, which we will not discuss in this book. Of course, Jones’ program management team had other (program layer) practices, which were different from the development teams’ practices.

Regardless of whether they were working on the team level or the program level, or what specific practices they applied, their practices were all defined on top of the kernel (see Figure 160), and the progress and health of both programs and teams could be based on kernel alphas and the alphas introduced via their chosen practices.

Figure 160     Different Practices for Different Teams

From what we have discussed thus far in this book, there are several important benefits of using Essence for both small teams and large teams (i.e. a program with subordinate teams):

  1. Providing a lens to evaluate the way of working, what practices to use, highlight risks, and so on.
  2. Provide a means to make practices explicit.
  3. Providing a means to evaluate the progress and health of each endeavor.

We have seen these benefits in action throughout the book with Smith’s story and in this part where we have introduced the Jones’ story.