1 From Programming to Software Engineering

The goal of this chapter is to set the scene with respect to Programming and Software Engineering. The important issue is that Software Engineering is much more than just Programming. Of course, the running system created by Programming is an important ingredient for getting the reward of the entire effort of building the right system and it is important that the reader is actually able to use a programming language to create a program, at the least a small one. But it is by no means everything. Thus, the goals for this chapter are for the reader to attain the ability to

  • identify that software development is more than just putting a program together
  • explain the differences between programming and software development
  • identify the motivations behind the discipline of software engineering
  • explain some important elements of software engineering apart from the necessary bits of programming and explain how they relate to each other

What is fascinating about this aspect of software development is that it is more than just programming.  Rather, it is to learn the whole picture and as a software engineer solve a problem that users may have.

 

Understanding what software engineering is about as a new student is not easy, because there is no way we can bring the realities and complexities into the student's world. Nevertheless, it is a student's responsibility to embark on this journey of learning and discovery into the world of software engineering

Throughout this entire book, we will trace the journey of a young chap, named Smith, from his days in school learning about programming until he grew to become a software engineering professional and continued his on-going learning in this ever changing and growing field. In a way, we are compressing time into the pages of this book. If you are a new student, you are considered to be the primary audience for this book. Smith will be your guide to the software engineering profession, to help you understand what software engineering is about. If you are already a software engineer by profession, or you teach and coach software engineering, you can reflect on your own personal journey in this exciting profession. As an experienced developer you will observe a fundamentally and exciting new way to understand and practice software engineering. Regardless of your current personal level of experience, through Smith’s experiences, we will distil the essence of software engineering.

1.1 Beginning with Programming

The focus of our book is not about programming, but about software engineering.

Programming is here used as a synonym for implementation and coding. From Wikipedia we quote: “Related tasks include testing, debugging, and maintaining the source code, ... These might be considered part of the programming process, but often the term software development is used for this larger process with the term programming, implementation, or coding reserved for the actual writing of source code. Software engineering combines engineering techniques with software development practices.”.

However, understanding programming is an obvious place to start.

As a frequent user of applications like Facebook, Google, Snapchat, etc. whether it is on his laptop or his mobile, Smith knew that software forms a major component in these products. From this, Smith became strongly interested in programming and enrolled in a programming course where he started to understand what program code was and coding was all about. More importantly, he knew that programming was not easy. There were many things he had to learn.

The very first thing Smith learned was how to write a program that displays a simple "Hello World" on his screen, but in this case, we have a “Hello Essence!” as in Figure 1. Through that he learned about programming languages, programming libraries, compilers, operating systems, processes and threads, classes and objects. These are things in the realm of computer technology. We expect that you through additional classes would have learned about these things. We also expect you as a student to have some knowledge of these things as a prerequisite to reading this book. We expect that you have some knowledge of programming languages like Java and JavaScript.

Figure 1         Hello Essence

1.2 Programming is not Software Engineering

However, Smith quickly learned that programming on its own is not software engineering. It is one thing to develop a small program, such as the “Hello Essence” program; it is a different thing to develop a commercial product.

It is true that some fantastic products such as those that gave birth to Apple, Microsoft, Facebook, Twitter, Google and Spotify once were developed by one or a few individuals with a great vision but by just using programming as a skill. However, as the great vision has been implemented be sure that these companies are today not relying on heroic programmers. Today, these companies have hired the top people with long experience in software engineering including great programming skills.

So, what is software engineering? Before we answer this question, we must first make it very clear that there is a remarkable difference between hacking versus professional programming. Professional programming involves clear logical thinking, beginning with the objective of the program, and refining the objective into logically constructed expressions. Indeed, the expressions are a reflection of the programmers’ thinking and analysis. Hacking on the other hand is an ad hoc trial and error to induce the desired effect. When the effect is achieved, the hacker marvelled without really understanding why it worked. Professional programmers understand why and how it worked.

As such, professional programming is highly disciplined. Software engineering takes this discipline to software teams working on complex software. A typical software engineering endeavor involves more than one person working on a complex problem over a period of time to meet some objectives. Throughout Smith’s introductory software engineering course, he worked on several assignments, which frequently required him to work with his fellow students, and which included tasks, such as:

  1. Brainstorming what an event calendar app would look like
  2. Writing code for a simple calculator in a small group
  3. Writing code for the event calendar app, and hosting the app on the cloud
  4. Reviewing a given piece of code to find issues in it, for example bugs, and poor understandability
  5. Reviewing a fellow student’s code

Through these assignments, Smith came to several conclusions. Firstly, there is no one true way to write code for a given problem. Writing good quality code that fellow students can understand is not easy. It often takes more than one pair of eyes to get it working and comprehensible. He learned that:

  1. Testing, i.e. checking that the program behaves as intended is not easy. There are so many paths that executing the code can follow and all have to be tested.
  2. Agreeing on what the application would do was challenging. Even for that simple event calendar app, Smith and his team debated quite a while before they come to a consensus on what functionality ought to be available, and how the user interface should be laid out. Some discussions were rather heated and animated, and the true character of his fellow students started to surface.
  3. A simple application may require multiple programming languages. For example, the event calendar app would need HTML5 and JavaScript for the front end, and the Java and SQL database for the backend. Consequently, Smith found that he had to spend a significant amount of time learning and getting familiar with new programming languages, and new programming frameworks. Although, he endeavored to learn about all these, it was certainly not easy with the limited time that was available.

As Smith was preparing for his internship interview, he tried to summarize on a piece of paper what he understood software engineering is about, and what he had learnt thus far. Smith drew what he understood many times, and he observed that he couldn’t get it quite right. In the end, he settled for what is shown in Figure 2.

Figure 2         What software engineering is from the eyes of a student

To Smith, software engineering was about taking some idea and transforming it to requirements for a team to turn them into a software product. To do this, the team would engage in some kind of brainstorming, consensus, writing and testing code, getting to a stable structure, maintaining user satisfaction throughout and finally delivering the software product. This requires the team to have competencies in coding, analysis, and teamwork. In addition, the team needs familiarity with some programming language, like Java and Javascript, which Smith knew. What Smith didn’t yet know was that the tasks he had been given were still relatively simple tasks compared to what is typical in the software industry. Nevertheless, with this preparation, Smith marched towards his internship interview.

1.3 From Internship to Industry

With some luck, Smith managed to join the company TravelEssence as an intern trainee. Dave the interviewer saw some potential in Smith. Dave was particularly intrigued that Smith managed to draw the picture in Figure 2. Most students couldn’t and would get stuck even if they attempted to.

TravelEssence is a fictitious company, which we will be using as an example throughout this book. TravelEssence provides online hotel booking services for travelers (see Figure 3). In addition, TravelEssence provides “Software as a Service (SaaS)” for the operation of hotels. SaaS means that the owner of the software, in this case TravelEssence instead of selling it to its clients, in this case the hotels, provides the software as a service over the internet and the clients pay a monthly fee. Hotels can sign up and use the TravelEssence service to; for example, check-in and check-out their customers, print bills and compute taxes.

Figure 3         TravelEssence Home Page

Smith’s stint in TravelEssence provided a whole new experience. To him, his new colleagues seem to come from two groups, those who stated what they want the software to do, and those who wrote and tested the software. Figure 4 highlights the dramatic changes Smith experienced. While everyone seemed to speak English, they used words, which he did not understand, especially the first group. As a diligent person, Smith compiled a list of some of this jargon:

  • Book - To sell or reserve rooms ahead of time.
  • No-Show - A guest who made a room reservation but did not check in.
  • Skipper - A guest who leaves with no intention of paying for the room.
  • PMS - Property Inventory Management System which maintained records of items owned by the hotel such as items in each room including televisions, beds, hairdryers etc.
  • POS - Point of Sale Systems (Used in Restaurants / Outlets) that automated the sale of items and managed purchases with credit or debit cards.

It took Smith a little while to get on “speaking terms” with his new colleagues and mentors.

In his student days, Smith always wrote code from scratch, starting with an empty sheet of paper. However, at TravelEssence, it was mostly about implementing enhancements to some existing code. The amount of code that Smith saw was way above the toy problems he came across as a student. His development colleagues did not trust him to make any major changes to the system. Developers in TravelEssence emphasized code reviews heavily and stressed the importance of “Do no harm” repeatedly. They would repeatedly test his understanding of terminology and their way of working. Smith felt embarrassed when he could not reply confidently. Smith started to understand the importance of reviewing and testing his work. After his internship, Smith attempted to summarize what he understood software engineering to be about (see Figure 4). This was quite similar to what he thought before his internship (see Figure 2), but with new knowledge indicated in red and an emphasis on testing and doing no harm as he coded changes to the software product. Smith came to recognize the importance of knowledge in different areas, not just about the code, but also about the problem domain (in this case, about hotel management), and the technologies that were being used.

Figure 4         What software engineering is from the eyes of a student after internship

Competency not only involved analysis, coding and teamwork, but also extensive testing to ensure that Smith did no harm. Understanding programming languages was no longer sufficient, a good working knowledge of the technology stack was critical. A technology stack is the set of software technologies, often called the building blocks that are used to create a software product. Smith was familiar with multiple technologies that were being used including Java, JavaScript, MongoDb, and MySQL. Never mind if you do not know these specific terms.

Note: There are myriads of technology stacks available, and it is not possible for anyone to learn them all. Nevertheless, our recommendation to students is to gain familiarity with a relevant technology stack of your choice.

Smith graduated and was employed at TravelEssence. A few years later, at a get together, Smith and his old classmates shared their newfound experiences in the real commercial world. Smith, said, “At TravelEssence even though everyone seemed to be using different terminology, and everyone do things differently, there seemed to be something common to what they were all doing.” One of his old classmates asked Smith if he could explain more, but Smith just shook his head and said, “I don’t know exactly what it is.”

Some years later, Smith became a technical lead for a small group at TravelEssence. As a technical lead he found himself continually thinking about that discussion with his old classmates as he tried to figure out just what it was that was common about the way everyone worked at TravelEssence.

One evening the old classmates got together again. This time the discussions were a blend between technologies and people management. The old classmates were also talking more about their experiences dealing with people including their colleagues, managers, and their customers; consequently, they were talking more about the way work got done in their organizations. Managing stakeholders and their expectations became more important as they started to take on more senior positions.

After the meeting with classmates, Smith started to draw what he then thought software engineering was about (see Figure 5). The changes compared to Smith’s internship experience are highlighted in red.

Figure 5         What software engineering is from the eyes of a young professional

Stakeholder collaboration played an important part of Smith’s work. Collaborating well involved having an agreed set of values, principles and practices. These values included agreeing upon a common goal, respecting and trusting team members as well as being responsible and dependable. All of these values are qualities of a good and competent team player. Principles include, for instance, having frequent and regular feedback, and fixing bugs as soon as they are detected. All of these principles identify good behaviors in a team. Practices are specific things the team will do to deliver what is expected of the team consistent with the above values and principles, as well as good quality software.

1.4 Journey into the Software Engineering Profession

Smith through his experience at TravelEssence thus far has started to appreciate the complexities involved in producing and sustaining high quality software that meets the needs of stakeholders. He now appreciates that while programming is an important aspect, there is much more involved. It is the engineering discipline that is concerned with all aspects of the development and sustainment of software products.

Smith then reflected upon the knowledge he has attained thus far in his career. As a student with no other experience than having done some programming, it is quite difficult to understand what more is involved in software engineering. Typically, when creating a program in a course setting, the exercise starts from an idea that may have been explained in a few words – say less than one hundred words. Based on the idea Smith and his classmates developed a piece of software, meaning they wrote code, and made sure that it worked. After the assignment they didn’t need to take care of it. These assignments were small and to perform them they really did not need much engineering discipline. This situation is quite unlike what you have to do in the industry, where code written will stay around for years passing through many hands improving it. Here a sound approach to software engineering is a must. Otherwise, it would be impossible to collaborate and update the software with new features and bug fixes. Nevertheless, the experience in school is an important and essential beginning, even though he wished that it were more like the industry.

The authors of this book have all experienced, through their personal journeys, the importance of utilizing an engineering approach in providing high quality software. Thus, we can characterize, for you, what is important in respect to software engineering.

Considering the software industry, let’s put the success of Microsoft, Apple, Google, Facebook, Twitter, etc. on the side because they are so unique relying on innovative ideas that found a vast commercial market and programming, per se, was not the root cause of their success. In a more normal situation you will find yourself employed by a company that as part of their mission needs to develop software to support their business or to sell a product needed by potential customers. The company may be rather small or very large; and you will be part of a team. The reasons you won’t be alone are many. What needs to be done is more than what one person can do alone. If the software product is large your team will most likely not be the only one; there will be many teams, which have to work in some synchronized way to achieve the objectives to your company.

As a young student having spent most of your life at school and not yet worked in industry you may be more interested in the technologies related to software – the computer, programming languages, operating systems, etc. and less interested in the practicalities of developing commercial software for a particular business.

However, this is going to change with this book.

First let us consider the importance of a team. The team has a role in the company to develop some software. To do that they need to know what the users of the software need, or in other words they need to agree on the requirements. In some cases, they will receive the requirements indicating that they want software that does what another piece of software does. In these cases, the team must study the other product and do something like that product or better. In other situations, someone will just tell them what to do and be with the team while they do it. In more regulated organizations, someone (or a group of people) has written a document specifying what is believed to be some or all of the requirements. Typically, people don’t specify all the requirements before starting the development, but some requirements will be input to the team, so they can start doing something to show to the future users of the product. Interacting with users on intermediary results will reveal weaknesses and tell the team what they need to do next. These discussions may imply that the team has to backtrack and redo parts of what they have done and demonstrate the new results to the users. These discussions will also tell the team what more needs to be done.

Anyway, the team will in one way or the other have to understand what requirements they should use as input to the work of their team. Understanding the requirements is normally not trivial. It may take as much time or even more as it takes to program a solution.  As we just stated, you will typically have to modify them and sometimes throw away some of the requirements as well as work results before the users of the software are reasonably satisfied with what they received.

As a newcomer to software engineering but with some background in programming, you may think that working with requirements is less rewarding and less interesting than programming. Well, it is not. There is an entire discipline (requirements engineering) that specifies how you dig out the requirements, how you think about them to create great user experiences supported by the software, how you modify them to improve and sustain the software. There are requirements management tools to help you that are as interesting to work with as programming tools. There are many books and other publications on how to work with requirements, so there is a lot to learn as you advance in your career. So, working with requirements is one of the things to do that is more than programming but part of software engineering.

Another thing to do that is more than programming is the design of the software. Design means structuring the code in such a way that it is easy to understand, easy to change to meet new requirements, easy to test, etc. You can describe your design by using elements of a programming language such as component, class, module, interface, message, etc. You can also use a visual language with symbols for such elements that have a direct correspondence in the programming language you are utilizing. In the latter case you use a tool to draw diagrams with symbols representing for instance components with interfaces. In brief you express the design in a diagram form. The visual language can be quite sophisticated and allow you to not just express your design; for example, you can do quality controls using a visual language tool as well as testing the design to some extent. And, doing design is as interesting and rewarding as programming and it is an important part of software engineering.

Apart from working with requirements and creating a design there are many other things we need to do when we engineer software. We do extensive testing of the software, we deploy it on a computer, so it can be executed and used. If the software we have developed is successful, we will change it for many years to come. In fact, most people developing software are engaged in changing existing software that has been developed, often many years ago. This means we need to deal with versions of existing software and if the software has been used at many places (even around the world) we often need to have different versions of the same original software at different locations in the world. Each version will change independent of the other existing versions. And, the complexity of the software product just continues to increase. The only way to deal with this complexity is to use tools specifically designed for its purpose: testing, deployment, version and configuration control, etc.

So, you see that software engineering is certainly much more than programming. While definitions of software engineering are always a subject of debate amongst professionals, the following neatly summarizes our view:

Software engineering is the application of a systematic, disciplined and quantifiable approach to the development, testing, deployment, operation and maintenance of software systems.

It includes understanding what users and other stakeholders need and transforming those needs into clear requirements that can be understood by programmers. Stated in simple terms software engineering is bringing engineering discipline to software development. It also includes understanding the specific technologies needed to build and to test the software. It requires teams that have the social skills to work together, so each piece of the software works with other pieces to achieve the overall goal. So, software engineering encompasses the collaboration of individuals to evolve software to achieve some goal.

Programming is very rewarding since you immediately see the impact of your work. However, as you will learn during your journey, the other activities in software engineering – requirements, design, testing, etc. - are also fascinating for similar reasons. However, it has been more difficult to teach these other activities in a systematic and generic manner. This is due to the fact that there are so many variations of these activities and there has not been a common ground for teaching them until now as presented in this book. In the past, most students who have studied in the software domain have initially a desire to work with programming. However, as these people become more and more experienced they gradually move into the other areas of software engineering. This is not because programming is not important. In fact, without programming there is no product to use and sell. No, it is because they find the other areas to be more challenging and since these other areas require more experience to be successful. By essentializing software engineering as presented in this book, the full scope of the software engineering discipline will be easier to teach.

A personal experience
This is a story by Ivar Jacobson: “When started working with a new computer-based telecom switch, I learned to know many new people working in our teams. They were very fascinated with the new technology: computer, programming language, operating system, compiler, debuggers, etc. However, they were less interested in the application of this technology. Even many people who had worked with the telecom application for 3-4 years felt that the technology was most interesting: “If we just didn’t need to care about the telecom switch software, we could develop the best system in the world.” It is quite natural that you as a student are more interested in the technology than in the more sophisticated issues related to software development, because that is where your focus is during your studies.

 

What should you now be able to accomplish?

After studying this chapter, you should be able to

  • explain the relationship of programming to software development
  • name important motivations behind the discipline of software engineering
  • name important elements of software engineering beyond programming
  • explain these elements and their interrelationship at a high abstraction level

In order to support your learning activities, we invite you to visit www.software-engineering-essentialized.com. There one can find additional material, exercise related to this chapter and some questions one might encounter in an exam.

In addition to this find a short account of the history of software engineering in the appendix