From User Stories To Architecture (long version)

This is the slide deck of my LAS 2010 presentation: From user stories to architecture.

In this presentation, I’ll show you how we evolve the architecture of our software Sprint per Sprint (we develop software using Scrum).

You’ll see the architectural challenges you have to face when developing software in an agile way and how we cope with them to get an architecture that grows together with software.

Using Scrum, we build our software incremental and iterative:

Incremental means, building software feature by feature. Always a complete part at one.

Iterative means, building a first version of the software, get feedback on it and then improve the next version. Until the software is finalized.

When you combine both, software is built feature by feature (or user story by user story), get feedback on it and improve. It is important that a feature is always something complete. Something a real user can give feedback on.

This approach of building software leads to two problems regarding software architecture:

The first is incompleteness.

Incompleteness because the requirements are never complete and will change due to feedback provided by your customer on your progress.

The team will learn while developing. The team members will get better at using the technology used in the project and they will get a deeper understanding of the problem domain solved.

We always have an incomplete picture because we cannot anticipate every change that will happen during the project lifetime.

The second problem we have to face is change.

Change happens because requirements get clearer. During development, the team will ask questions to the customer to get a better understanding of the problem to solve. These discussions result in requirements that are more precise as the project goes forward.

Plans may change due to a changed business situation. If your customer’s competitor made a move then your customer may have to react and change the scope of your project.

In projects that take more than a year to develop, or during maintenance, technology will probably change. Take the .NET framework for example. There is a major update about every two years. Or the operating system changes (from 32bit Windows XP to 64bit Windows 7). Your architecture will have to take these steps, too.

Finally, people change. Either because they learn new things, simply change their minds, or they change teams. Your architecture has to take this into account, too. The architecture is no good if the team does not understand it or thinks it’s bad.

We have to design and evolve our architecture in a way so that these problems are under control.

We design our code – and therefore our architecture – in a way that it is open for new features.

However, it is important that we can extend our design without breaking existing abstractions. Otherwise, we will end with endless refactoring. We can add new features without changing the existing code base.

When requirements change, code has to change. If we want that our architecture remains stable, we have to ensure that we can make changes that are local.

Local means, a change influences as little code as possible. If possible, even without breaking any abstraction defined by our architecture.

Most important is simplicity. Simple things are easy to understand and therefore easy to extend or change. Simplicity means, the architecture fulfills all needs but in the least complex way possible.

There is no big design up front. Too much thinking ahead often leads to too complex architectures. Architectures, which are built to be flexible and extensible from the start, often break when the real problems arise.

There exist a lot of principles, practices and patterns helping to achieve a simple, extensible and flexible architecture and design:

One of the most prominent is S.O.L.I.D by Robert C. Martin.

Once you have an architecture in place, how do you keep it stable?

We write automated tests, which guarantee us that the tested properties of the architecture remain stable while we extend and change our software.

We use mainly two kinds of tests to achieve this:

The first kind of automated tests are unit tests.

Unit tests help us to build our software as a combination of individual units (classes, components). We can rearrange these units to achieve a change in the architecture while knowing that the building parts remain stable and will still work correctly.

The second kind of automated tests is acceptance tests.

Acceptance tests check a complete feature – end to end. Note: almost end to end: we replace for example the UI representation layer, data access layer and file system access with mock objects.

When we have to refactor parts of our software to enable the integration of a new feature, these acceptance tests guarantee that the existing functionality (functional and non-functional requirements) is not broken.

We use Scrum to develop software.

In Scrum, the project starts with a Kick-off meeting of some sort. Before the first Sprint (an iteration in Scrum), an initial product backlog is built to get a rough understanding of what is to be built.

Development happens in Sprints. Each Sprint starts with the Planning Meeting to plan the current Sprint (detailed discussion of what is to be built). The result is the Sprint Backlog.

Each day, the team stands together for the Daily Scrum Meeting to get a picture of the current status of the project.

To get feedback from any stakeholders, the newly built parts of the project are demonstrated in the Review.

Finally, each Sprint ends with the Retrospective, which the team uses to find improvements in the process as well as in the way they work (for example tools).

Now, let me add our activities regarding the lifecycle of the architecture:

Let’s have a detailed look at the different actions.

After the initial product backlog is created, it’s time to start thinking about the architecture:

The granularity of the initial architecture depends a lot on the type of the project. It has to be clear enough to get the development team running, but it should not be too specific either.

Firstly, because we want to start producing working code as fast as possible to get real feedback as soon as possible.

Secondly, architectural design decisions at this early stage may be difficult due to lack of know-how on the problem domain. If something does not yet have to be defined, then I’ll defer the decision to build up more knowledge first.

Therefore, the initial architecture is “only” a guideline for the team, describing the project in a very coarse way.

The architecture will gradually be refined during the project as you will see.

Every Sprint, my team holds an architecture workshop. Normally, on the first or second day of the Sprint.

The main topic is to build team-wide common knowledge needed to implement the User Stories (and other Backlog Items) of the current Sprint. Note that the team will come up with questions in the Sprint planning meeting while discussing the break-down of stories into tasks.

This meeting also helps to have equally distributed know-how of the architecture; resulting in better designs in each developer’s daily work.

Finally, the team may come up with architectural problems that were identified in previous Sprints. Either the team can solve them in this meeting or a team member takes responsibility for actions during the Sprint (normally the architect, but can be anyone on the team.).

While the developers are working on the code, they gather feedback about the architecture: Can the new feature easily be integrated? Are there new insights that lead to changes of the architecture?

It is important that the team keeps track of architectural issues. We write these on cards and hang them on our Scrum Board.

But how do you generate feedback on your current architecture in an early Sprint, when only parts of the system are built?

The key to this are simulators. Provide a simulation of the environment in which your architecture lives. Simulators can be mock objects used to isolate a dependency away from your system (e.g. the database, file system, user interface representation), or a real simulator for an instrument or other external device.

With these simulators you can check how your architecture will likely behave in the real world.

With every new feature in the software, the architecture is evolved further – always changing together with the code that it supports.

This leads to an architecture that really solves the problems at hand without far reaching assumptions that may never become valid.

Due to the fact that every developer drives the architecture, it becomes a team responsibility. Everyone can and should influence the architecture. That is important because the primary function of the architecture is to support the developers to solve the problem. Therefore, the responsibility lies with the ones who benefit.

Normally, evolving the architecture starts with a Spike. A limited amount of time to try something out. The important thing is that a spike is implemented end-to-end. Otherwise, it is probably not possible to predict how the architecture will influence the solution.

A spike is just a minimal path through the system.

An example: you have to build a search dialog, which allows a user to search for customers by specifying the search criteria (prename, name, address, age, …). The result is displayed afterwards in the user interface in a grid showing the complete data of all found customers.

The first step would be to implement the complete search functionality (UI, logic, query, display of results), but only with a single search criteria (e.g. name) and a very reduced results grid (show only name). This stripped down solution can be implemented very fast, because the individual layers are all simple (just for one criteria, show only name).

In upcoming Sprints, this Spike can be extended with further search criteria fields and a better results grid.

The benefit of using a Spike is that you can get feedback about the architectural impact with it, with just a fraction of the effort needed to build the complete search feature.

When everyone is changing the architecture just in the scope of their current problem, the design decisions taken may not be optimal from a broader point of view. These local decisions have to be transformed into global decisions.

For example, the same problem should be solved in the same way throughout the whole project; otherwise understandability of the code suffers.

Therefore, the team should review the current state of the architecture at the end of a Sprint to be ready for the future and refactor where necessary.

We use so called “commit-reviews” to reduce the amount of real reviews needed. In a commit-review the developer who coded a new feature shows all changes made to a fellow developer prior to commit the changes to the source code version system.

Reviews of the architecture are done on the code. And only working code is considered. We want to review the current state of what we really have not of …

… what we would like to have.

The purpose of documentation is to facilitate communication of the design and architecture. It helps new team members to get running fast, or to define interfaces between different teams.

However, documentation is a bad basis for architecture validation.

The software architect is responsible to drive the architecture: Provide input to the team when asked and take the lead in the review of the architecture.

As I said earlier, every developer makes design decisions every day. Therefore, the software architect has to coach the team to help them take the right decisions.

Finally, whenever the team is not able to decide for itself which way to go then its the responsibility of the software architect to decide. The reason for this is that it is still better to decide wrongly (and hopefully correct it later) than not taking a decision at all and probably stalling parts of the development.

We have seen that agile architecture is not a “do it once forever”. Architecture is a process starting after the initial product backlog is written and ending when the project ends.

There is only a valid architecture if there is working code to proof it. Code to get feedback whether the architecture holds or has to be changed.

Architecture is a team effort. Every team member is responsible that the architecture is evolved to support the currently needed features.

Requirements and plans change and so will the architecture. Inspect and adapt continuously to keep up with the changed environment.

Finally, if the architecture is continuously evolved, it will help you to keep up your development speed throughout the whole project lifecycle.