Ative at Work

Good Project, Bad Project

Martin Jul — Wed, 11 Aug 2010 11:42:00 GMT

The secret to writing good automated tests is to be a good software designer.

It sounds trivial - but the sad fact is that more often than not the attempt to implement test automation falls due to poor software design where people are struggling with strong dependencies where you cannot test a single class or module without also configuring a whole environment of databases, webservices, populating whatever tables etc.

When the novice experiences this the natural, but incorrect reaction, is to write fewer and fewer tests at a higher and higher level of integration, the limit case being manual end-to-end integration testing through the GUI.

It is a symptom of bad software design, not automated testing being worthless.

In a good project the test suite does not need to be changed when external data changes. In a bad project, the test suite breaks when time passes or every time someone runs the end-of-month job to add interest to the accounts, or when the ticker price changes for a stock.

In a good project, only a few localized changes to tests and production code are needed as for a requirement change. In a bad project, there is impact analysis and lots of similar changes all over the code-base to accomplish this.

In a good project most of the code does not know from where it gets its data - in a bad project, even a simple interest calculation business rule knows about how to look up the database and the interest rate web service.

In a good project there is typically 1-2x as much test code as production code. In a bad project there is often 5x-25x as much production code as test code.

In a good project most tests exercise only a single class or a few functions. In a bad project every tests activates a code-path through many parts and tiers of the application.

In a good project most of the test code is asserting that things went as expected. In a bad project, most of the test code is setting up the environment to run the test.

In a good project there are tests at all levels of the application. In a bad project they are mostly through the UI and often even manual.

In a good project you can change the application without risk. In a bad project change is painful, infrequent, risky and requires executive sign-off.

In a good project you can deploy new code to production safely many times a day. In a bad project you only do it reluctantly every monthly or a few times a year.

Tweetable Code - Introducing Docjure for Succintly Reading and Writing Spreadsheets

Martin Jul — Fri, 30 Jul 2010 14:08:00 GMT

How would you like to be able to SMS or tweet your Excel data-crunching code?

With Docjure, it is possible:

(->> (load-workbook "spreadsheet.xlsx")
     (select-sheet "Price List")
     (select-columns {:A :name, :B :price}))

This reads the A and B columns from the Price List sheet in the spreadsheet.xlsx file into a list of maps where the a column has key :name and the B column has key :price.

I believe that is pretty good for code small enough to fit in an SMS.

Exporting your business data to spreadsheets for further analysis is just as easy:

;; Create a spreadsheet and save it
(let [wb (create-workbook "Price List"
                          [["Name" "Price"]
                           ["Foo Widget" 100]
                           ["Bar Widget" 200]])
      sheet (select-sheet "Price List" wb)
      header-row (first (row-seq sheet))]
  (do
    (set-row-style! header-row (create-cell-style! wb {:background :yellow,
                                                       :font {:bold true}}))
    (save-workbook! "spreadsheet.xlsx" wb)))

In just a few lines of code and you have exported the price list to a worksheet, complete with a yellow-background bold header line for extra style points.

In our business applications, bridging to Excel has provided huge benefits:

Users love it - having their data in Excel enables them to do much more than a static report that can only be changed by a software developer.
Developers love it - It eliminates a lot of tedious code for generating bespoke reports as we can easily export data into an Excel report template.
Project managers love it - using spreadsheets provides an easy to understand data exchange format and the flexibility to change reporting features quickly late in the project.
Sponsors love it - the project saves a lot of time and reduces training cost by leveraging an application the users already know.

As an inspiration, here are some of the things we have used it for:

Use Excel to calculate currency trading strategies - the traders calculate their currency trading strategies in Excel then import it into the trading system. They benefit from the flexibility of a powerful tool they already know and the trading system takes care of the technical details of the actual trades.
Exporting to Excel for bespoke analysis - would you like to check your currency trading record? Export all the information to Excel and the traders can slice-and-dice it easily with little or no technical assitance. This is much easier than setting up a reporting database for them to link to.
Using Excel sheets for content management to facilitate translation - translating an application to all the languages of an international enterprise was easy. Rather than using complex technical XML-files we made Excel sheets the data format for the application strings and had the subsidiaries add their translations to that. Then we put an adapter on the application to convert back and forth to XML-config files and RESX files used internally in the web and Silverlight parts of the application. The translators had a great experience, they had a familiar tool with spell checking and it reduced waste in the translation process by presenting a well-known easily accessible format.
Exporting to Excel for reporting - set up a spreadsheet template with a data sheet and some reports based on that, then populate it with data from your application. This allows the users to easily change or add reports with no change to the software application itself, thus dramatically reducing the turn-around time for new reports.

We believe this is so great that we just have to share it:

Therefore, Docjure is free and open-source.

It is written in Clojure for the JVM. Get it at our GitHub page here: http://github.com/ative/docjure and be sure to follow our updates on Twitter: @ativedk (http://twitter.com/ativedk)

How Scrum Reduces Rework

Martin Jul — Thu, 25 Mar 2010 22:05:00 GMT

I was talking to a group of project managers today in one of our inspiration meetings. One of the inspiring questions that came up was related to why we should use Scrum or agile methods.

One short answer is this: if we routinely plan to spend 1/3 of our project on testing and fixing defects there is a big benefit even if Scrum only helps reduce the amount of rework.

The next obvious question, then, is what are the most common causes of rework and how Scrum deals with these. Luckily we have some data on that, thanks to Otto Vinter (http://www.ottovinter.dk) and Søren Lauesen who wrote an article on the causes for rework (cited below).

In the project they analyzed they found four main causes of rework:

   ⁃   38% was caused by misunderstandings.
   ⁃   29% by missing requirements.
   ⁃   24% by changed requirements
   ⁃   9% by other causes.

Let's have a look at how Scrum addresses these.

Scrum Reduces Misunderstandings
The main problem is misunderstandings. This is why the "let's discuss the acceptance criteria" discussion between the Product Owner and Team before the development on a given feature starts is valuable. If we can avoid misunderstandings altogether the potential upside is a reduction in defects of 38%.

Scrum Reduces Superfluous Requirements
Then, the missing requirements. The just-in-time planning of Scrum and the demonstration of running tested software at the end of every iteration makes the identification of missing requirements trivial, so we will learn about these quickly. As such this does not reduce the amount of work due to missing requirements.

However, as we only commit to one sprint at a time it helps us to focus on building only the relevant, valuable features, which means that we also get a lot of feedback on which requirements are superfluous.

This means that we can more easily kill the half-baked ideas that we came up with in the beginning of the project when we had very little experience in the domain. Rather than building everything we thought of and then adding the missing stuff we look at the actual project and add the highest valued missing features week by week.

Thus, it gives the potential benefit of making us able to make the project smaller than if we had to commit early. That means less work, less rework and less complexity to maintain afterwards which is a great win over the lifetime of the project.

Scrum Makes Changing Requirements Inexpensive
One of the cornerstones of agile development is to make change cheap and easy. The reason we have Change Control Boards and other bureaucratic measures to prevent change from happening is that it is often very expensive and therefore we want to control it.

Agile development is all about making it inexpensive to adapt. If change was simple and did not introduce any huge risks of defects, downtime, deployment costs etc. we could happily embrace it. If we learned something new we could change the system accordingly, thus improving it and increasing its value. That is why we cherish changes.

Scrum tackles this head-on. Going through a full development cycle from idea to production ready code every iteration it highlights all the expensive steps in our development process. It spotlights error-prone development practices, expensive manual testing, the bureaucratic inter-departmental coordination and the costly deployment procedures. By forcing our way through the entire pipeline every cycle we force ourselves to optimize the delivery pipeline. Not doing so is simply too expensive.

The key is to embrace the pain and use it as an impulse to fix the root causes of the problem rather than yielding to the lure of the palliative solution of resigning to doing it less frequently or managing the complexity rather than removing it.

It may take years to improve the organization to be able to do this efficiently, but the way there is rewarding. It gets better step by step, iteration by iteration. If you want to get to somewhere else you need only clear thinking and perseverance. It is as simple as that. Just get started - and keep going.

Read the article:

Analyzing Requirements Bugs (2000), Vinter O., S. Lauesen - Bug Report Department in Software Testing & Quality Engineering Magazine, Vol. 2-6, Nov/Dec 2000

For extra points, analyze your own projects and fix the causes that are most relevant to your organisation.

Reboot Your Management

Martin Jul — Fri, 26 Jun 2009 13:05:00 GMT

How would you rebuild a country after a world war? How would you reboot your management?

This was the topic for my presentation at the Reboot conference.

It concerned Deming's 14 principles for management, the humanistic, long-term thinking, keep learning, use the scientific method, and build from quality philosophy, that helped shape companies such as Toyota and Honda..

Researching the background of his work, it was interesting to see how the World-War 2 Training Within Industry principles played a big role in shaping his ideas and how its focus on operating from a basis of scarcity - such as saving material, time and labour to win the war faster, plays so well in an entrepreneurial setting and in cutting through the big-company trap of just throwing more money at problems.

The slides from the presentation are in English and I have posted them for download and comments here:

http://www.ative.dk/om-ative/downloads.aspx

Read more Reboot stuff:

Web: http://www.reboot.dk
Twitter: #reboot11

What is a Truly Lean Company? - Deming Distilled

Martin Jul — Thu, 07 May 2009 13:12:00 GMT

Deming's work is the foundation of the Toyota Production System and modern-day lean. I recently conducted a workshop his 14 points for management. One exercise was to group them to get to their essence. Here is the the result:

“Practise engaged leadership” - hands-on and collaborative (points 7 and 2 – institute leadership and lead the transformation)
“Improve constantly” (points 1, 5 and 3 – provide jobs by building quality in and constantly improving the system)
“Look at the organisation as a whole” (points 9, 4 – end silos and look at total cost)
“Develop you competencies” (6, 13 – institute learning while working)
“Measure the system, not the individuals” (11, 10 – remove slogans, quotas, substitute leadership)
“Engage the people” (14, 8 – include everyone in improving the organisation and remove fear)
“Develop intrinsic motivation” (12 – remove the barriers to doing great work)

As you see, the lean principle of Respect for People is right at the core: leadership is participatory and collaborative, and the aim is to create jobs that people are happy to do.

A more elaborate description of a company based on these principles would be something like this:

A “Deming 14” company aims to sustainably provide great jobs by being highly competitive.

It is based on respecting the people and driven by their intrinsic motivation and the passion for doing a good job.

It leverages this passion to constantly improve the quality of the work by removing obstacles in the production system as they are identified. The focus is always on the end-to-end process rather than the individual steps.

It constantly takes the time to make experiments to learn how to do things better and uses the best new knowledge to improve.

To enable free, objective experimentation the company removes fear and focuses on the system rather than the individuals.

As a basis for the improvements, the company actively takes the time to learn on all levels, all the time in the context of the actual work.

The success of the company is built on engaging everyone in the work, and leadership is hands-on with an understanding of how everyone’s knowledge and contribution is needed to succeed.

Think about that for a second. How does your company compare?

PS: If you are interested in learning more about the 14 points and their application in IT there is a new edition of my handbook "Ud af krisen" available on our website as a free PDF-book in Danish. Get it here: http://www.ative.dk/om-ative/downloads.aspx

New Lean Reading List: Shingo Prize Winners Announced

Martin Jul — Tue, 31 Mar 2009 12:46:00 GMT

The recipient's of this year's Shingo Prize have been announced.

The Research and Professional Publication Prize category is a great reading list for new lean books:

Steven Spear - Chasing the Rabbit. A book on how to catch up with your lean competiors. HBR review here.
Mark Graban - Lean Hospitals. The book by the productive author of leanblog.org.
John Shook - Managing to Learn. Focuses on Toyota's visual A3 management tool.
Professor Peter Hines; Dr. Pauline Found; Gary Griffiths; and Richard Harrison - Staying Lean: Thriving, Not Just Surviving.
Jeffrey K. Liker, Michael Hoseus and the Center for Quality People and Organizations - Toyota Culture. Great author, new book. What's not to like?
Durward K. Sobek II and Art Smalley - Understanding A3 Thinking. Another winning book on Toyota's A3 tool from Productivity Press.

Off to the bookstore!

Improve your IT-organisation with Deming's 14 points

Martin Jul — Fri, 12 Dec 2008 10:20:00 GMT

I have written an introduction to Deming's 14 points and how they can help you improve your IT-organisation. It is a 15-page PDF in Danish. You can download it here:

Ud af krisen - http://www.ative.dk/media/989/ud%20af%20krisen%20-%20ative%20-%20a4.pdf

Enjoy - and please send me feedback on how to improve it. :-)

All the best,

Martin

Out of the Crisis - Deming's 14 Points

Martin Jul — Wed, 05 Nov 2008 13:57:00 GMT

In a time where so many are excusing their failure with externalities and saying "it's the crisis" we need leadership and entrepreneurship more than ever. We are in a time of creative destruction and it is precisely these moments that offer great opportunities to those who seek out the new ways of success.

In the early 1980es, the West faced a similar situation. We were blaming our failure on the Japanese manufacturers "dumping" products - selling below cost, because "no-one could create products so cheaply". Inefficient companies were threatened. Unproductive jobs were lost. The dinosaurs who would not or could not adapt lost their positions. And those who found news ways to improve productivity prospered.

At that time Dr. Deming came to the fore. His book, "Out of the Crisis" provided a comprehensive overview of the profound system of production that he had taught the Japanese to help them recover from the Second World War. The time had come to apply it in the West.

It has much wisdom and it is as relevant today as it was then.

Here are his 14 points, the essence of his teachings:

1. Create constancy of purpose toward improvement of product and service, with the aim to become competitive and stay in business, and to provide jobs.

2. Adopt the new philosophy. We are in a new economic age. Western management must awaken to the challenge, must learn their responsibilities, and take on leadership for change.

3. Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place.

4. End the practice of awarding business on the basis of price tag. Instead, minimize total cost. Move towards a single supplier for any one item, on a long-term relationship of loyalty and trust.

5. Improve constantly and forever the system of production and service, to improve quality and productivity, and thus constantly decrease cost.

6. Institute training on the job.

7. Institute leadership. The aim of supervision should be to help people and machines and gadgets to do a better job. Supervision of management is in need of overhaul, as well as supervision of production workers.

8. Drive out fear, so that everyone may work effectively for the company.

9. Break down barriers between departments. People in research, design, sales, and production must work as a team, to foresee problems of production and in use that may be encountered with the product or service.

10. Eliminate slogans, exhortations, and targets for the work force asking for zero defects and new levels of productivity. Such exhortations only create adversarial relationships, as the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force.

11. (A). Eliminate work standards (quotas) on the factory floor. Substitute leadership. (B). Eliminate management by objective. Eliminate management by numbers, numerical goals. Substitute workmanship.

12. (A). Remove barriers that rob the hourly worker of his right to pride of workmanship. The responsibility of supervisors must be changed from sheer numbers to quality. (B). Remove barriers that rob people in management and in engineering of their right to pride of workmanship. This means, inter alia, abolishment of the annual or merit rating and of management by objective.

13. Institute a vigorous program of education and self-improvement.

14. Put everyone in the company to work to accomplish the transformation. The transformation is everyone's work.

Reading assignment:

Read (re-read!) Deming's "Out of the Crisis" to get the basics.

Read Peter Scholtes' "The Leader's Handbook" for a comprehensive guide on to how to use it.

Then get back to work.

Stop blaming the crisis. Now is the time to lead.

The 14 Most Common Project Management Mistakes of and How To Avoid Them

Martin Jul — Sat, 06 Sep 2008 20:38:00 GMT

The Danish Computerworld magazine had a few pages on the 14 most common mistakes of project management recently. It is very illuminating to see how agile practices target these mistakes and help avoid them.

1. The project does not have the right people with the right competencies.

While ComputerWorld offers more analysis of skills and workload as a solution, agile practices provide a simple solution: a cross-functional, dedicated team is the basic building block for success.

When this is not possible use the retrospective mechanism to highlight our impediments (e.g.. overburdened key people) and continuously work on improving the situation. For example, I am working with one team that had this exact problem and they made a working agreement that the key people should always pair with the one of the others while doing the work only they can do. This provides a fast-track learning for the others and results in a situation where more team members can help on the resource-constrained tasks, thus reducing the workload on the overburdened key people.

2. The Project Lacks an Experienced Project Manager

The article states that a project can often run off track if there is no experienced project manager at the helm.

You can almost feel the underlying assumption that more management is the solution for all problems.

Scrum has a very radical approach to this, based on a completely different assumption. Eliminate the project manager and give the team the mandate to do what it needs to make the project succeed, then get out of the way.

It is a radical approach - but it works. The key assumption here is that the team disciplined enough to actually follow any practice, but if they are not, no amount of experienced project managers will be able to save the project from trouble. It also relates to the next common mistake:

3. The IT-department Does Not Follow a Standard Project Management Process

You will be be wasting a lot of effort if you don't have an explicit process that people are actually following. And "oops" - it's not enough to have a corporate procedure handbook - only practice counts. Even if people try do do heavy processes and spend a lot of money on training I frequently see that they get very little benefit, since people are still doing what they like if the process is broken. This creates a cognitive dissonance where the training material is focused on one thing - "learn the little red book by heart and be a good party member" and the real world is completely different. It is a situation that is well described in Solzhenitsyn works on the Soviet era where everybody was lying to themselves.

The only way to sidestep the need for process is to have a small group of highly experienced professionals who will follow good practices intuitively without noticing.

For the normal situation, lean offers a better solution:

The take here is to have a simple process, follow it rigorously, and when you master it, continuously work to improve it. This is why we like Scrum so much - it is simple: 3 roles, 3 activities and 3 artifacts, and yet very powerful.

In our training we frame it this way: "Everyone starts with the Scrum, but end up with exactly what they need". That is the whole point - not writing a nice handbook and doing something else, but rather doing a working agreement to work in a certain way, and continuously reflect on these agreements and revise them as needed to fit the situation.

4. The IT-department is Paralyzed by Too Much Progress

The key issue here is keeping to the specification to keep the project on plan (as if!) rather than adding a few new features that the customer might like.

The agile approach here is simple. We never overplan, and we talk intensively with the customer, showing them the real application and getting real-world feedback on where the project needs to go. We even do "Customer Co-Location" and have the customer and team work in the same office to consciously boost this communication and learning cycle.

It is value-driven software development, rather than plan-driven development, thus acknowledging the fact that the plans were done in the beginning of the project when we have very limited knowledge of the business and implementation domain.

As an aside, this is also why I like to speak of "the plan hypothesis", rather than just the Plan since it does not lead to the confusing doublethink that the plan is more real than reality.

5. Changes to the Project Scope are Not Tracked.

The issue is that the budget explodes - and the timeline.

This also reflects an underlying assumption, namely that the project must deliver the initial specification to succeed.

Agile takes another approach. Again, we are back to value-driven development and incremental delivery. By delivering working software frequently - for example, every two weeks, there is no need to track changes to the scope too much since we don't plan more scope than a few weeks ahead. Problems solved by not getting into the bad situation in the first place, rather than putting in a Change Prevention Committee and a big bureaucracy to keep the project in stasis.

6. Missing Up-to-Date Data on the Project Status.

Computerworld gives a one-word advise to remedy this: "Solution: Software."

If you chose to use software at least I would advice to be very careful about what you measure. Earned Value and other methods might look fine, but unless you have a proven track-record in your organisation about being able to use them to RELIABLY PREDICT the delivery of software I would advise to use a simpler metric:

In lean as in agile, we focus on value as defined by the customer: Running, Tested Features working flawlessly in the production environment.

This is a metric that is very simple to track, and hard go game. No more 80% done for many months.

They key agile practices here are Frequent Delivery, Information Radiators like kanban boards with backlogs, burndowns to communicate the current status of the process.

The Japanese call this Mieruka - making information visible so we can act.

7. They Ignore the Problems

In my book this is one of the most frequent problems I meet in large organisations. The "I know it sucks but I am not allowed to fix it", "It's the other department's fault" and all this silo-think.

The lean approach is to map out the entire end to end value creation process and continuously optimize the value delivery flow, not just in one department, but the whole thing.

Key agile practices here are current state analysis to uncover the flow, Retrospectives and other activities as a source for inspiration to Kaizen, the continuous rigorous experimentation with new practices and measuring how they work, one small change at a time so as not to introduce to much variation into the system that might slow down our evaluation of the new practice.

Scrum does this very explicitly by tracking these "Impediments" as on a day-to-day basis combined with a working agreement that the ScrumMaster gives to priority to ensuring that they are removed promptly.

8. They Don't Spend Enough Time to Define The Project Scope

The issue here is that if the project scope is not well defined it will explode.

Agile is very disciplined about this. We take our production capacity as a given and actively manage the scope every sprint. There is even a role for this. The hardest job in Scrum is that of the Product Owner who has to manage all the new ideas and trade-offs every day. It is difficult and is based on the principles of maximizing the value created by constrained set of resources similar to capitalistic entrepreneurship rather than the plan-economy approach of fixed scope as suggested by Computerworld. Come to think of it, next time someone suggests fixed scope on a large timeframe look puzzled and say, "oh, you mean production quotas as they tried in the Soviet Union?"

9. The Dependencies Between Projects are not Visible.

This is a frequent killer in large organisations and the bane of many a SOA implementation. Layers and layers of code from various projects, "The Enterprise Service Bus project", "The Mainframe Project", "The Workflow Project" that all need to coordinate to deliver the end-to-end features.

Computerworld cites an expert who advise to map out the dependencies to better manage them.

Agile advises to eliminate them altogether.

The keys to success here are cross-functional teams that have all necessary competencies to deliver the whole functionality, and, if that is not possible (and remember - "thinking you can't" is one of the biggest wastes), to have all teams integration continuously and track the impediments on a day-to-day basis.

Rather than managing dependencies we aim for just-in-time synchronized delivery by all teams. The key here is solid organisationwide product management, as autonomous, independent teams as possible, frequent delivery, integration and continuous automated testing to make it all spin. It is hard but the alternative is not very attractive, so start going in that direction.

The secret to success is that of Bounded Context - don't try to standardise everything on a corporate-wide basis but let teams do what they need to succeed inside clearly bounded contexts and only spend your effort on managing a limited number of integration points where these Bounded Context need to communicate.

10. They Don't Take Murphy's Law Into Account

The issue here is to mess with the plan meet a deadline set by an external party, like the CEO. Computerworld offers to talk to the CEO about what it takes in terms of time, resources and money to meet the deadline.

Agile calls this Customer Collaboration or engaging the stakeholders in the work to profit maximise the portfolio. The key here is to be honest and transparent and measuring only delivered features, not milestones. Otherwise the Portfolio management will be just like the planning meetings in the Führer bunker where the generals were talking about all their imaginary shadow armies and their victories to please the Führer rather than confronting reality.

I think every project manager should make a decision to never lie, speak against better knowledge or make wildly speculative statements about the future - in fact, the project status reports that I have encountered over the years and their "green-shifting" of bad news to good news as it filters up the chain of command have often indicated that the companies would be better off abolishing these report altogether and measuring status only in terms of what has been delivered without defects into production.

I think the former Avis car rental CEO, Robert Townsend, put it well when he refused to do earnings forecasts saying that, "we don't do in public what we can't do consistently well."

11. They Omit Transformation Management

This mistake is related to build "the next big thing" and forgetting to manage the transformation to make it work in the daily life: simply building nice new software that is not used.

There are various aspects of this, and it is at the heart of the agile practices. Value is measured in customer terms so it is a waste to produce things that are not needed, or to produce software that is needed and not provide the necessary support and training to make it useful in the daily work. This change management process and the resistance to change should not be omitted.

Some of the agile practices in that area are Customer Collaboration (on-site customers to make sure that we build what is needed), Frequent Feedback by delivering software to be used in the daily work and getting feedback about what needs to be improved and how we can focus the effort to make it even more valuable - in short - the big Lightbulb Idea is to simple have an ongoing conversation about the project with the people who will benefit from it.

12. Incomplete Project Plans

The symptom of this is that the project members don't know what should be finished when and thus have a hard time meeting these secret expectations.

This touches one of the areas where agile is really a hardline process. Planning-wise it is much more like situational planning in the war room than the big Politbureau meetings and their production quotas and five-year plans.

We work a lot with management to make this happen and it is very, very difficult to do well.

First, we advise a simple visual roadmap for the portfolio to put on the wall - just the big themes or strategic priorities on a timeline (don't add the details until the last responsible moment). In Scrum terms this is the "long-term" Product Backlog.

Then, for the next 2-3 sprints we have a Product Backlog consisting of a prioritized list of features to be done. They are explicitly managed according to trade-offs such as between the value they represent, the cost of doing it.

Then, for each sprint the team picks the top priority items and commits to delivering them at the end of the sprint. Along with this we have not only a short description of the feature to be done, but also two other things: Conversation and Confirmation.

Conversation means talking to the Product Owner to really understand how the feature should work. Confirmation is talking about the acceptance criteria, putting up a target and making it crystal clear for everybody how we will verify that we have completed the job.

The last thing is that the Team commits to what they can finish. We don't try to whip them into doing more. We say, "here is the most valuable things we can work on, let's estimate it and make good trade-offs to maximize the value created by your work, and then you, the people who will do the work, pick as much of the most valuable work that you can do without cutting corners."

It is simple, direct and takes a lot of discipline. It could also be seen as the Stephen Covey "Begin with the end in mind"-habit for software development.

13. The IT-department Does Not Reject Unrealistic Deadlines

The symptom is that the IT department has a reputation for not delivering on time.

The agile approach as outlined above is to have the team commit to what they can do, based on evidence of what they did last time. It is empirical and if we are honest and keep a consistent definition of done, delivering good, tested software at a constant high quality every time, our production capacity will soon be quite stable and predictable and something that can be measured rather than gamed. It is the end of overdue projects since we basically only commit to very small projects at a time.

On a bigger scale - say a 6-12 month project timeline, agile shifts the focus back to entrepreneurial thinking and saying, "OK Product Owner, we can deliver at this rate, how do you want to prioritize the effort." The Product Owner then, continuously evaluates the project business case and as long as the marginal utility of new features is better than stopping the project and starting work on something else, the project continues. It agile business, not just agile software development. And it is very very hard to do well since so many organisations are built on smoke and mirrors.

14. Poor Communication with Sponsors And Stakeholders.

Back to the basics again - "Customer collaboration", "Visible Status", "Frequent Deliveries", "Inspect and Adapt"... if you miss out on this point it is one of the best places to get started. Even if your organisation is not ready for "this agile stuff" you can try some ideas as suggested by Lee Henson:

"Instead of that weekly hour-long status meeting, how about meeting for 10 minutes every day to talk about the project"?
"How about keeping a visible list of the highest priority features to be built next?"
"Would you mind looking at the system every couple of weeks as we build and and giving us some feedback on how we need to adjust the priorities for what to do next?"
"Since we have already build the most valuable features first, shouldn't we put it into production and get a faster ROI? Maybe it is already "good enough" and we can finish early, under budget?"

I wish everyone great success in overcoming these mistakes and showing our profession better ways to work. The need is clear - and it is up to us to make it happen.

How to Improve Your Development Process Using Factory Physics

Martin Jul — Mon, 21 Jul 2008 15:35:00 GMT

There are only three types of buffers: extra capacity, extra inventory and extra time. This is one of the great take-aways from Factory Physics that I have frequently quoted since Thomas Blomseth introduced me to it on our lean study-tour to Japan earlier this year.

It is an essential piece in approaching process optimization.

Most processes have (and this is not a bad thing) critical steps that govern the throughput of the whole process.

For example, consider a development organisation that wants to improve its output. Assuming that the development process consists of the definition of work (the work of the Product Owner), the implementation of this (the work of the development team), the acceptance of the work (the PO again) and the user testing (the test team).

In our case, let's assume that we have an overburdened product owner who does not have enough time to specify the work properly and is hard pressed to check the work done by the development team, finding many misunderstandings after the work has been done, leading to much rework. We also assume that the PO is actually involved in doing a lot of testing - something that is often the case, but many times also a sign of poor acceptance criteria in the requirements.

In our case, the work is governed by the PO - in the work definition role and in the acceptance role. Until this bottleneck is removed, we can quickly conclude that there is no point in hiring extra developers since they would just be sitting idle. The PO capacity is the bottleneck.

Let's try the exploring the issue from the perspective of the three buffers.

First, we consider adding extra capacity for the PO role.

There are several approaches to think of - the first being "eliminate waste". We look at the PO's work and see if he is spending time on non-critical or non-value-adding work. This can either be eliminated or delegated - in fact, it is often a good idea to delegate to someone who are not so good at it if they have spare capacity. This includes training others to help extend the capacity of the PO-role.

The next thing is to look for extra inventory. To use this approach we would build up inventory of PO work in front of the development team, for example by giving the PO a head start on the project to build up a huge backlog so the team cannot catch up. Call this an analysis phase and we get to waterfall development. The downside here is that by doing this we practically eliminate the learning about the project that occurs in the feedback loop between work definition and implementation so it is probably not a good idea either.

This also applies to having the PO focus on the definition only and building up inventory before the "to be acceptance tested by the PO" stage, effectively building a huge inventory of work-in-progress of an unknown quality - something that is generally regarded as a very risky endeavour.

An alternative approach could be to eliminate the queue in front of PO acceptance could be to keep the work flowing and the cycle-time in check could explored by testing the hypothesis that we could off-load the queue by timeboxing on the PO testing and automatically promoting features to system testing after, say, one week in the "Waiting for PO acceptance" state, or by having the PO do prioritised testing of the key features inside this timebox or even uniting the PO and user testing into a single team. The hypothesis we should test here is that if there is extra capacity in the user testing stage it can be used to substitute for PO capacity even if it mean more rework or more work for the testers, using some of their spare capacity. By applying an end-to-end process this could turn out to be an improvement of the total system.

The last approach is to look for the extra time angle. The approach here is to lower the capacity of subsequent steps to near that of the PO throughput (usually with enough extra capacity to absorb the variation in the PO step so the system is kept stable). In our case this could be moving people from the development team do useful work on other projects or let them use their spare capacity to improve their development process - cleaning up legacy code, improving the build process, learning new skills etc. By trading for the buffer of extra time we slow down the delivery of the project to the gating factor (PO capacity), but waste less developer time in the meanwhile. In some cases this is a very good approach as a smaller team has less overhead for communication etc.

Turning back to the upstream steps we must also examine the PO role with respect to constraining the definition of work. We can address this in many ways, but the best way is to ask the team and the people involved to see what is really going on in the "Gemba" where the value-creating work is being done.

In some cases it could be that the PO is so stressed by the amount of work that badly defined work is being accepted by the team which in turn increases the burden on the PO to check and suggest changes to the work produced in the "PO acceptance" stage - something that impairs the POs total capacity by a vicious cycle on the constrained resource.

In this case we can explore approaches such as involving the team or the downstream testers more in the work definition stages to improve the requirements and make PO acceptance a lesser burden by delivering the right product.

By starting with Factory Physics, systematically exploring the problem from the angles of extra capacity, extra inventory and extra time gives us a simple, yet powerful tool to ask good questions and analyze the process. As such it is a great starting point for the team to generate ideas to try out to improve the process - and that's the way to creating better software faster.

Learning TPS from Toyota in Nagoya

Martin Jul — Thu, 05 Jun 2008 13:53:00 GMT

One of the high points of my recent Japan excursion was visiting the Toyota factory in Nagoya. We did a factory tour and meet with Keiichi Fukunaga, the manager of Toyota's department for teaching the Toyota Production System to other companies.

One of the things we discussed was the tricks speed up the transformation to TPS.

Keiichi-san's experience is that top management and front-line workers are usually quite enthusiastic about the transformation. For the front line, he noted, all you need is to go into their context and make some changes that are helpful to them. It is not just about efficiency but also about caring for the human side of the equation. He stressed the point that TPS should never be used as an excuse to fire people (the latter is what is called LAME – Lean As Misguidedly Executed on the www.LeanBlog.org).

For top management it should be easy to understand the TPS agenda – after all it’s translates to a better competitive position.

For middle management TPS can sometimes be a bit of a challenge. I think it is related to the fact that a lot of middle management consists of managing “waste” efficiently. Eliminating waste also reduces the importance of many of the bureaucrats. At this point he mentioned that it has taken Toyota 71 years to get to where TPS is today, so I think he was hintingat the need for patience when tranforming organisations. That also reflects our own experience in Ative.

In any case, he said, the key principle when doing transformation is to point people’s attention to the things that allow them to learn and improve.

For example, he mentioned an example from working with a company that made ashtrays for cars. They used a huge oven, bigger than the size of the conference room we met in, to do this – and it took a long time. This looked like waste to him so they had a discussion about why a room-sized oven would be necessary to make such a small thing as an ashtray. Following this line of questioning and measuring the temperature and other variables of the ashtrays in process they were eventually able to arrive at an elegant solution with a small table-sized oven and a much faster process, too.

From perspective of the ashtray company it could be seen as a big step forward – something called kaikaku – radical improvement – in some lean literature – but he really did not like this word. He said that even getting rid of a huge oven follows the principles of small step by step improvement, so even when it seems radical for the uninitiated we should call it just what it is: kaizen.

We also talked about the role of Dr. Deming and statistical process control. He said that it is very important to know this but it should not be the only tool in the toolbox. In the example just mentioned there was no need for Deming to improve the situation – it was all a matter of seeing the waste and removing it. This common sense approach is reflected in the spirit of the Toyota credo of “Good Thinking, Good Products”.

PS: For more information about the factory tour my travel companions from BestBrains have written an article here: Lean Study Tour day #3.

What's Your Favourite Lean Books?

Martin Jul — Mon, 11 Feb 2008 12:39:00 GMT

People often ask me about recommendations for the must-read books for lean and agile software development.

Here is a list of some of my favourite lean books:

W. Edwards Deming - the grand old man of the field, building on a strict statistical discipline. His book Out of the Crisis is a wonderful treatise on his thinking including his famous "14 Points for Management". This is definitely a must read that will change the way you think about management and quality!
Taiichi Ohno - one of the greatest industrial innovators of the 20th century, the father of the Toyota Production System. After spending his career relentlessly optimizing manufacturing at Toyota he wrote the book Toyota Production System: Beyond Large-scale Production that describes his work.
Womack & Jones - Their books are great and it is well worth to read them all to see a lot of the principles and case studies for lean thinking. Also, it is quite interesting to see that software development is now rediscovering some of the things that manufacturing learned much earlier - in the case of Toyota as early as in the 1950s and 1960s. Begin your studies with The Machine That Changed the World, a five-year study of the global auto industry from MIT and go on with the Lean Thinking and Lean Solutions. They give a fascinating perspective on manufacturing and plenty of examples of the lean principles and they applications. These are the books that brought lean to the mainstream.
Mary and Tom Poppendieck - with a background in manufacturing and software they have translated the concepts of lean to software development. They have written two great books and Implementing Lean Software Development and Lean Software Development - an Agile Toolkit. Both books are well worth reading a present a both the principles and lot of cases in a friendly, colloquial manner. Highly recommended!
Matthew May - I really like his approach to elegance and simplicity. May has worked with Toyota and their corporate university and his book The Elegant Solution offers insight into their innovation process - the principles it is built on and the practices that make it work.
Jeffrey Liker - his The Toyota Way is a very good introduction to the application of lean methods at Toyota. This is one of the best lean books I have read. Definitely a favourite!

What is missing? What are your favourites? Don't hesitate - post a comment now :-)

Beyond The Valley Of Tears

Martin Jul — Mon, 07 Jan 2008 14:51:00 GMT

Sabine Canditt of Siemens showed an interesting curve for Scrum implementation in her excellent ScrumGathering presentation in London recently. It shows how introducing the Scrum initially creates chaos and then stabilizes with a quiz success and good motivation. This is when the team gets into the rhythm and starts eliminating the waste that's directly under its control. After that we often see the velocity stabilizing and the the team's motivation plummeting as it becomes more and more clear how much of the waste cannot be removed by the team alone. This it becomes clear that the organization itself is the main impediment. When the team realizes this and has no mandate to remedy the situation we end up in The Valley Of Tears.

At this point, some organizations will decide that Scrum does not work and give up.

Better organization can look to Deming for inspiration on what to do.

Leadership is the key word and it is the key success factor for going beyond this point. The reason is simple. Large organizations are built to be robust so that they can accommodate a high turn-over of average employees and still generate predictable results. They are designed like supertankers to stay the course and to prevent change. Management, then is about keeping the machine running. Leadership is providing the vision and support to rebuild it in a better way as we go along, upsetting the status quo and going against the very nature of the robust organization. This is not only a middle management issue. Leadership and the constancy of purpose to drive improvements and innovation starts in the boardroom and it builds on the concepts of continuous improvement, learning and quality work.

It is a great challenge and our best wishes for the New Year go out to all the leaders who will make this happen in 2008.

Happy New Year!

Scrum Gathering, Fall 2007 – Open space

Kim Horn — Tue, 20 Nov 2007 08:08:00 GMT

The Open space on the last day (Friday) of the Scrum Gathering 2007 offered many interesting sessions and discussions. Two of these sessions were: “Product Owner practices – prioritizing backlog items isn’t enough!” and “Integration Teams (from Scrum in the Enterprise) – I just don’t get it!”

The product owner role
Scrum has been used for some years now and is starting to face some challenges. One of these is the Product Owner role. Many projects and companies have introduced Scrum with great success (improved performance for the dev team, progress is visible for the team and the management etc.). But after a period of time the progress is stagnating (after aprox. 3 iterations). The improvements made in the first period are related to issues internal in the team, things they can change them self. After proximally 3 iterations the team starts to face issues that are related to the rest of the organisation and the PO has a central role in this interaction.

The overall conclusion was that Scrum and the current PO role don’t solve these problems. There is a big gab between the PO and the senior management, sales, marketing etc. This area needs more focus and we must find some solutions in order to insure that Scrum continues as a successful framework.

To share our challenges and experiences we decided to create a public malinglist. Information will be published when the list is up and running.

Integration Teams – “I just don’t get it!”
Henrik Kniberg opened the session with the question: “does anyone have a example that shows how the Integration team works?”.

There were many god suggestions:

The integration team is responsible for infrastructure etc. to make it possible for the underlying teams to test the solution. This is close to Kens defition in “The Enterprise and Scrum”. No one had any expirence with this approach and we didn’t find any explanation on why you need a hierarchy of integration teams to support your dev teams. For me it starts to smell like silos!
The integration team is a virtual team of team members from the underlying dev teams that is responsible for the integration. No one had tried this and I don’t think that it will work.
The integration team is only working with the Backlog, breaking down the epics into smaller chunks and defining the user stories. In other words: a team that helps the PO doing his job. This is only suitable in large organisations. The question is wetter this will enforce silos and information handover?
If the project contains hardware it is not possible to do Continues Integration. Therefore you might need an integration team (the silo way).

Many god ideas but no real life example that shows how this is supposed to work.
It would have been nice if Ken had staid for the open space and joined this discussion!

I need to look deeper into Kens book, because I still doesn’t get it.

Variation And Waste - The Case for Frequent Delivery

Martin Jul — Wed, 24 Oct 2007 07:23:00 GMT

Imagine you are in an organisation with a one-year release cycle and you miss your annual deadline by one minute. The cost? A one-year delay.

Now, try to invent a control system to ensure that we avoid this kind of delay.

You would probably add some middle management to manage the dependencies between the various parts that need to be integrated for the release. You would need more management capacity for expediting work when interdependencies juggle the plans and you would need to spend time on replanning.

You would do detailed analysis to identify risks and try to mitigate the risk, no matter how little the probability that the event would occur. You need more people to sit and worry about possible future events.

Then, you would probably also make sure that leading up to the deadline nobody was on vacation, and you might order a code-freeze for the last months to avoid adding risk late in the project.

You see where we are headed. Step by step, we are building the standard, enterprise IT-organisation.

Since the cost of missing the deadline is so high, we have created a big bureaucracy and complicated our everyday work to deliver on time. Productivity is sacrificed to reduce the variation.

The elegant solution is different. What if we tried to make missing the deadline a no-stress event, where a little variation in the development process would not cause a full-year delay? What if we designed our system to be robust with limited variation so that small things have small impact?

This is the lean approach. We try to drive down the variation and create a simple management system rather than accepting it and building a complex system on top to tame it.

Imagine we reduced the release-cycle to one week, shipping roughly two percent of our annual work at a time. The variation is smaller. Miss a deadline by a minute and you release the following week. The cost of slipping is much smaller. There are no more critical weeks or month-long code freezes. You need less bureacracy. There is less replanning. People can go on vacation when they want.

Releasing becomes a no-stress event, something we do every week. It becomes just another day at the office.

On top we get some business benefits. Since we ship every week we can deliver incremental value faster. The cash-flow position is better since we get an earlier return. We become more responsive. If the market suddenly fancies red buttons rather than grey buttons we can ship that next week. Our annual-release competitors will have to wait until their next release - or even more if we make the change during their code-freeze.

The net result is an organisation that is sound from a business perspective, flexible, simple to control, less stressful and more fun to work for.

That's why we call reducing variation an elegant solution.

JAOO 2007 - Uncle Bob, Agile on the Mainframe, Jazz and Intentional Software

Kim Horn — Sun, 30 Sep 2007 12:53:00 GMT

Robert C. Martin did the opening keynote with the title “Clean Code II – Craftsmanship and Ethics”.

As always Uncle Bob is an inspiring speaker and the keynote was a good kick start of the conference – why we are here. Some of his well-known issues to focus on and improve:

Discipline
Short iterations
Don’t wait for definition: The customer doesn’t know what they want anyway. The team should participate in the definition of requirements.
Abstract away volatility: Don’t place code together that change changes for different reasons. Eg. don’t mix GUI with business logic.
Commitment > Omitment: Help find a solution to the problems instead of just blaming someone else.
Decouple from others: Build stubs and simulators for your external dependencies.
Never be blocked: This is the prime directive. If you can’t get a QA environment when you need it, use a dev machine.
Avoid turgid viscous architechture: There is no single perfect architecture. Ivory tower architects are useless - architects must write code.
Incremental improvement: If you have bad code, improve you code incrementally. Every time you work on a part of code you should improve the quality, gradually improving the entire system.
No grand redesign: Related to the above point. Grand redesigns always fail. And where are the requirements for the redesign? In the current system, which will be a moving target and the redesign will never catch up.
Don’t write bad code. The only way to go fast is to go well. As a development team our product is code. If the code is a mess the product is a mess. Focus on code quality, you can change the development team, management or the process – but bad code stays.
Clean code: clean code is code with no surprises. Make small functions (20 lines).
TDD
QA should find nothing: Probably they will, but this should be the mindset.
100% code coverage
Avoid debugging
Manual test scripts are immoral
Definition of done: Don’t have more than one definition of done, done counts.
Test through the right interface
Apprenticeships
Use good toolsConclusion: as always – he is so right.

Experiences with Agility in the Mainframe Environment
Dr. Jan Mütter, LSD NRW.

Dr. Jan Mütter from a German government mainframe project, told about his experiences with agile methods . The first year they worked the “traditional way”: the customer (the government) sending specifications to the development team. The team checked the specification. If it had errors or the team was responsible for any issues in it, the specification was send back to the customer.After one year and almost no progress, they decided to change the process:

Customer co-location, the whole team in one building.
Findings: Remarkable improvement in team communication. Extensive rework on nearly all results (code and documents).
Planning game, weekly status meetings and adjustments. Detailed tracking of work packages.
Findings: low leadtime from requirement to implementation.
Weekly build and deployment.
Findings: problems with configuration and release management.
Weekly test, full automation on all tests.
Findings: Lot of rework because of requirement changes. Test scripting very costly.
Release and configuration management.
Findings: necessary but very costly.
Refactoring
Change Request management
Findings: Lot of rework. No differentiation between work package, change and defect. Large positive effect on culture and discipline.
Project organization and communication. Software design team makes FQA in relation to requirement analysis.
Pair programming. The developers were encouraged but not forced to pair program.
Finding: Low acceptance.
Common goal for the team and the customer (working together).

They used VisualAge as development tool on the mainframe and the build in ITF test tool. This combined with the fact that they didn’t have a lot of “old” mainframe developers made it possible to make weekly builds and automated tests.

The conclusion: The environment (goverment company and goverment customer) were a big challenge. E.g. the “service” organization (system operation) did not have the same goal as the team, it took one year to install and configure WebSpherer on the maninframe.The transition gave the project a major improvement, but are still a lot of things to improve.

Developing Software like a band plays Jazz - From Eclipse to Jazz"
Erich Gamma, IBM.

Erich Gamma is a Distinguished Engineer at IBM Rational Software's Zurich lab. He is one of the leaders of the Jazz project. He was the original lead of the Eclipse Java development environment and is on the Project Management Committee for the Eclipse project.Erich gave a introduction to “jazz” and used the Eclipse project at reference (“the eclipse way”).The Eclipse way (in short):

6 week iterations.
Milestones first (time boxed). Make milestone results visible.
End game: Tight schedule. Process rules have high priority, bug rate and velocity rates go down.
Team first. Focus on the team. Small teams. Different time zones (8 locations). Each team have their own plan. Many roles in the team -> every one can “step in”.

The Jazz product gave the team a lot of information about the project (current status etc.) and can ease many tasks. E.g. it can auto generate a server environment as a copy of the last build (very usefull if/when the build fails).

Conclusion: Jazz has some interesting features, but it is targeting big organizations (like IBM) or control freaks. If you use the full feature set, it will not necessarily give you a more agile project team. A more lightweight product and process might give you a better result.

"Intentional Software - Democratizing Software Creation".
Charles Simonyi, Intentional Software Corporation. Henk Kolk, VP and CTO, Capgemini

Charles gave an introduction to how they try to accelerate innovation by integrating the business domain experts into the software production process. They used a case from Capgemini (Netherlands) where they used the Intentional Software tools to create pension systems.

Their approach is to ask the business domain experts how they describe a new pension product. In this case they use a large Excel spreadsheet to describe the business rules. Capgemini made a customized view/ editor to the customer that looked like the spreadsheet they know.

Before this the customer used 5 very skilled developers and 3 years to create a new pension product. With this new approach (combined with the Intentional Software tools) it took one person 3 months to create a new pension product. They also converted apox. 40 products within 6 months.

Charles used a CAD application as semaphore: the designer draw a circle, it can be a cylinder or a ball. If you change the view you can see how the object looks. The designer doesn’t care about how it is stored in the database. The base system (the CAD system) can be used in many different types of design projects, it is not customized for one specific customer or project. It holds common information’s and properties for “objects” regardless of their usage.

Conclusion: This session showed a red line from the 1970’s where Charles created the first WYSIWYG editor, his time from Microsoft where he was the father of Excel and Word.

The session opened our eyes for a “new world” of software development. The question is when will it be available to the general market, and not just selected partners?

The TDD Controversy - JAOO 2007

Martin Jul — Fri, 28 Sep 2007 07:12:11 GMT

TDD will ruin your architecture! When Jim Coplien said this in his presentation about Scrum and Architecture at the JAOO 2007 conference he really got people's attention. His statement sparked one of the most passionate debates at the conference.

On Roy Osherove's initiative we did an open-space session discussing the topic.

The controversy seemed to be related a lot to what TDD really means. It became quite clear that TDD means different things to different people.

Starting with testing we did reach consensus on some key issues, however:

Automated testing is a key enabler for agile development.
Test-first is the most effective approach for this.
Test-first on the micro-level such as writing a "SaveModifiedInstanceShouldUpdateAuditTrail" test for a Save method seemed to be acknowledged as a Good Thing. Micro-level here means developer testing on a sufficiently low level of integration. There was no consensus on whether to call this unit testing or subsystem/component testing.
Testing only the return values of methods has limited benefit, and many of the people attending did not write tests for simple state like property getters and setters. Also, Jim mentioned some studies showing that only a small percentage of actual bugs are related to state.
Testing the behaviour - especially the interactions between objects - is where the real value is at. This is one of the reasons that mocking frameworks are so popular.

There was no clear consensus on whether or not to use a test-first approach for testing the application from a business perspective, ie. testing it on the macro level. One argument was that you need to have a little bit of the application in place before it makes sense to integrate it so it would not be truly test-first. However, no-one questioned the value of automated testing on the macro-level (see the post on why acceptance tests matter) and we all recommend doing it at the earliest possible moment.

The microlevel testing mentioned above is what I usually call unit testing (although it may not be what unit testing originally meant). It involves a class under test in a controlled environment. This means that we have mocked the interesting classes that it interacts with, but full domain classes may be used as well. For example, if we are testing a mortgage calculator that takes a house and a loan profile as inputs and interacts with pricing service I would usually recommond creating a semantically valid, complete house with the real domain classes (from a factory - the ObjectMother pattern). I would explicitly create the loan profile (amount, duration, interest rate etc.) in the test since it contains the "interesting" variables related to the calculation and I would mock out the service that provides the current ticker price for loans by interest rate that the service uses to calculate the mortgage. The point is that even though I call it a unit test we are at a low but non-trivial level of integration.

Learning to be good at this takes some time since it is easy to write bad tests - in fact, Roy is currently writing a book on this called The Art of Unit Testing to help beginners climb this curve. Jim mentioned Behaviour Driven Development several times as an example of how to do this. From my personal perspective BDD looks like an elegant expression of where you can go with micro-level testing and experience.

Now, what about TDD then? Jim quoted some studies of student programmers showing that TDD done strictly from the YAGNI principle leads to an architectural meltdown around iteration three. One of the examples he mentioned was that of a bank account where TDD would lead to modelling the account as the balance (possibly with an audit trail) whereas a domain analysis would lead to modelling it as a collection of transactions with the balance being a function of these transactions.

However, let's also note that it is entirely possible to do bad architecture without TDD. From my perspective it is also very hard to do no architecture even with TDD if you have experienced people on the team. The reason is that they will know sound patterns and structural concepts that they will use, even unconsciously. For example, Jimmy Nilsson wrote a book to define a default architecture for .NET business applications, and Ruby on Rails provides a common architecture for web applications. Using these appropriately help keep your large-scale structures sound. This is one of the reasons why alpha architects working in known territory can avoid architectural meltdown in the large even with TDD. You can still make mistakes in the domain-specific modelleing. In fact I don't recall a single sizable project, big upfront design or not, where we did not learn something that would guide us to a better architecture if we had to do it again.

But, alas, as someone said in the conference: Good judgement comes from experience - and exerience comes from bad judgement. The goal is not necessarily to avoid failure entirely but rather to make it short-lived and inexpensive to correct. We are aiming for cost-efficiency rather than paralysis.

I will leave the controversy at this. As you can see there is a consensus about most of the issues related to test automation and test first. If that's your definition of TDD then by all means, please keep doing it!

Jaoo 2007

Kim Horn — Mon, 24 Sep 2007 05:17:00 GMT

We're off to Jaoo 2007 next week.
We are bringing some poker planning cards and some "I O Scrum" t-shirts, please give us a hint if you are interested.

Every day we will post a summery from the sessions of the day.

Bye, bye, "Done, but" - The Flat Organisation and Enterprise Scrum

Martin Jul — Mon, 17 Sep 2007 18:24:01 GMT

Lean organisations tend to be very flat with few layers of management. They rely on self-organisation and leadership. In this context, then, what is the role of the middle layers in the organisation? This is one of the points Ken Schwaber addresses in "The Enterprise and Scrum". The answer is amazingly simple.

The middle layers are Scrum teams that prioritize the work for the subteams and integrate the results.

In other words they prepare the product backlog for the subordinate teams and provide leadership, coordination, infrastructure and support for the teams below. This means that the integration teams are complete, cross-functional Scrum teams with full delivery capability. Compared to many traditional organisations this means that release engineering activities, QA, etc. move up in the organisational hierarchy, making the downstream steps in the value stream the integrators and bosses of the earlier steps. We are, in effect, a creating kanban system for the whole development process. Through this we create flow and remedy the common failure of deferring integration work to the end only to learn too late that the parts do not fit well together.

This organisation of Scrum teams builds on lean principles such as frequent integration and "jidoka" - stop-the-line bug fixing. Every team is responsible for integrating its own work and that of its subordinates. If a subteam delivers something that is not good enough, the work is pushed back down and not accepted. This way, we make sure the details are always correct and shippable. As a result, the whole product remains integrated and shippable.

The ScrumMaster on the integration team is facilitates the removal of impediments that cannot be removed by the front-line teams themselves. This creates a system for multi-layered kaizen where the change is always initiated by the people who experience the problems and escalated to the proper level of authority. Through this the integration teams switch from command-and-control to serving the teams below.

One of the benefits from this structure is that the Product Owner on the integration team defines the product backlog for the subteams based on the coarser-grained backlog and priorities from the next level up. In this way the integration team and its subteams is treated as a virtual, cross-functional team. Since we are not allowed to produce loose ends, it forces us to actually start only activities that can be completed in a sprint with the necessary coordination between subteams. We have to begin with the end in mind.

For example, if a backlog item for adding a new report for to a reporting application is selected, we have to make sure to request the whole feature - both the new GUI (a pretty diagram, maybe), and the necessary updates to the back-end server application to provide the data. If there are separate teams working on this the feature is split across different team backlogs. In many cases this requires coordination between multiple teams when planning. The key point is to never commit anything to the integration sprint backlog unless all aspects of it are committed to the sprint backlogs of the subordinate teams.

If this is not possible we have to break it down into smaller pieces or leave it in the product backlog. We never accept work that cannot be completed and demonstrated at the end of the iteration.

When we implement Scrum in organisations this is something that it usually perceived as quite frustrating in the beginning. Many people have a strong habit of doing for partial work. However by only committing completable work we reduce one the great time-consuming activities in project management, namely managing dependencies between projects, trying to get features done in isolated, non-integrated layers and the coordination and "expediting" work to lobby different teams to change their plans to get it all correct and integrated on time.

The end result is a system where business goals and priorities flow from the top down and demonstrable business results flow from the bottom up. Every iterations ends with completely integrated realisations of the top level backlog items. It means goodbye to, "we're done with the new report, but we have to wait three months for the database team to expose the data for us", and hello to, "here's your new report, ready for use". That's a pretty good return on eliminating bureaucracy.

Implementing Scrum in the Enterprise - Agile 2007 Day Four

Martin Jul — Fri, 17 Aug 2007 08:06:00 GMT

Ken Schwaber has been working on enterprise-wide Scrum implementations recently. In his session today he shared his experiences and offered a roadmap for the process.

First off all there is no "Enterprise Scrum" - it is the same simple, empirical framework for developing complex systems that is applied everywhere rather than just in software development.

This means that even senior management will also be working iteratively with a backlog and showing demonstrable results at the end of every iteration. This in turn, means radical transparency through the whole organisation at all levels.

In Schwaber's experience the enterprise-wide roll-out of Scrum take some time. He offered the case of a 1000-developer organisation. Here, they spent six months on the roll-out (training everybody in the Scrum practices) and after that 3-5 years to implement it make it stick by iteratively removing the impediments to producing quality software in the organisation. As he said, "Culture eats strategy for breakfast", so this phase is all about changing people's habits and getting the waste out of the system. During this implementation a senior management executive Scrum team is in place using Scrum to deal with the impediments and demonstrably solve the top issues as they become visible during the transition.

There is no "Done" - once the Scrum framework is implemented the organisation is in a state of continuous improvement, reflecting on its changing environment and evolving to stay ahead of the game.

He offered this road-map:

First, do a pilot project to learn the Scrum basics, then try Scrum on the most challenging, critical project in the organisation to make a proven case for everybody that is works.

Then do the enterprise roll-out. This is where it becomes top-down and is implemented at all levels.

Here is how the executive team works:

First, create a backlog of what is wrong today - what is in the way of developing quality software (eg. "we have too many projects", "we don't integrate often so we have no visible status", "we produce too many lose ends"). Then commit to fixing it and use the Scrum process to fix it - first things first. Senior management has to demonstrably fix the top issue or issues in one iteration. Then attack the next top one, etc. etc.

Meanwhile, the Scrum training for the entire organisation starts. This is the roll-out.

For all the work, the Scrum team (max 9 people) is the basic building block, and Scrum-of-Scrums is use for hierarchical breakdown of the organisation.

The benefits is a very visible status in the entire organisation, that it is all working from a single backlog and aligned with creating business value rather than some arbitrary and sometimes detrimental performance metrics (Jim Highsmith did a talk on agile performance management at the conference with some very good findings on this topic). There will be a very simple and open control structure encouraging an open environment over command and control - no need for additional committees and boards to control things at an often too late time.

Corporate Judo - Guerilla Tactics For Agile Transformation - Agile 2007 Day Three

Martin Jul — Thu, 16 Aug 2007 00:59:00 GMT

Today, we did a Discovery Session at Agile 2007. The topic? How do we implement agile. We worked in five teams that each created a roadmap for implementing agile based on their experience and their role in the organisation: management, team and customer.

A big thank you goes out to everyone who participated. Please add your comment and notes to this blog entry.

Michelle K. Cole, VP of Operations at ENVISAGE Technologies kindly wrote up the findings from the five teams as they presented. Here they are.

Where Should a Manager Start?

get executive sponsorship
create vision & trust
education about benefits of agile
create common language and measurements
show value to team (what’s in it for me), benefits to the projects
provide a supportive environment
figure out success factors
set up agile advocate group
identifying if you need external coaches
establish team maturity model
staffing – hire those people that might be good in this environment (you might even need to fire some if there is too much resistance).
empower them to come up with a better solution
create a system of custom ownership
use peer pressure to bring other people along

The other Management team offered this approach:

do a big kick-off event
send key people to conferences like Agile 2007 etc.
provide readers/skeptics with books - some people learn well from books and many techies like to read a lot.
pilot a project & praise success
encourage culture of trying things
meet with individuals to understand pain points and processes to keep
Establish brown bags/training for people that like that style
Rearrange workspace so team sits together

Guerilla Tactics for Teams - Developers/Project Managers/QA

how to overcome resistance to sitting in team room

have stand up in team room and don't let them leave afterwards!

how to get space for collocation

squatting long term
have stand up outside mgmt office
furniture on wheels
midnight runs

resistance to pairing

o stealth pairing (ask people to help another)
o set up a pair station
o coolest equipment in the pairing machines
o have appropriate hardware to pair (2 keyboards, two monitors, movable desks)

resistance to collaboration

make sure there are common goals

resistance to agile

convince the cool kid it is cool
explain there isn’t that much change
bust the myths

mandated process

ask forgiveness

resistance to TDD & automated tests

demonstrating the usefulness
pair with fans of TDD

involve QA

have the business analysts work with QA on acceptance tests

PM resistance

Show other non-software projects
Shadow existing projects
Send the PM on vacation

Also from a Team perspective

Agile is not binary; it is a continuum. You are someplace. Where do you want to be tomorrow?
Change should not be inflicted upon people. You should figure out what you can do and start to make that change.
As a developer, start doing TDD.
As a team, start doing stand up.
Agile provides many benefits. Figure out which benefit you are trying to achieve most and start with the following:

Focus on Feedback - Fix length iterations with retrospective
If you need Flexibility - TDD is a good practise
Communication Issues are addressed Daily stand ups

Customer team

This was a very interesting team. Most of the people in the session had experienced resistance to implementing agile. The people from Yahoo, however, had not - for their two teams they just did a team meeting, introduced the agile practises and since the teams liked it they decided to do it. Unlike most of the other people in the session they were completely self-organising and met no resistance to the change from their team, the management or the rest of the organisation.

The group presented some suggestions for going agile

Get trained together to improve buy in
Be where the developers are; wait for them to ask questions
Us v. them – start doing things together as a team, may want to get an outsider to help build trust
Define product backlog & prioritize

Other Ideas

Doing Agile with Outsourcing

Use Scrum of scrums to synch the teams and beware of time-zone issues.
Have people go to other locations for a while to build trust and improve collaboration.
Find a balance of documentation – increase to improve communication.

Partial change - do it gradually (actually, other cases like the Salesforce.com experience report we mentioned the other day advocates going all-in).

Survey people on what they need and fix it - it is often small things that make a big difference

If the rooms feel to sterile let people hang personal items from ceiling (eg pictures)
Add plants

Empowerment based on company directives

Room inversion

Switch to work in conference rooms and use offices for personal time (eg. phone calls).

Impressions From Agile 2007 - Day Two

Martin Jul — Tue, 14 Aug 2007 23:42:20 GMT

Dave Thomas told about his experience with large-scale agile projects. His take on plans:

The Plan is really just The Best Wrong Answer

Having said that however, he also noted that he had a very good track record with hitting the shipping dates - the worst project was Visual Age for Java which was three weeks late. His secret? Letting the team own the plan and doing estimates in ranges (wideband delphi) to prepare the planning session, then iteratively negotiating over the estimates until a consensus is reached.

Release Engineering - Getting the Development Infrastructure Right

Dave Thomas also named Release Engineering - the work to create the development infrastructure, automated builds and testing and providing dashboards with status for the project to be a key activity. In fact, his experience is that around 10% of the total effort should go into this area. The key points:

it is not something you outsource to the IT-department - instead, use real developers and co-locate them with the development team the first 3-4 months of the project as they work to set up configuration management, automated build, integration and testing.
Do focus on dependency management - you need to have thin slices of all the components with automated acceptance tests for the interfaces in place early so no-one is blocked by missing components.
All components should be tested independently - and provide acceptance tests for all interfaces.
Set up an automatic "Product Development Dashboard" with all the vital information - the goal is to eliminate the need for asking, "how's it going". This includes build status, test status, coverage etc.

Jim Highsmith did a nice presentation on agile from a management perspective, stating that the key to success with agile is to implement it throughout the enterprise. Doing it in silos is a suboptimisation.

The key benefits of going agile:

it helps solve issues on developing features in a large, complex system.
it improves quality, and
it improves predictability

The big value in agile is all the things you don't do.

not wasting time on the features you don't need (most features aren't used anyway).
the value is driven by the business needs, not by some arbitrary priorities.

The essence - from John Wooden: "Be Quick, But Don't Hurry".

Impressions From Agile 2007 - Day 1

Martin Jul — Tue, 14 Aug 2007 00:05:02 GMT

Finally, the biggest agile event of the year is under way! We have a big Danish delegation here this year - other than Ative we have met people from Bankdata, BRFkredit, GoAgile, BestBrains, NDS, Systematics here.

Agile Metrics

The first day of the conference was a half-day with only a Research-in-progress track in the morning. One session focused on metrics for agile.

There was a big discussion about the usefulness of metrics. It was clear from peoples experience that good metrics can be useful but using the wrong metrics can be devastating. To this William Krebs of IBM listed 5 Deadly Sins of metrics - metrics that had a lot of overhead to report, the scorecards with traffic lights (people tend to game them to make everything "green"), not fixing the problems that were identified by the metrics, using an inconsistent format so that data could not be compared, and forcing a process on the team.

Coni Tartaglia and Prasad Ramnath presented a list of metrics that their team at Primavera Systems had decided to use for their project. As part of their Retrospective they had decided on a number of metrics called their "Sprint Build Inspection".

To measure "Coded to Standards" they used FxCop (for .NET) and FindBugs (for Java). These are static analysis tools that spot common problems in code such as naming conventions, performance and design issues etc.

To measure the unit tests they used code coverage. It was interesting to see that they defined "good enough" as a coverage of 56% or more. However they mostly used this metric to track how they were progressing towards better coverage as they were developing from a legacy starting point.

For "functionally tested" they mandated an automated test for each requirement and reported on the number of automated tests. They segmented these into areas - manual (they had gone from many to none), FIT tests, Silk tests and other tests.

For measuring if the application was "Integrated" they reported on the percentage of unit tests passing, the percentage of FIT tests passing, the percentage of Silk tests, manual and other tests passing. This was used to gauge the quality of the application.

Additionally, they measured "number of priority 1 defects". The team had made a working agreement and decided that at the end of each sprint no priority one defects were allowed, and during the sprint a maximum of 3 priority one defects were allowed per team and no more than 40 total. Also, no bugs older than 30 days were allowed.

Finally they used an interesting qualitative approach to overall shippability, namely asking the team to assess whether they would be able to ship the application in 30 days or not. This would be the lithmus test of the overall quality.

After this we had a big discussion in groups and people also named a number of other useful metrics like not allowing code with higher cyclomatic complexity than 10 (to eliminate the worst design issues), measuring compliance with various internal and external standards (go/no-go) and to measure positive approval from the customer (some would say that would be the ultimate metric!), and also to inspect and adapt the set of metrics on the way, adding and dropping metrics as needed.

Experience Reports - Implementing Agile in the Enterprise

The really big case study here was Chris Fry and Steve Greene from Salesforce.com. Their case: converting a 200+ person development organisation from waterfall to agile in three months. Yes, three months. They didn't like the way their velocity and release frequency had dropped and bet everything on fixing it quickly.

As Chris Fry noted - there's never a good time to convert to agile, so just pick a bad time and go.

Here's how they did it: they created created a cross-functional roll-out team with the leads of all the involved silos to oversee the transformation project - development, QA etc.

Then, everybody got a copy of Ken Schwabers Scrum book and over a two-week period, everybody received a two-hour crash course on agile development.

Then they converted the organisation into 30 Scrum teams and let go. The company had a 30-day sprint cycle, but teams were allowed to subdivide these sprints into 1 or 2-weeks sprints if they liked.

Additionally, the sent 30 people from the development organsation for ScrumMaster certification and 35 product owners were also certified to ensure buy-in from the business side. One of they key findings was that they should have started with training the product owners earlier. This is also our experience.

Some of their key findings

The Product Owners should be trained first (ScrumMasters can be trained later)
They would like to have used more open-space style events with the individual team members to get more buy-in from the people doing the work faster.
They should have introduced outside coaching help earlier.
They gave key executives deliverables for the roll-out to assure their buy-in.
As they moved to self-organising teams the had very clear rules about the "Done"-ness. This was not up to the individual teams. Instead they set a corporate standard for QA, documentation, test etc.

The listed a number of keys to success

Get executive commitment - this also means that shipping dates are never negitiable. The sprint length and shippability can never be compromised.
Focus on the principles - not the mechanis. For example, some teams did not like the Daily Scrum agenda, so they allowed them to make their own agenda as long as they kept meeting daily to communicate about the project and share important information.
Also, they focused a lot on automation. Continuous integration with daily build and test was put in place and bugs were treated as stop-the-line issues.
Finally they adviced to provide radical transparency - in fact, they sent out a company-wide email with key daily metrics every day to provide a heartbeat for the transition.

Finally they offered some advice for people embarking on a similar transformation project:

Set up a dedicated cross-functional roll-out team to guide the transition
Get professional help - outside coaches.
Focus on moving the average teams up - get several teams to excellence. Don't waste your time on the high and low performers but focus on getting the bulk to a high level.
Provide a heartbeat in the form of a daily metric to everyone.
Decide on proper tooling early - for their organisation using Excel for backlog/burndown did not quite scale.
Encourage visibility and over-communicate. People need to hear the agile principles again and again to avoid fallbacks.
Experiment, be patient and expect mistakes. Just fix them quickly as they arise.
And don't take the slow way - go all in, and convert the whole organisation in one shot.

Meet us at Agile 2007

Martin Jul — Mon, 06 Aug 2007 21:33:00 GMT

We're off to Agile 2007 next week where I will be hosting a Discovery Session on how to go about implementing agile practises. It is called "Corporate Judo - Guerilla Tactics for Agile Transition". If you are interested in this topic and would like to share your experience to identify patterns for going agile, please join us there. It is on Wednesday at 16:00 in Meeting Room 2.

Here's the abstract:

How do we become better change agents? In the agile community we have all "seen the light." We know all the things that are broken in traditional IT organizations. We also know all the solutions. Yet for some reason we are not very efficient at making the transition happen - and even when we manage to change things for the better the organizations often revert to their old ways when we leave. In this workshop, we will work in groups and draw on our experience to identify the best "corporate judo moves": those that make the agile change happen, and make it stick.

Also, we would like are to meet our blog friends while we are there. If you are interested, please contact me at mj@ative.dk.

Planning Poker

Jan Elbæk — Thu, 02 Aug 2007 20:02:00 GMT

Estimation and planning is an essential part of development, even for agile projects. We have all seen lots of worthless plans, that many in fact that we are tempted to throw planning out altogether. Estimation does not need to be boring, it does not need to be that inaccurate. It can actually be kind of fun...

In his excellent book Agile Estimating and Planning Mike Cohn discusses the philosophy of agile estimating and planning and shows you how to get the job done.

In this post I will mention Planning Poker (coined by James Grenning - Planning Poker) as a simple but very effective technique. The rules are few and simple:

Each team member gets a deck of estimation cards.
The product owner presents one user story at a time.
The product owner answers any questions the team might have.
Each team member selects a card representing his estimate.
When everybody is ready with an estimate, all cards are presented simultaneously.
If estimates differ, the high and low estimators defend their estimates.
The group briefly debates the arguments (time boxed).
A new round of estimation is made.
Continue until consensus has been reached.

I have used Planning poker for sprint planning in scrum teams - but also in release planning when management required long term planning 6-9 months ahead. Using planning poker we had enough accuracy and did not use lots of time trying to detail tasks up front that nobody knew much about - and that perhaps would be replaced by more important stuff before being implemented.

So what you need is some planning poker cards to get started doing estimation the agile way. Contact Ative and get a deck of cards.

A full deck of Ative card is 13 cards spanning from 0 to 100 and 2 special cards:

"?" = "I have absolutely no idea at all" (to many of these played tells us that the user story most likely is not ready for the current sprint)

"Coffee" = "I need a short break. I'm too tired to think"