At a local talk in Hamburg, Kent Beck talked about G Forces in software, and what effects acceleration of the software process has. With regards to Continuous Deployment he talked about scaling up the deployment cycle from annually to a deployment cycle within minutes.
Kent Beck stated that you can have two people arguing rationally, but still saying two completely different things. He explained this by the statement that some say that a QA department is mandatory, while others claim that QA departments hold them back. He explained his motivation for continuous deployment. He noticed changing patterns if you deployed once per year, or deploy once a month, or even once a week. This view helped him to resolve the discussion about the earlier mentioned disagreements.
Beck showed a graph of the deployment cycle that existed in 1990. There were lots of companies deploying there software annually, while the tendency to deploy a new version of the software was few below the weekly, daily, or even hourly deployment cycle. The majority of software deployments happened at that time in the cycle of annual, quarterly or monthly releases.
Beck showed the graph for 2010, which showed a tendency to the shorter deployment cycle. Many websites currently have a weekly deployment cycle. Beck referred to platforms like Facebook, Twitter, Flickr who deploy several times per day. The larger scale the system gets, the more frequently you have to deploy, to keep a stable system, Beck claimed. The shorter feedback cycles in frequent deployments enable teams to localize changes which broke the system faster, since fewer changes piled up until deployment took place.
Beck projected for the next twenty years, that more and more deployments will move to the monthly and weekly deployment. The annual deployments will be the exception at that time. Some of the barriers to the shorter deployment cycle are technical, while others exist in the business. That is why Beck claimed that within three years of time the deployment shift will not be as dramatic. The organization has to change to get in line with the shorter deployment cycles. Annual budget planning is not in sync with this short feedback cycle.
Beck stated that he was asked about the motivation for deploying on a shorter timescale. The reasons he came up with is to operate at scale. For large complicated systems the system can just take small changes. A second motivation is competition. If a vendor can deploy more often than a direct competitor, has an competitive advantage, as that vendor can react to market changes far more quickly. He referred to Google who have build their whole business on the shorter deployment cycle. Another motivation Beck claimed is to keep in scale with the number of people. While this might sound contradictory, the larger the number of people who change the system, it is way safer to have frequent deployment in place, since the amount of intermittent changes that gets into your way before deploying the system. For companies at the early stage frequent deployment provides a rapid feedback to align their business model while they explore the market situation for their business. Many of the techniques Beck explained are way more easier in a start-up situation.
Beck explained that during the transition from one deployment cycle to another, there are some things you start doing, while you stop doing others. For annual to quarterly deployments he mentioned that teams need to start automated acceptance testing, refactoring, continuous integration, and subscription. Change happens at the pace people accept, Beck explained. Changes need a clear goal and vision. When the individual steps are crisis, this leads sooner to exhausting. Referring to the repetitious process of testing, it’s essential to get automated acceptance tests in place, when you move from annual to quarterly deployments. Most of the confidence you need, you get out of automated tests. Referring to refactoring, he mentioned that for quarterly deployments, you can not make all the design decisions up-front. In quarterly deployments there is not enough time any more for a two month design phase in the software development project. On the same level, in three months, you don’t have the time for one month of a manual integration. Closing the feedback loop, you want to notice problems in integrating the software the day they happen, rather than eight months later. On the difference to the business model, Beck referred to a change on the sales perspective. Subscription-based pricing is crucial to keep your customers up-to-date with your rapid software releases, as your customers might accept to be visited by your sales persons, but not every three months.
Beck sketched that the need for shorter deployment cycles shortens the time it takes between an idea how to make money, and finally making money with that idea. In most organizations, even start-ups, it takes a year between the idea and getting money out of it. Referring to Lean software development, he stated that between this is mostly waste. Beck explained that the organizational structure does not scale up to reach shorter deployment cycles. This is where technical changes alone are not enough.
In order to move from quarterly to monthly deployments, he mentioned to start developer testing, stand-up meetings, cards on a wall, and to introduce pay-per-use business models. On the list of things to stop, he had the QA department, multiple deployed versions which you need to maintain in parallel, design documents, change requests, an analysis team, and a build team. Referring to developer testing, the number of defects in a software has to be dramatically low for the shorter deployment cycle. When you are going to deploy monthly, you have to switch between design and implementation way more often. Similarly, hand-offs between departments is just costly. Beck explained that stand-up meetings reduce the amount of meeting overhead. Instead of planning documents, cards on the wall provide a very low overhead planning technique. Since the plan gets updated multiple times per day, the overhead of re-planning needs to get as low as possible. Beck referred to pay-per-use models like cell phones, or stock exchange. The pay-per-use model has a risk in the longer deployment cycle. If you make a mistake on the quarterly deployment cycle, you lose a lot of money with a pay-per-use business model. If you do the same with a monthly deployment cycle, the loss is less dramatic. Money is the best form of feedback. The person giving you the money, gives up something in order to get your service.
Beck explained that installing a QA department might get a company out of their chaotic ways to work. But a QA department brings in distance between the introduction of a defect, and it detection. WIth an organization not caring about quality, a separate QA department may help, but at the sacrifice of some overhead. When going to the shorter deployment cycle, a separate QA department though provides too much overhead. In regard to multiple installed versions, when moving from quarterly to monthly deployments, you can not maintain multiple versions in parallel, as it costs too much overhead. You can not expect to have ten different versions deployed, and get any new work done while maintaining these. The good news is that with monthly deployments no one has to wait more than a month for the new version of the software. If there is a defect in one version, the bug will be fixed within a month, and be shipped to the customer. For most customers this is an acceptable amount of time to wait. Instead of design documents, Beck said that you need to get better communication within the team in place. The big change request process is critical for an annual deployment. If you have a less rigid change process on a monthly cycle, you can get the changes in place within a single month. For the analysis and the separate build team, you can not allow the organizational distance to take over on a monthly cycle. You still needs the skills in place, just as with testers, but you need shorter feedback loops.
When moving to weekly deployments, more pressure is put onto the technical skills and the organizational structure. The organizational distance needs to be closer, and you can allow fewer defects in your software. Therefore you need a two-way data migration, in order to rollback data changes after a faulty deployment. He referred to a client who had a defect database with some of them being a year old. The amount of waste produced by managing the defects got into the way to go for the shorter deployment cycle. In order to get faster, you need to get rid of your defects. Beck claimed that teams deploying weekly need to have zero defects in their development process. He referred to zero defects as being the Inbox Zero of defect management. For weekly deployments, having a weekly planning cycle gets into your way. Referring to keystoning, for two week feature, you need to separate changes on the inside, and changes in which the user sees just after the second deployment. He claimed that it’s skill to learn to develop in that style. He proposed to use a Kanban system where a stack of tasks which are dynamically prioritized. That way you get way more feedback into the system. Beck explained that you also need bootstrap financing in place for weekly deploys. Weekly deployments is an enabler for bootstrap financing.
On the things you should stop with weekly deployments, he claimed that testers need to be embedded completely into the development team. The organizational distance of a test team is too large to keep up the pace. You also need to stop working with release branches, and patching the defects you found. Up-front usability design is also not possible within the time given for a weekly cycle. Venture capital is additionally inefficient for weekly deployments.
Beck explained that he was surprise to find daily deployments work. He initially thought it was a disaster to go home, and think about the things that you broke today. Just as the good deeds come to you back today, the bad deeds will come back on you as well. Working carefully enough to avoid the bad deeds tomorrow is the way to go. Beck explained that it’s much lower stressful to deploy daily – if you do it well. Immunization to roll back changes directly is necessary. Maintaining a second system provides a safety-net to do that. On a daily cycle, you can try out new things using two different user interfaces by applying A/B-testing. On a daily cycle you also need complete automation for your deployment.
The things you stop on a daily deployment cycle, you eliminate the staging pipeline. Instead you work on avoiding bringing in defects in first place, instead of providing a quality gate as a replacement. So, when scaling down to the shorter deployment cycle, you get rid of the separate operations teams. At this point you integrate operations into your project maybe using Dev-Ops to operate your business. You also work on lower latency communication habits. At that time you get rid of stand-up meetings, as the overhead is too high.
When going from daily to hourly deployments, you strive to release every feature. If something on the fully automated compilation and deployment cycle fails, the system rolls down the changes, locks everyone else out of bringing in new changes, and the team gets immediately together to fix the problem. The most relevant metrics to monitor a system on an hourly deployed system are business metrics rather than technical metrics. He mentioned that the most critical number in the amazon dashboard was dollars per second. Since every problem in the system leads to a business problem, focusing the technical staff on the business impacts of their actions, helps them see the larger picture.
Beck said that hourly deployments are not rocket science, but instead all about discipline. Steady progress brings steady results. Beck called out to test the waters for shorter deployment cycles. When asked whether hourly deployments on a web site would confuse users of that site, Beck explained that there are companies who does that. Though, the changes are less more dramatical. The web has changed the recognition of smaller changes. Instead of asking “Oh, google has changed, is that in the manual?”, we find a way to work with this change. When asked about technology changes in hourly deployed system, Beck explained that changing the database is way more gradual way. He referred to twitter’s change from MySQL to Cassandra. They hooked up the new technology, watched what happened. Then they gradually started to put new data into the new database, and watched what happened. Step-by-step they then replaced all reading mechanisms from the old database to the new database, and finally in the end stoping to write data into the old database, just using the new database at that time. Beck explained also that you build pair programming into the process in order to have each changed reviewed, and see that it fits to the larger picture of the architecture and the design. These control mechanisms help you to deploy on the hourly cycle.
Beck claimed that 2.2e-27 is the most important number in software development. It refers to how much a single electron costs. The company finding out how to get an advantage to write programs which costs fewer electrons, will have a dramatic competitive advantage. Beck explained that at the time he started to work on software, computers where expensive, and programmers were relatively cheap. Later, computation turned cheaper, while programmers where relatively expensive. Reducing the amount of energy your programs use is the next frontier in software development. He explained that being able to put a money number on deciding about an alternative design, provides a tremendous advantage.
Marcus,
Excellent write up. I heard Kent give a version of that same presentation a couple weeks ago at STPCon (where he gave it as a keynote presentation) and had the good fortune to eat lunch with him, his wife, and Catherine Powell. I’ve thought about his G-forces presentation since then quite a bit.
One of the things that impressed me with his talk is that he avoided “the two easy extremes.” By that I mean, it is relatively easy to create a presentation that says “this is the one true way to test; if you do anything else, you’re making a mistake.” Similarly, it is relatively easy to deliver a presentation that says “Everything is context dependent; whether or not you should do X, Y, or Z, or if A, B, or C are likely to be true depends on multiple factors, including…” I think Kent’s talk does an impressive job at describing multiple different contexts that exist (through the lens of varying amounts of time between deployments), and putting forth his general guidelines about what kinds of changes should be made when an organization shifts from one context to the next. That feat is easier said than done. Regardless of whether you agree with everything he’s saying, Kent’s one hour presentation pulled it off quite well and gave listeners a great deal of food for thought.
SLIDES –> Adam Goucher helpfully posted a link on Twitter to an earlier version of Kent’s slides for this talk. For anyone interested, they can be found
– Justin Hunter
Thanks for the comment, Justin. Indeed, Kent put his advice into some context, so that it’s not context-lacking advice. Instead he used a variation of a pattern format, and stated simply things he observed. I spent the whole week with the man, and he talked often about these when referring to Responsive Design, TDD, or Lean Start-ups.
Thanks for the hint to the slidedeck on slideshare.