Matt started with a calculation from CAST 2010. Someone explained that if you automate a test, you can have it run it ten thousand times easily. If you run it in a continuous integration system, then you can run the same test 50 times par day, thereby saving lots of money. Matt explained that he doesn’t believe such easy calculations.
Matt stated that he worked with SocialText for three years. They built there the leading Selenium test automation framework. Matt presented four different variables which can help us model test automation: Cast, features, defects and release timing. With the skills he has to do, he can have a discussion based on these four factors.
Matt explained on his model that in the beginning test automation on most projects is going to slow us down. Costs go up, features are dropped. In medium term the argument is that we are going to find more defects with the tools. In general there are two approaches to test automation. One is to describe exactly how the software is going to look like. This means that every time something changes in the interface, you have to adapt that test. On the other hand you can use keywords to abstract from that. But then the tester has to spent more time on maintenance of those keywords.
Matt suggested to go back to our organizations and apply the model he provided based on costs, features, defects, and release timing. To Matt test automation based upon domain-specific languages makes sense. But we should keep in mind that test automation is a mean. Matt suggested that automation should drive the application to a starting point for meaningful manual testing to set in.
When you find bugs in your iteration, you should reflect upon them, and seek to find different strategies to tackle that risk. Matt concluded his talk with this.