Here’s some thoughts on Google+ hangout series on Is TDD dead? by David Heinemeier Hannson (Ruby on Rails), Martin Fowler (do I have to list anything?) and Kent Beck (author of the book TDD by example). I’d like to provide a short summary of their opinions and discuss the key points in regard to other experiences and some empirical data.
For those who are on a short coffee break: Of course TDD is not dead, Heinemeier Hanson just likes to code like in the good’ol days of wizard programming ;-).
Short Summary of Beck, Heinermeier Hannson and Fowlers points
Kent Beck, author of TDD by example, obviously propagates TDD. He explains how it fits his way of tackling complex problems and how it helps to overcome anxiety when facing complicated tasks. A developer is allowed to have confidence that his code will work and he claims TDD provides just that. The question of too much tests and test to production code ratio is interesting, as he explains why the ratio depends on the type of project and basically on the amount of coupling between components. Beck points out that deleting tests is not only valid but also necessary. This is also already mentioned in his book.
The corner stone of TDD is that breaking problems into smaller problems always yields a next step, some achievable task at hand. Chop up a hard task until it’s easy to write a test, come up with a solution, then refactor to have a SOLID foundation for the next layer. (OK, I made the SOLID up myself 😉 ).
Fun quote in response to Heinemeier Hannsons claim that unit-testing always leads to bad, layer-bloated design (Martin Fowler laughing really hard):
David, you know a lot more about driving a car than me. But here in Oregon, if you get out of your car and you are some place that you don’t wanna be, getting a new car is not gonna fix that.
I have to remember that one ;-).
David Heinemeier Hansen states with firm conviction that TDD does not fit his way of work – he does simply not like TDD. He claims that code written by TDD is not better than conventionally tested code, often even worse because of the use of Mocks. He presents a bad example of code, probably derived by TDD. That’s where Beck responds with the fun quote. In the last episode he argues that because it is so difficult to scientifically prove the superiority of one approach over another we shouldn’t even try.
He points that he once tried TDD but when it came to web programming he found it just didn’t work for him. He firmly relies on the regression test suite, though. To summarize, he rejects Unit-Testing more or less as a whole which includes TDD.
Martin Fowler points out that he likes to have self-testing code that can be shown to work with the press of a button. He doesn’t mind where it comes from, which does include TDD. He says that getting the right amount of test coverage for a person individually and a team is a calibration process – you have to find the patterns where you screw up and write a test for them next time. That may lead to an overshoot when developers tend cover each feature. He also points out that it is always valid to reject practices that do not fit and one should not apply practices without reflecting on them – self-evident but still important.
I must admit that I find it extremely hard to listen to Heinemeier Hannson. His way of argumentation is very subjective without leaving too much room for other opinions. Every statement – or should I call it rant? – serves as a kind of (lame) excuse for his way of work which he claims is as just as good or even superior to other approaches. It sometimes feels like Fowler and Beck are discussing with a Creationist about Evolution. He is probably very convincing to those who do not have a (computer) science background and where anxiety of leaving the good’ol unstructured trial and error process behind for something unknown might cause one to fail. I hope that my discomfort with the personality does not influence the conclusions too much.
A matter of discipline
Going with Heinemeiner Hannson, from what I understood is, if the problem is too hard, when it’s to boring to figure out the details, it is all right to go with trial-and-error. Yes, doing HTTP/HTML/Database stuff can be boring some time and you want quick results. But with an approach like TDD finding the right abstractions can be both fun while still being disciplined engineering. Heinemeier Hannson states that the ActiveRecord pattern cannot be well unit-tested in isolation. But bad testability is just an indicator that ActiveRecord is probably just an anti-pattern.
Heinemeier Hannson claims that TDD leads to too many tests (“overtesting”) which makes it hard to refactor. I’ve seen this happen without using TDD going to extremes, so, again this is not related to TDD but related to coverage requirements (“more than 80% unit-test coverage”). Some CI Test-Coverage Plugins require the default public no-arg constructor in a class full of static helper methods to be untested. A class of helper methods should trigger an architecture alert by itself, but to get 100% test coverage you can be sure that some developer is writing a test to make the warning go away. Or he creates a private no-arg constructor. I don’t know what’s worse.
He also falls for a common misunderstanding. Having good black box tests does not make white box tests irrelevant. I don’t mean those beginner’s white box unit-tests where they peek around in objects using reflection to do state verification, but a minimal setup that puts the tested component in the center of attention.
Imagine the development of a car. Even if you test a car under all circumstances in the real world this does not eliminate the need to test the engine, the gear and the electronics separately (and the subcomponents they are made of). After that you will want to test various combinations of integration setups. Even if there were only one type of engine and gear, it would simply be a waste of resources to only to high-level black box testing because most problems would have shown earlier in a much cheaper test. Additionally, testing error cases or distater recovery is often just not possible without a special setup. Wasting high-end gears to test your high-end engines in a high-end chassis on a desert race-track is nothing you want to do on a regular basis, if only because it takes too long.
Maintaining the developer test suite(s) is as much as hard a problem as maintaining the production code. It may be viable to even have a separate role for that, something like a Test-Code Manager or Archtitect. I find myself sometimes in a situation where I discover that somebody else already wrote a test that I could have extended or adapted and refactored. It was just in an unexpected class, had a misleading name or I was just too lazy to search for it. In general I found, test code quality is often way below production code quality. In part this is because writing tests in some circumstances (like web applications) is more a chore than the fun part. You would have to refactor big parts of the application or even drop a bad framework to make the test look good. Hardly anybody does that.
Where you would reject a 50 line production code method, developers easily get away with a 50 line @Test method that mainly replicates the fixture and verification logic from the previous @Test method. By writing tests first, test quality usually also improves as it isn’t done in a hurry to meet some metrics. When tests are considered first class citizens developers tend to take greater care to make them as good as they can.
Empirical background on TDD
The software industry lives off its unproved best practices. That’s why Heinemeier Hansen can claim that TDD is dead – there is simply little empirical evidence if TDD is better. But there are some studies that at least are close enough to back up the benefits. Continuous Testing, that is, automatically executing unit-tests after each change in the IDE was actually empirically studied, some background information can be found here. Reducing the time between introducing an error or misconception and fixing it seems fundamental to me. As TDD per se can speed up the cycle from minutes (or hours) to a few seconds it is valid to assume that TDD done right is an improvement over test-last approaches.
Bridging the gap between exploration and engineering
Sometimes is the need to explore systems, libraries and frameworks, one answer is already in TDD by example: In the chapter Red Bar Patterns there is the Learning Test. Write a test for externally produced software. This not the only way. Visualization by writing a simple UI is good start. But to verify this knowledge, just create a test.
As an example, setting the read-only flag using the DOS view on a folder in Windows does not make the folder unwritable. But removing the permission WRITE_DATA for the current user or group does. Here’s where TDD kicks in – state what you expect – but don’t expect that what you state is necessarily correct.
There are certainly some points from the 3hrs+ videos that I dropped or missed out. The talk clearly showed to me that TDD is not dead. Given a focus on quality and longevity of a software product it is more alive than ever. It delivers excellent results in the engineering categories of correctness, robustness, maintainability and extensibility. It is a natural barrier towards rushing a system into production that is just not well designed. In the aircraft and space industry there is a saying:
We do not build anything until we know how to test it.
If you ever worked in the industry you know they mean every single unit and every little component down to a screw.
Classical engineering is also one of the main sources where software engineers can gather knowledge from. There’s 150+ years of experience, from building cars, to power plants to space ships. If you are a serious software engineer you will want to benefit from this experience.
Why do we write tests in the first place?
We want our software to be a delight for the customer. It deploy without error, it should start without an error we could have prevented and it should help the customer doing what he wants it to do. If there is some undesired situation, like disk full, low memory or an unreachable server, we want to point out as precisely as possible to the stakeholders that are interested what happened at an appropriate level of detail. And obviously, stack overflows, off-by-one errors or null references should not occur in a final product. Because we designed it’s contract carefully and tested it. Designing the contract becomes much easier with TDD because for a short time we can change our perspective from Provider to Client.
Some approaches on the other hand, like the one Heinemeier Hansen describes as his own, are hardly engineering. Resisting a disciplined process because it’s not fun (or it shows your design is no good) sounds a bit like handicraft work to me. You simply cannot rely on it. You cannot promise a customer that it takes 12 person days to implement a feature when there was some self-proclaimed code-wizard at work before.
The major personal driver of spreading TDD for me is that my work gets easier when I can rely on results yielded by a disciplined apporach. It happened and still happens too often that I have to work with code that is just not finished, there are some alibi unit-tests but they only cover the happy-path and a few obvious error cases. Much more often than not, I run into the untested or bad designed parts. I don’t like that. It’s not fun. You might have had fun developing that code. I have to clean up the mess. I have to go to our customer and break our promises. Who wants that?
That’s why TDD is not dead, but is still the Future. It’s alife and kickin’. On the other hand, despite it’s undisputed impact on web development in genral, Ruby-on-Rails is dead. It showed that it does not scale, is hard-to-test, slow and hard-to-maintain. I call it the Deus-Ex-Machina or “Where the hell does does that method come from?” problem. Ironically, with TDD, the design of Ruby-on-Rails might have been up to the challenge.