I can see and agree with lots of the points in the article (also huge fan of randomized testing, so much that I’m in the 3rd iteration of my own library for that). But I disagree with a few things or at least see them in a different light.
This is more a critique of the unit/integration split here. People understand it differently but the way I use “unit test” is: If I have a module/component A then unit tests for A test the boundaries of A, the api of A, and the internal helpers of A if it makes sense. Now if I have a module C that depends on A and B internally, then my unit tests for C already test for interactions between A and B, at least in the usage and context of C. I would not stub them by default. So in a way, for a module graph, the way I would use “unit test” is already what might be “integration test” for others: unit tests of a module M already test the interactions of the transitive closure of dependencies. Not exhaustively, but in the way they are used. So my use of “unit test” here already finds lots of nasty module interactions. (I can see that this way is a luxury I have because my work is locally and does not need complex infra)
I can agree that if you only test small localized parts of your software, a lot of hard interaction bugs are invisible. But I also starkly remember a time where localized testing was not done (pre-2010 sounds about right as a whole but there’s enough shops that didn’t get the message ;) ). And in every such project I was one the “hard interaction bugs” were of little concern because the surface software layer was so bug-ridden already. Simplest stuff didn’t work as soon as you moved away from the handful of cases the dev ran. So in a way I agree with “localized unit tests don’t find interaction bugs” but it is also a certain kind of luxury if these are your bigger worry.
Finally, and this relates a bit to 2: I always have to consider such advice in the context of some abstract “fixed engineering resources”. “Just additionally do this resource-intensive extra thing, ideally enlist X customers to preview test, …” is technically increasing the quality but not within a fixed budget. I agree that only doing localized tests and not any integration/randomization/real-user tests on top is usually not ideal, but neither is dropping all unit tests. In this perspective, I firmly believe unit tests to be a low hanging fruit with a lot of bang for buck. The detailed resource allocation is nuanced but I’m willing to defend that most if not all optimal allocations have a big chunk of unit testing in it.
I can see and agree with lots of the points in the article (also huge fan of randomized testing, so much that I’m in the 3rd iteration of my own library for that). But I disagree with a few things or at least see them in a different light.
This is more a critique of the unit/integration split here. People understand it differently but the way I use “unit test” is: If I have a module/component A then unit tests for A test the boundaries of A, the api of A, and the internal helpers of A if it makes sense. Now if I have a module C that depends on A and B internally, then my unit tests for C already test for interactions between A and B, at least in the usage and context of C. I would not stub them by default. So in a way, for a module graph, the way I would use “unit test” is already what might be “integration test” for others: unit tests of a module M already test the interactions of the transitive closure of dependencies. Not exhaustively, but in the way they are used. So my use of “unit test” here already finds lots of nasty module interactions. (I can see that this way is a luxury I have because my work is locally and does not need complex infra)
I can agree that if you only test small localized parts of your software, a lot of hard interaction bugs are invisible. But I also starkly remember a time where localized testing was not done (pre-2010 sounds about right as a whole but there’s enough shops that didn’t get the message ;) ). And in every such project I was one the “hard interaction bugs” were of little concern because the surface software layer was so bug-ridden already. Simplest stuff didn’t work as soon as you moved away from the handful of cases the dev ran. So in a way I agree with “localized unit tests don’t find interaction bugs” but it is also a certain kind of luxury if these are your bigger worry.
Finally, and this relates a bit to 2: I always have to consider such advice in the context of some abstract “fixed engineering resources”. “Just additionally do this resource-intensive extra thing, ideally enlist X customers to preview test, …” is technically increasing the quality but not within a fixed budget. I agree that only doing localized tests and not any integration/randomization/real-user tests on top is usually not ideal, but neither is dropping all unit tests. In this perspective, I firmly believe unit tests to be a low hanging fruit with a lot of bang for buck. The detailed resource allocation is nuanced but I’m willing to defend that most if not all optimal allocations have a big chunk of unit testing in it.
score 1 · canonical