If I explain myself or add nuance, it won’t be a “hot take” anymore, but here goes…
I’ve mostly worked on small teams, but have maintained fairly large codebases (I think the largest single repo I’ve worked in was about 1.5 million LoC). I’ve also maintained software used by other teams, though in fairly small software organizations (under 100 developers total).
I’ve written code with 100% unit test coverage, and code with almost no coverage. The 100% coverage was in a library that was intended to be core functionality shared across multiple teams.
I definitely agree that they can be useful as both usage examples and as a way to enforce some rules (and consistency) in the API. But I’m not sure I’d go so far as to call that a “spec”, because it’s too easy to make a breaking change that isn’t detected by unit tests. I also feel that mocking is often needed to prevent unit tests from becoming integration tests, but it often hides real errors while excessively limiting the amount of code that’s actually being exercised.
I also think that actual integration tests are often a better demonstration of your API than unit tests are.
I haven’t used any of these methods as much as I’d like to, but I suspect that each of them is strictly more useful than standard unit testing:
contract tests (arguably just a form of unit testing, but not the way I’ve generally seen it done)
property-based tests
mutation tests
preconditions & postconditions (with language support)
Makes a lot of sense. I figure contract tests is more or less what I have been doing then.
I think there is that misconception that unit tests are about validating each line of code and preventing logic bugs. Though obviously you understand that this isn’t just about that, or not at all about that I would argue. Unit tests won’t prevent accidental breaking changes, but you can at least add new tests every time this happen, so you’re at least guaranteed that the next maintainer won’t be doing the same mistake.
In an ideal world we could probably have nothing but integration tests, but in my experience if you only do integration testing you end up working really hard maintaining tests that are prone to break and costly to run. Unit tests are cheap to code and cheap to run, it is a great place to enforce your “contracts” with tests that are theoretically immutable. After those tests are out of the way, you can focus only on the actual interaction between your systems with the more expensive tests.
Anyway, you have a good take I am just blabbering. This is based on my personal experience as someone who only cared integration tests but was later converted by a person much smarter than I am. And then later on by joining a team that exclusively did integration testing, hundred of tests. It was hell (imo), we had to change dozens of tests for every little bit of modification. The rest of the team seemed fine with it though. We rarely shipped bugs but progress was incredibly slow for a module of such low complexity. Testing wasn’t the only issue with that codebase though.
The one part of that that sounds weird to me is needing to change integration tests frequently when changing the code. Were you changing the behavior of existing functionality a lot?
Yes, and the test suites were insane. The program was outputting a lot of data, and we basically asserted on anything and everything for any given integration. I mentioned that testing wasn’t the only issue, well there was a lot of issues. Unfortunately the behaviour changes were requested by the stakeholders and there was no way around it. That being said, had this thing we maintained been properly developed those changes would have been a breeze imo. The actual requirements were very simple.
But anyway, I realize this is maybe an extreme example to paint integration tests negatively, but the point remain. In this scenario, every time we changed a bit of code it broke dozens of integration tests instead of breaking just a relevant integration test, had everything that could have been unit tested been written that way. The integration tests could probably also had been less… exhaustive, but it was probably for the best considering the codebase.
If I explain myself or add nuance, it won’t be a “hot take” anymore, but here goes…
I definitely agree that they can be useful as both usage examples and as a way to enforce some rules (and consistency) in the API. But I’m not sure I’d go so far as to call that a “spec”, because it’s too easy to make a breaking change that isn’t detected by unit tests. I also feel that mocking is often needed to prevent unit tests from becoming integration tests, but it often hides real errors while excessively limiting the amount of code that’s actually being exercised.
I also think that actual integration tests are often a better demonstration of your API than unit tests are.
I haven’t used any of these methods as much as I’d like to, but I suspect that each of them is strictly more useful than standard unit testing:
Makes a lot of sense. I figure contract tests is more or less what I have been doing then.
I think there is that misconception that unit tests are about validating each line of code and preventing logic bugs. Though obviously you understand that this isn’t just about that, or not at all about that I would argue. Unit tests won’t prevent accidental breaking changes, but you can at least add new tests every time this happen, so you’re at least guaranteed that the next maintainer won’t be doing the same mistake.
In an ideal world we could probably have nothing but integration tests, but in my experience if you only do integration testing you end up working really hard maintaining tests that are prone to break and costly to run. Unit tests are cheap to code and cheap to run, it is a great place to enforce your “contracts” with tests that are theoretically immutable. After those tests are out of the way, you can focus only on the actual interaction between your systems with the more expensive tests.
Anyway, you have a good take I am just blabbering. This is based on my personal experience as someone who only cared integration tests but was later converted by a person much smarter than I am. And then later on by joining a team that exclusively did integration testing, hundred of tests. It was hell (imo), we had to change dozens of tests for every little bit of modification. The rest of the team seemed fine with it though. We rarely shipped bugs but progress was incredibly slow for a module of such low complexity. Testing wasn’t the only issue with that codebase though.
The one part of that that sounds weird to me is needing to change integration tests frequently when changing the code. Were you changing the behavior of existing functionality a lot?
Yes, and the test suites were insane. The program was outputting a lot of data, and we basically asserted on anything and everything for any given integration. I mentioned that testing wasn’t the only issue, well there was a lot of issues. Unfortunately the behaviour changes were requested by the stakeholders and there was no way around it. That being said, had this thing we maintained been properly developed those changes would have been a breeze imo. The actual requirements were very simple.
But anyway, I realize this is maybe an extreme example to paint integration tests negatively, but the point remain. In this scenario, every time we changed a bit of code it broke dozens of integration tests instead of breaking just a relevant integration test, had everything that could have been unit tested been written that way. The integration tests could probably also had been less… exhaustive, but it was probably for the best considering the codebase.