Testing and Mocks

Posted on Sep 28, 2020

Let’s just start this by saying that mocks are useful in certain situations. Use of mocks isn’t a problem. Poor use of mocks is a problem.

There are people who like to sound clever and draw distinctions between different types of mock (What’s the difference between faking, mocking, and stubbing?), but there seem no universally accepted distinctions. So when I refer to a mock I am referring to any one of those definitions.

There are also people who like click-bait rules, like never use mocks1, or mocking is a code smell. But in general, apart from the rule about not using goto2 discretion is the key.

As I see it, the three main problems associated with using mocks are:

  1. They couple the tests to the implementation of the code under test.
  2. They test assumptions about how a component works, not how it actually works.
  3. Setting up and asserting on mocks can be overly verbose and maintaining and understanding those test can be genuinely problematic.

Let’s look at these in turn.

Coupling

They couple the tests to the implementation of the code under test.

There is often a lot more test code than production code; that’s how it should be for code that is tested at any rate. I almost always spend a lot more time writing tests for something than I do working on the code itself. So when you’ve decided to mock out all of the interfaces / abstract classes in the code under test, you end up heavily invested in the signature of the mocks. A small change in that signature can result in a large change to the test code. This creates a significant disincentive to modify that signature. And with this disincentive comes an incentive to hack into the production code to protect the tests. The production code becomes more convoluted than it needs to be because no one wants to deal with those mocks.

Remember the great promise of testing is that testing enables change. And it does. However poor use of mocks can have the opposite effect. There is a fantastic quote by Ayende Rahien, author of RhinoMocks, one of the original C# mocking libraries.

Mocks are something that I would generally use nowadays only for external components. Stuff that is entirely out of my control. I would mock the CreditCard company service, for example. For internal stuff, I don’t do that. Testing object interaction independently from the system only means that you are setting in stone those interactions. If you need to change them in some way, that is going to come back and bite you later on. Consider a scenario in which we change the implementation to increase performance, without changing observable behavior. With mocks, tests will usually break because of that.

This isn’t enabling change, this is shackling it with iron chains. [Emphasis added]

I had worked in a team where they considered their mocking library on a par with The Queen (beyond reproach). And, despite thinking they had many things wrong, mocking seemed to me a wonderful black magic for some time after my initial encounter with it. However, over time things were just not sitting correctly for me. I could tell there were problems, but I had fallen under the spell of the the mocking mantra. It was only when I read this that my thoughts finally dropped into place.

Assumed Behaviour

They test assumptions about how a particular component works, not how it actually works.

This comes under the category of the tests all work, but there’s a bug in production. Why? Well because your assumptions were wrong, or were not sophisticated enough.

Assumptions is the mother of all failures.

Assertion against the code that runs in production requires less assumptions and so is more likely to find defects. In my personal experience this has been borne out many times. Nothing beats a big system level test for catching bugs. The more granular the test the easier the bug is to isolate. The more broad the test the more bugs are likely to be found.

On one project I worked on (sizeable and numerically complex), we had a comprehensive unit test suite and two huge smoke tests that hit most of a fully wired system. Those smoke tests were worth their weight in gold.

Complexity

False gurus like Uncle Bob try hard to spread this idea that things under test need to be forced into total isolation. I’m not even going to start on his crazy ideas about methods never being longer than 2 lines or whatever it is, although he has made some valid points about naming. The issue is that to get this level of isolation you either end up with exponential class / interface growth to keep things small and isolated, which makes finding anything in a code base like searching for a grain of rice in the Sahara, or you end up with test methods that are 95% calls to the mocking framework.

I have a lot of experience of reading and understanding code I’ve not written. And I’ve got a lot of experience of reading code written by people of varying abilities. So generally I’m not phased by much, but when I get to a test that is 30+ lines long and almost all of that is lambda methods and mock calls I do start questioning if some programmers actually think at all about what they write.

When Mocks are Good

There are times when using a mock is the right choice:

  • When testing a component that accesses some external service. That’s not to say you shouldn’t have tests that use the external service if it has a test endpoint. Those are great tests. But here is a situation that you probably do need to make some assumptions.
  • When the production component needs to call through to a UI component, for instance a file save dialog. That’s an external service, but not necessarily an obvious one. UI testing is one of those things I’m still on the fence on. I like the idea in principle, but I’m yet to see a solution (for WPF at least) that actually works, is maintainable and isn’t a huge time sink.
  • When testing against the real component is really (time) expensive. It’s a rare joy to work on corporate software that doesn’t have a database, however it is often an expensive task to deploy a new database and to execute tests against it. Again these are great tests and you need them, but if having a full suite of database tests is causing your build to take ages then you are probably better off by doing a smaller number test against the database itself and then assuming that it works in your other tests. Never underestimate the importance of a fast build.
  • When you don’t use anything but an object itself (you don’t use any of its members), then using a mock could simplify your testing.

In general, the rule I apply is that when I evaluate using a mock against the disadvantages that are inherent in mocking and the balance of the argument comes out in favour of the mock, then I will use one3.


  1. That particular post seems to say don’t use mocks, but use something similar that probably has worse tooling. ↩︎

  2. The goto keyword is almost without exception only ever to be used by code generators. Generated code is not usually meant to be read and goto can be a more succinct way of doing things like state transitions. ↩︎

  3. Maybe that’s a bit of a cop out because that’s pretty much the rule I apply to all the code I write. Be wary of movements and thought leaders. Go to the source, read code, write code, be critical of the code you write and look for the strengths in the code written by others. Form your own opinions based on arguments that make sense to you and don’t be afraid to change those opinions when you hear better arguments than your own.

    The problem with the world is that the intelligent people are full of doubts, while the stupid ones are full of confidence.

    Wherever the crowd goes, run the other direction. They’re always wrong.

    ― Charles Bukowski

    I do, however, pay careful attention to what good programmers say. For example when Eric Lippert has an opinion different to my own that is as good an indication as any that I’m wrong. ↩︎