Fooled By the Failure to Fail

In Part 2 of my “Predictive Test-Driven Development” series, I talked about how predicting the failure of a test is not always enough. Predicting precisely^[1] how the test will fail helps me stay alert for tests that fail for a different reason, or perhaps don’t fail at all (i.e., they pass). I want to share another example of what can happen when you ignore the Predicted Failure. This happened during one of my “Refactoring to Testable Code” training classes^[2], which again shows that experts make mistakes!

Test-Driving an HTTP Endpoint

In the third session of the class, I demonstrate how to test-drive a new web-based user interface using Spring MVC and the Thymeleaf templating engine. The code I’m working with is a single-player Blackjack game. In preparing for this particular class, I had updated the Spring Boot dependency to the most recent version—something I usually avoid because it’s risky. I felt OK doing it this time, because I used the same version in other projects with no problems^[3].

I start with a few cycles of TDD, where I create Spring MVC tests to drive the creation of code to handle incoming requests. I get to a point where a template is returned to show the state of the game when it starts, triggered by a button (using POST^[4]). I continue to follow the process, first creating a failing test like this:

@Test
public void postToStartGameEndpointIsStatus200Ok() {
    mockMvc.perform(post("/start-game"))
           .andExpect(status().isOk());
}

Here, the test wants to verify a successful response when requesting (via POST) the page at the endpoint /start-game. Before I run the test, I state my Precise Prediction:

This test will fail with a 404 (Not Found) status instead of the desired 200 (OK) status.

This is because there’s no code responding to the endpoint, and Spring MVC will default to returning a 404 in that case. I run the test, and it fails in the predicted way:

java.lang.AssertionError: Status expected:<200> but was:<404>
Expected :200
Actual   :404

My next step is actually not to make the test pass, but to write just enough code to make it fail for a different reason. This is because writing enough code to make it pass does too much in one step. The two steps I want are:

1. No more 404: Spring MVC calls the method I want for the /start-game endpoint, and

2. Return a valid template name that Thymeleaf can find and process.

I want to write just enough code so #1 works, but still fails for #2, because the template file matching the name I return (blackjack) doesn’t yet exist. Once the test fails for reason #2, I can drop in the template file into the templates directory, and finally watch the test pass^[5]. The code I write next (in the @Controller class) is:

@PostMapping("/start-game")
public String startGame() {
    return "blackjack";
}

Now my prediction is:

This test will fail with a Thymeleaf exception, complaining that it can’t find the template named blackjack.

I run the test, and it passes. I am appropriately surprised by this, given my prediction! Note that I have done this exact scenario every time I’ve taught this class, well over 30 times by this point, and it always failed for the expected reason.

Watch Out for the Availability Heuristic

You may have heard of the Availability Heuristic, a cognitive bias that leads us to believe “the correctness of a hypothesis based on how easily that hypothesis…comes to mind”. Well, the most “available” hypothesis that came to mind for me was: “oh no, I shouldn’t have upgraded the Spring Boot dependency, something must have changed, and now it just works”. I ignored the fact that the test didn’t fail, and continued with the next step in the TDD cycle: I copied the Thymeleaf blackjack template into the correct directory, and told the class “now we can run the application to see the generated page”.

You might guess what happened: I got an error page that said:

There was an unexpected error (type=Not Found, status=404).

Now I’m confused. The test passed (when I didn’t expect it to), but at runtime it’s failing in a way I can’t explain. 🤔

At this point, I go into troubleshooting mode: I turn on debug-level logging, rerun the application, and notice a lack of Thymeleaf code anywhere in the output. 💡 I now have a new hypothesis, one grounded in observation (rather than guessing or availability): do I have the Thymeleaf dependency in my project’s Maven file?

Of course, I don’t. 😯 Somehow it got dropped when I initially added Spring Boot dependencies to the project during my demonstration. I added the missing dependency and “rewound” things. I restated the prediction:

The test will fail with a Thymeleaf exception, as it can’t find the blackjack template.

Then ran the test. It now failed for the right reason:

Caused by: org.thymeleaf.exceptions.TemplateInputException: 
  Error resolving template [blackjack], template might not exist
  or might not be accessible by any of the configured Template Resolvers
at org.thymeleaf.engine.TemplateManager.resolveTemplate(TemplateManager.java:869)

With a sigh of relief at the correct failure, I continued on.

Lessons For Everyone

What did I learn?

1. Don’t ignore tests that fail for the wrong reason, or pass unexpectedly. You could be making a wrong assumption. It’s worth the extra time to track down and understand why it didn’t fail as expected.

2. Go back to a known state. I could have changed the version of Spring back to the one I had been using for months to eliminate that as a cause.

3. Don’t update dependencies right before teaching a class. If I hadn’t, I would have searched for other reasons for the problem much earlier, as the upgrade of the dependency would not have been my most “available” hypothesis.

4. Making mistakes is OK (even in front of an audience). Often those mistakes make great content for articles and future classes. I now make this mistake on purpose to show how valuable it is to pay attention to unexpectedly passing tests.