Increasing integration test quality
Should we hand-write mock objects or read API responses from disk? Reviewing the tradeoffs when storing response payloads as part of test suites.
I find myself conflicted when facing a common situation encountered when writing integration tests.
I was testing services code which called nba.com’s API and returned a JSON payload with 300 game events. The question comes from Line 4 of this code:
describe('LineupService', () => {
let boxScore: BoxScore;
beforeEach(() => {
boxScore = // read from file or create a mock object?
});
it ('returns away team starting lineup', () => {
const lineups = new LineupService(boxScore).getAwayLineups(boxScore);
expect(lineups.length).toBe(5);
});
});
How should I create the boxScore
object:
Should it be hand-written where I construct it using object instantiation?
Should it read a sample JSON payload returned by the service from disk?
There are pros and cons of both approaches:
Hand Written
Pros
Minimal object containing only what the tests needs
Self-contained test; don’t need to open any other file
Super fast, avoids disk reads
Cons
Doesn’t resemble a real payload returned by service
Can make copy-paste errors when constructing objects for other tests
Makes assumptions on how deserializer will handle empty/null values
Read from Disk
Pros
Higher confidence that the code works as it’s acting on real-world data
Better conversion of empty/null values as deserializer used in test code can be made similar to what’s in production
Can reduce the need for end-to-end testing as real payloads are used
Cons
Duplication in maintaining payloads for different test cases which only differ slightly
The test file has a dependency on a file which needs to be organized, named properly with a clear mapping to the test
Slower as disk read is needed; can be problematic for large test suites
As with most things in software development there is no right or wrong, only trade-offs. Here are things I consider when deciding which option to use:
How big is the API’s state space? If it’s an API that has many optional fields then hand-constructing the object can be risky.
How typed is the API? If we’re returning numbers as strings then we introduce a risk during deserialization of constructing the object incorrectly.
Do I control the API? If I control the API, then I have greater confidence that I won’t mess something up while hand-writing.
What’s the cost of generating responses? If creating sample responses is expensive or time-consuming, it may be better to hand-write.
How confident am I of hand-modifying payloads to test edge cases? Not all payloads will be available, so you’ll have to modify some by hand.
Can I generate the payloads myself or do I rely on someone else to do it? Adding a dependency should always be a thoughtful decision.
Can the payload be used to collaborate with other team members? There are benefits of having a common language we all understand.
Lately I have found myself favouring storing the payload on disk for a few reasons:
It helps reduce end-to-end testing efforts by giving greater confidence that the code works against real-world payloads. I see little value in end-to-end tests.
It enables better collaboration between and within teams because the ubiquity of Chrome Inspector and other proxy tooling. For example, people using the UI can provide a “failing test case” by capturing the payload.
Edge cases can be simulated with greater confidence by hand-modifying payloads.
If you got this far I hope you found this post valuable. If there are aspects that I missed which can be useful, do let me know.