🧪 Testing

Updated at 2021-03-02 09:01

This note is about testing in software development.

Testing Types

Test types:

Unit tests test a single function or functionality.
Service tests test a collection of features without any UI and consumer centric.
End-to-end tests can be slow and are ran more frequently as part of final CI verification pipeline, gives higher confidence that the product works.

You'd usually have on the scale of:

* 5000 unit tests
* 100 service tests
* 5 end-to-end tests

All testing types provide value but the cost varies:

Higher in the stack, the bigger the cost.
Higher in the stack, the further away we are from the data.
Lower in the stack, the closer we are at the data.
Lower in the stack, the less we test at once so less room for errors.

Unit tests should be blazing fast and technology centric. Slow unit tests will cause developers to run them less, introducing more bugs.

Unit tests makes development faster. Developers do not need to manually try out every possible scenario after changes. Thus developers can add new and improve existing functionality easier.

Unit tests reduces time bugs stay hidden. Tests allow developers see if their new change breaks anything related. The sooner bug is found, the less time it takes to fix it.

Bug found during...
    Development     (cost to fix a bug: 1) (the imaginary base value)
    =>
    Automated Tests (cost to fix a bug: 2)
    =>
    Local Testing   (cost to fix a bug: 4)
    =>
    QA              (cost to fix a bug: 8)
    =>
    Prodction       (cost to fix a bug: 16)

Unit test coverage matters. Test coverage means how big portion of your scenarios are being tested. In general, you should have at least 95% of your code tested. Any less than that will not help you detect errors after refactoring and code changes. Avoid overlap in test coverage. There always be some overlap but you should avoid testing the same code paths multiple times.

Tests help using the related code. Tests document the code, helping in collaboration. For example, unit tests work also as usage examples for other developers.

When an end-to-end test fails, write a regression unit or service test for it. You don't want to rely on those slow tests while developing, they are for extra confidence.

End-to-end tests are journeys. For example "create a new user, order an item, check status, mark as delivered, done".

Performance tests are important but can come a bit later. Using production data volumes in quality assurance environment etc. You can alternatively record durations of long running operations to somehow cover this.

Writing Tests

Bottom-up testing is more cost-efficient. You should start testing as close to the data as possible and move up from there. The less area you cover in each test, the more accurate your tests will be showing what is wrong. If you build higher level tests on top of code base without unit tests; it's frequently hard to tell where the execution went wrong.

Fail first, fail fast. Write tests before the implementation so it fails on the spot you expect it to. You want to see tests fail; this tells that tests are working. This reduces bugs in the tests themselves.

Use mock implementations if needed. If you find yourself writing a lot of stub functions and explicit return values of mock calls, it's worth looking into writing a mock implementation. Such as in-memory blob storage in-stead of S3 calls.

Don't be afraid to use files as input and expected response. Two files with one containing a HTTP response and another the expected response to be tested can work quite well. But use only if needed as there are plenty of frameworks available.

Always write regression tests. If you find a bug and fix it, always write a test so it cannot happen again. Even if you are the only developer that saw it while working on your branch.

Don't overengineer your tests. Example of a good unit test:

We have a sorting algorithm `sort`.

Test sort(3 1 4 1 5 9).

Do NOT check that it contains each of the original numbers.
Do NOT check that an index is smaller number than the next index.
Check that it is exactly [1 1 3 4 5 9].

Always remove or fix flaky tests. Errors that come and go cause error blindness. All failures should indicate no-go and should be fixed.

Do you get thanked if you remove a test?
No.
But you sure will be blamed if it lets bug through.
Annoying.

Running Tests

Automate everything. You should have a framework that runs your tests and show results. Running tests should be automatic or done with one button press. When you push your code to version control, there should be automatic verification at that end that your code passes all tests.

Avoid running the application to check if your code works. When developing, running tests should be your only way to validate that your code works. After you feel like you are done with the feature, you can try it by running the application, but tests should your main way to develop.

CI/CD/CD

Terminology:

Continuous Integration (CI) means frequently merging your changes to your main branch supported by automated testing.
Continuous Delivery (CD) means that releasing to production one click way if your automated tests pass.
Continuous Deployment is supercharged CD where each change is automatically released if it passes tests.

CD aims to optimize efficiency to release new production versions. For organizations this means releasing faster and more frequently.

Test-driven Development

Three Laws of Test-driven Development:

Write a failing test before the implementation.
Only test aspects concerning the task at hand; nothing extra.
It's good enough when the implementation passes the test.

Keep your test code clean. Test code should be cleaner than the implementation code. Otherwise your tests will be costly to maintain and reduce the whole benefit.

Don't apply TDD for the first time in a real project. Make sure each member of the team knows basic concept of testing, like mocks, stubs and spys. Organize a workshop to learn TDD, usually takes a day to learn the basic concepts.

TDD affects your software design, as it should. Many, myself included, first mention the bug catching benefits of TDD. But the most important aspect of TDD is how it affects your application design. You start to form your applications from components that have as little dependencies as possible.

TDD is all or nothing. Make sure all developers agree to do TDD. It takes one developer to ruin many major benefits of TDD. Pair programming helps teams to adopt TDD, makes harder to cut corners when deadline is close.

Integrating TDD usually has three stages: