We expect unit tests to verify the correctness of modified logic and help catch unintended changes in application behavior. When developing with AI, it is also essential to check that functionality outside the task's scope remains intact.
Verifying that a new feature works is therefore only part of the job. We also need to check that the change has not broken existing application rules.
What unit tests should protect
Unit tests verify the behavior of individual parts of the code. They also act as a safeguard against regressions, bugs introduced by changing behaviour that previously worked.
This protection is particularly valuable for AI-generated code. AI may fulfill a specific requirement while also changing logic that was supposed to remain intact.
Well-designed tests help define the boundaries of a change:
This is the behaviour we want to change. These rules must continue to hold.
Test according to risk and actual dependencies
We do not need to automatically write a separate test for every getter, setter or simple data class.
A test that merely repeats the implementation may increase the test count without adding much value. It also creates more maintenance work when further changes are made.
A practical rule is:
For a specific change, focus on the modified behaviour and the parts of the application that depend on it. Determine the scope of testing based on the potential impact.
For simple methods without decision logic, a separate unit test may be unnecessary if other tests adequately cover their behaviour.
However, simplicity alone is no reason to assume that AI will not make a mistake. Even a one-line method can determine an order's price or a user's access rights.
Before adding a test, ask three questions:
- What behaviour is changing in this task?
We verify that the change meets the requirements.
- Which rules and dependent components must remain intact?
We look for places where the change could cause a regression.
- Will the test catch a real bug?
We verify expected behaviour rather than simply copying the implementation.
Tests must connect to business logic
The most valuable unit tests protect specific application rules.
For a discount calculation method, for example, checking that it returns a number is not enough. We need to know whether these rules still hold:
the discount does not exceed the permitted limit;
the final price does not fall below zero;
a customer who is not eligible for a discount pays the full price;
the calculation follows the defined rounding rules.
The connection between a method and a business rule is what gives a test its purpose.
The implementation may change. A rule outside the task's intended scope must remain intact.
How AITM approaches this
AITM connects AI development with project context, contracts and dependencies between different parts of the application.
Before implementation, the architect identifies what should change, how the relevant parts are connected and which rules must remain intact. The testing step then uses this information.
The key question is therefore more than whether AI can write a test. It is whether AI knows what the test should protect and why.
Consider a change to DiscountService.calculateDiscount(). If the contract uses dependsOnModules to record that OrderSummary.render() also depends on the discount calculation, the testing step has a reason to check the impact on the order summary as well.
The calculation itself can be checked with a unit test. An integration test may be more appropriate for checking how the calculation works together with the order summary.
Tests are therefore designed around the actual dependencies and risks of the specific change.
Ten tests for one task, two thousand for another
Not every task needs the same scope of testing.
As an illustration, in a project with 32,467 tests, a small change may require running ten relevant tests. A change to shared business logic may require two thousand.
The purpose of targeted testing is to select tests that cover the modified behaviour and its dependencies. This can shorten the feedback cycle and focus verification on the risks of the task.
Broader regression checks or a full test suite still have their place for extensive changes or before a new release.
The number of tests run should reflect the impact of the change.
When a test fails, we need to understand why
A failing test does not always indicate the same problem. We need to distinguish between these situations:
a bug was introduced in the implementation;
the change unintentionally violated an existing business rule;
a new feature intentionally changed the expected behaviour and the test needs updating.
The task requirements, contracts and business logic descriptions provide the context for this assessment.
If a requirement has legitimately changed, the corresponding test must change, too. If AI has violated a rule that should have remained intact, the implementation needs to be fixed.
Simply modifying a test so that it passes again could hide a real bug.
Unit tests are the first layer of protection
Each level of testing has a different role:
Unit tests verify individual parts of the logic.
Integration tests verify how modules and systems work together.
E2E tests check complete user journeys in a running application.
These layers complement one another. Each catches different types of bugs and helps verify a change from a different perspective.
When developing with AI, we therefore check two things: whether the task was completed and whether behaviour outside its intended scope remained intact.
Connecting task requirements, business logic, dependencies and tests is an important part of AITM orchestration.
Eg. unit tests
Learn more at https://www.aitm.cz
#AITM #UnitTesting #SoftwareTesting #AI #SoftwareDevelopment