Skip to content

Capability 011: Test Kinds and Coverage

Status: Accepted

Context

A developer verifies an artifact with automated tests that run from the IDE and in the pipeline. They come in three kinds:

  • A Unit Test checks the smallest unit, such as one function, often without a mock.
  • A Component Test checks units working together, such as a use case, with mocks at the layer boundaries.
  • An Integrative Test adds one real dependency, such as a database in a container. The integration test of a deployed image is not one of them.

A test is programmed (classic JUnit, Catch2, Testify, etc.), or it is a scenario in a feature file (Gherkin language). Both forms suit every kind, that is to say, unit and component test can be either programmed classically or formulated as Gherkin tests. The scenarios of a generated repository are Component Tests, as these more complex tests are more readable in the form of Gherkin tests.

make coverage reports one figure for the code that Unit and Component Tests reach, in every form. This capability defines the kinds, their names, the targets that run them, and the figure. How each language measures it is the subject of a capability per language.

References

Details

Names of the test kinds:

Language Unit Test Component Test Integrative Test Scenario runner
C++ *_test.cpp *_ct_test.cpp, Catch2 tag [ct] *_it_test.cpp, Catch2 tag [it] pytest-bdd in test/bdd/
Go *_test.go *_ct_test.go *_it_test.go, build constraint integrative Godog in test/bdd/
Java *Test.java *CT.java, JUnit tag CT *IT.java, JUnit tag IT Cucumber suite without either suffix
TypeScript *.test.ts, *.test.tsx *.ct.test.ts, *.ct.test.tsx *.it.test.ts, *.it.test.tsx playwright-bdd in test/bdd/

A scenario lives in test/bdd/features/. Its directory marks it, not a suffix. A scenario that is an Integrative Test carries the Gherkin tag @integrative.

Targets, for every profile with tests:

Target Runs
test Programmed Unit Tests and Component Tests
bdd-test Scenarios without @integrative
coverage What test and bdd-test run, measured together

make coverage prints Coverage: <n>% and fails below 80 %. The figure uses the metric of the language's capability. The GitLab pipeline reads it with /Coverage:\s+(\d+(?:\.\d+)?%)/.

Capability per language:

Language Capability
C++ 008
Go 012
Java 013
TypeScript 014
KiCad None, an electronics artifact has no tests

Functional Requirements

  1. Generated test files and scenario runners follow the names of Details.
  2. test and bdd-test run what Details lists, and no Integrative Test.
  3. make coverage reports one figure for what test and bdd-test run, and fails below 80 %.
  4. Code that only a scenario reaches counts as covered.
  5. make coverage prints the figure as Details defines, and the GitLab pipeline reads it with the regular expression of Details.

Non-Functional Requirements

None.

Technical Requirements

None.

Error Handling

None.

Out of Scope

  • How a language measures coverage, which its capability defines
  • A target that runs Integrative Tests, and sample Integrative Tests
  • Coverage of the generator itself
  • The coverage report format

Acceptance Criteria

  1. Every profile with tests, on both forges: the coverage target prints Coverage: and the figure. On GitLab, the pipeline's coverage job uses the regular expression of Details.
  2. Every profile with scenarios: bdd-test leaves out scenarios tagged @integrative.
  3. Every profile with tests: test leaves out Integrative Tests by the name or tag of Details.
  4. Every profile with tests: no generated test file or scenario runner has the name of an Integrative Test.

Decisions

  1. One figure for programmed tests and scenarios. A scenario is a form of test, not a kind. Counting it in one language but not in another makes the figures incomparable. Consequence: every language needs a way to count what a scenario reaches, as its capability defines.
  2. Integrative Tests do not count. Unit and Component Tests aim at particular code. An Integrative Test checks a feature and passes through much code on the way, without checking it. Counting it would raise the figure without saying more about how well the code is tested. It also keeps make coverage free of real dependencies such as Docker. Consequence: code that only an Integrative Test reaches, such as a repository adapter, is excluded from the figure by the developer, as the testing guideline says.
  3. Suffixes in every language, tags where the framework has them. Not every test framework filters by tag. A suffix works everywhere and shows the kind in the file list. Consequence: in Java, *IT.java means Integrative Test, so the Cucumber suite carries no suffix.
  4. No target and no sample for Integrative Tests. A generated repository has no real dependency to test against. A target without tests would only be noise. Consequence: the developer adds the target with the first Integrative Test, following the testing guideline.
  5. One output and one regular expression for every language. The tools print differently, and a regular expression per language had to follow each tool. Consequence: the label says Coverage, not Line coverage, since not every language measures lines.
  6. A common capability and one per language. Each language measures differently and reaches the target state at its own time. Consequence: a rule for every language belongs here, and a language's capability refers to it instead of repeating it.