CI/CD Guidelines¶

See specs/architecture/ADRs/ADR-004-ci_pipeline_design.md for the full rationale. Summary:

Pipeline Stages¶

Run in order; parallelize within each stage:

Pre-Check — lint & static analysis | secret scan (fast feedback, every push)
Build — generate changelog (tags only, before Build Docs) | build software | build docs
Test — unit tests (always against Debug) | integration/dynamic tests
Package & Publish / Deploy — package image/artifact | security scan | promote/publish

Never recompile between stages, and never recompile a released artifact — it is promoted (re-tagged) from an already-tested build, not rebuilt. (Build-once principle, Humble & Farley, "Continuous Delivery", 2010.) Inject all configuration via environment variables.

Trigger tiers¶

Trigger	What runs
Push to feature branch	Pre-Check only
Merge Request (MR tier)	Debug build; Unit + Integration/Dynamic Tests against Debug
Merge to `main` (Merge-to-main tier)	Everything the MR tier did already covered by branch protection (see below); additionally builds Release and runs Integration/Dynamic Tests against it; packages a commit-SHA-tagged intermediate image
Git tag `vX.Y.Z` (promotion tier)	No rebuild, no re-test: re-tags the Merge-to-main tier's intermediate image as the release; regenerates `CHANGELOG.md`; rebuilds docs to embed it; publishes

Precondition: a Merge Request may only be merged once its branch is up to date with main and its pipeline is green; direct pushes to main are disabled. This is what makes the Merge-to-main tier able to skip re-running what the MR tier already validated — the commit landing on main is exactly the tree the MR pipeline tested.

Registry Tagging¶

"Image" below covers both container images (services, CLI tools) and versioned packages (libraries — e.g. a Conan package, Go module, or Maven artifact) published to their respective registries.

Event	Tags	Note
Merge Request	`pr-<N>-sha-<short>` (manual only)	Debug-tested preview, not release-validated
Merge to `main`	`sha-<short>`, `latest`/`nightly`	Release-tested; this is the artifact a tag will promote
Git tag `v1.0.0`	`1.0.0`, `stable`	Re-tag (promotion) of the merge-to-main image — no rebuild

Package registries without a native retag-by-reference primitive (Conan, Maven — unlike OCI image registries) promote via a deterministic rebuild from the exact tagged commit instead of a registry-level retag; branch protection already guarantees that commit is content-identical to what the Merge-to-main tier built and tested (see ADR-004).

Makefile Targets¶

build — debug symbols, no optimizations; used on the MR tier for both build and test.
release-build — no debug symbols, speed optimizations; built and integration-tested on the Merge-to-main tier, then promoted (not rebuilt) at tag time.
release-test/release-coverage (C++ only) — re-runs the same test suite against the Release build. Only languages whose Release mode disables safety/correctness checks that Debug enforces benefit from this; see ADR-004's scope note for which languages qualify and why the others skip it.

Releases¶

A Git tag vX.Y.Z triggers two Build-stage/Package-stage jobs, in this order:

changelog (Build stage, runs first): regenerates CHANGELOG.md (full history) via git-cliff and commits it back to the default branch. Must run before the docs build, since the docs site embeds this file — the docs job depends on (needs:) this job.
release (Package & Publish/Deploy stage): generates RELEASE_NOTES.md (latest tag only, from the changelog job's output) for the GitLab release description, creates the GitLab release entry, re-tags (promotes) the merge-to-main image as the release, and triggers publishing the docs built alongside the changelog.

Required CI/CD variable: CI_ACCESS_TOKEN

Set this as a protected project CI/CD variable with write_repository scope (Project Access Token or Personal Access Token). It allows the pipeline to push the updated CHANGELOG.md back to the default branch. Without it, the changelog commit step is skipped and the job fails.

Documentation¶

A Git tag vX.Y.Z triggers publishing the documentation as a GitLab Pages site. Publish on release tags only — not on every commit — so published docs always match a stable, released version. The docs site embeds a page rendered from CHANGELOG.md, so the docs build must depend on (needs:) the changelog-generation job, not just on the code build — otherwise it risks embedding a stale changelog.

GitLab Pages requirements¶

Job name must be exactly pages — GitLab Pages ignores any other job name.
Artifacts must be in public/ — place the built site output there.
expire_in: never — expiring Pages artifacts takes the site offline.
Use needs: [changelog] so docs publish only after CHANGELOG.md has been regenerated (see "Releases"). No code-rebuild job runs on the tag pipeline under the trigger-tiered model (see ADR-004) — the binary was already Release-tested at the Merge-to-main tier, so there is nothing named build left to depend on here.
A job named pages always deploys when it runs. "Build the site as a test, without publishing it" therefore needs a second job under a different name — see the mono-repo split below.

Skeleton¶

pages:
  stage: docs
  image: <your-mkdocs-image>
  needs: [changelog]
  rules:
    - if: $CI_COMMIT_TAG =~ /^v[0-9]+\.[0-9]+\.[0-9]+$/
  cache:
    paths: []
  script:
    - mkdocs build --site-dir public
  artifacts:
    paths:
      - public
    expire_in: never

Mono-repos: one combined site, built often, published on a tag¶

GitLab allows only one Pages site per project, so in a mono-repo each artifact's own pages job is suppressed and the root pipeline builds a single combined site instead. That site is split across two jobs:

docs-build — runs on merge requests and on the default branch whenever a specs/ directory or an mkdocs.yml changed, keeps public/ as a short-lived artifact, and publishes nothing. This catches a broken docs build at review time.
pages — runs only on an artifact release tag and publishes.

The combined site is a catalogue rather than a released artifact of its own, so it carries a build date instead of a version number, plus a table of each artifact's released version regenerated from the repository's Git tags.

GitLab Hints¶

CI_ACCESS_TOKEN setup (required for the changelog job)¶

The changelog job pushes the updated CHANGELOG.md back to the default branch. This requires a token with push access and matching branch protection settings.

Required setup (once per project):

Protected Branch setting — Settings → Repository → Protected Branches → main
"Allowed to push": Maintainers (Developers still cannot push directly and must use Merge Requests.)
Create a Project Access Token — Settings → Access Tokens
Role: Maintainer
Scope: write_repository
Add CI/CD variable — Settings → CI/CD → Variables
Name: CI_ACCESS_TOKEN
Value: the token
Enable Protected
Protect the version tag pattern — Settings → Repository → Protected Tags
Add pattern v*

Protected variables are only injected into pipelines triggered by protected branches or tags. If the tag is not protected, CI_ACCESS_TOKEN is empty and the push fails with 401.

GitLab quirks worth knowing before your first release¶

Never push multiple tags in one git push (e.g. git push --tags) — GitLab only creates a pipeline for a limited number of refs per push; extra tags are silently skipped. Push each tag with its own git push origin refs/tags/<name>.
The changelog job moves the default branch by committing back to it. If you tag a second artifact afterward, always target an explicit commit SHA (git tag <name> <sha>), never "create from branch tip" — the tip may already be a sibling's changelog commit that never built this artifact's own image.
"manifest unknown" right after a successful push-image can be registry propagation lag — retry the job. If it persists, check the namespace's storage quota (Usage Quotas → Storage); job artifacts count against it and can silently starve the container registry.

Reserved job name `image:`¶

image: is a reserved keyword in GitLab CI/CD — it sets the Docker executor image for a job, not a job name. Using it as a job name causes a pipeline parse error.

Name Docker image build jobs build-image: and release-image: instead.

Registry promotion (retag without rebuild)¶

The tag/promotion tier re-tags the merge-to-main image as the release instead of rebuilding it (see ADR-004). Plain docker pull + docker tag + docker push works, but a dedicated promotion tool avoids pulling the full image layer set just to re-tag it:

crane tag <image>@<digest> vX.Y.Z (go-containerregistry)
skopeo copy docker://<image>:sha-<short> docker://<image>:vX.Y.Z
or the CI provider's native retag/promote mechanism, if it has one

For library projects, the equivalent is promoting/aliasing the already-published merge-to-main package version in the package registry, not rebuilding the package.

Overriding ENTRYPOINT for non-shell images¶

Some CI images declare a custom ENTRYPOINT (e.g. aquasec/trivy uses trivy, gcr.io/kaniko-project/executor uses /kaniko/executor). GitLab CI injects commands via sh -c, which fails when a custom ENTRYPOINT is set. Override it with entrypoint: [""] to restore normal script execution.

Always use the long-form image: block for these images:

scan:
  image:
    name: aquasec/trivy:0.72.0
    entrypoint: [""]
  script:
    - trivy fs ...

Scanning an image in the registry needs credentials¶

A filesystem scan (trivy fs) reads the checkout and needs no authentication. An image scan (trivy image) has to fetch the image first, and a CI runner has no Docker daemon, containerd, or podman to fetch it from — so Trivy falls back to the registry and, without credentials, gets DENIED: access forbidden on a private registry. The image-building job's credentials do not carry over: those live in that job's own container.

Give the scan job its own credentials, and pin the source so a failure names the real problem instead of listing four unavailable runtimes:

scan-image:
  image:
    name: aquasec/trivy:0.72.0
    entrypoint: [""]
  variables:
    TRIVY_USERNAME: "$CI_REGISTRY_USER"
    TRIVY_PASSWORD: "$CI_REGISTRY_PASSWORD"
  script:
    - trivy image --image-src remote ...

A source scan must know about the dependency cache¶

A pipeline that caches dependencies inside $CI_PROJECT_DIR — .m2/repository, .go-cache, .conan, .npm-cache — has that cache restored into every job, because the cache: block lives under default:. The scan job gets it too, and that cuts both ways.

Keep the cache out of the walk. Where Trivy's analyzers recognise what the cache holds, it becomes part of the scan. Go is the clear case: every go.mod below .go-cache/pkg/mod is a scan target of its own, so a freshly generated service reports 35 targets and 228 packages instead of 1 and 13 — the surplus being module versions the project does not depend on.

  script:
    - trivy fs --exit-code 1 --severity HIGH,CRITICAL --scanners secret,vuln --skip-dirs .go-cache .

Pass --skip-dirs for the cache directory in every language, not just Go. The other caches happen not to match any analyzer today — Trivy looks for pom.xml, package-lock.json and conan.lock, none of which a package cache contains — but that is a property of the current analyzer set, not a guarantee, and the flag costs nothing.

Let the resolver read the cache. For a Maven project the same directory is something Trivy needs. Resolving a pom.xml means following its parent and imported-BOM chain to find the versions of managed dependencies, and Trivy looks for those POMs in ~/.m2/repository — not where -Dmaven.repo.local put them. Without the pointer it fetches every POM from Maven Central, which answers a shared runner IP with 429 Too Many Requests and a 30-minute block:

  before_script:
    - mkdir -p ~/.m2
    - printf '<settings><localRepository>%s/.m2/repository</localRepository></settings>\n' "$CI_PROJECT_DIR" > ~/.m2/settings.xml

The effect is not subtle. On a generated Spring Boot service, scanning without the pointer resolves 2 packages; with it, 41 — and no longer touches the network at all. --skip-dirs and <localRepository> do not conflict: the first controls the walk over the scan target, the second the resolver's lookups.

Resist --offline-scan as the fix. It removes the network access, but on a cold cache Trivy then drops every dependency it cannot resolve — for a BOM-managed stack nearly all of them — logs a single WARN, and exits successfully. A rare rate-limit failure is a better outcome than a scan that passes because it found nothing to look at.

Order the scan after whatever resolves the dependencies, don't just point it at the cache. Pointing Trivy at $CI_PROJECT_DIR/.m2/repository only pays off once that directory holds something. Jobs in the same stage run in parallel by default, and cache: is restored at job start and saved at job end — a job never sees what a sibling job resolved while both are still running. So the very first pipeline to see a new dependency version (an ordinary Maven-BOM version bump, not just a brand-new project) races the scan job against every other job that needs that same version, over the network, at the same moment:

scan:
  stage: check
  needs: [arch-test]  # arch-test shares scan's rule set, so it always runs alongside it,
                       # and its own `mvn` resolves the graph before scan needs it

Pick a needs target that carries the exact same rules: as the scan job itself — otherwise a rule change to one, and not the other, can leave scan waiting on a job that never starts.

Gate on fixable findings, report everything¶

A base image regularly carries CVEs its distribution has not patched yet — Trivy reports those with status affected and an empty fixed version. Nothing in your repository can resolve them, so failing the job on them leaves it permanently red and trains everyone to merge past a failing pipeline. Run the scan twice instead: once reporting everything without failing, once gating on what has a published fix.

  script:
    - trivy image --image-src remote --severity HIGH,CRITICAL --exit-code 0 "$IMAGE"
    - trivy image --image-src remote --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$IMAGE"

The vulnerability database is fetched once and cached within the job, so the second pass is cheap. Keeping base image tags current remains a separate, ongoing duty — --ignore-unfixed silences the gate, not the risk.

Quality report artifact convention¶

All quality gate outputs must be written to test_reports/ and exposed as CI artifacts with when: always and expire_in: 7 days. This makes reports downloadable from any pipeline run — including failed runs where the report is most needed.

Sub-directory	Content
`test_reports/linting/`	Linter output (checkstyle XML, text)
`test_reports/arch-test/`	Architecture check output
`test_reports/test/`	Unit / BDD test results (JUnit XML)
`test_reports/coverage/`	Coverage reports (Cobertura XML)

CI artifact block for quality gate jobs:

  artifacts:
    when: always
    paths:
      - test_reports/<subdir>/
    expire_in: 7 days

Coverage quality gate¶

All projects enforce a minimum line coverage of 80%. The make coverage target fails if the threshold is not met — this works both locally and in CI. Coverage is measured against the Debug build, which is what Unit Tests always run against (see ADR-004, "Build Configurations").

GitLab displays the coverage percentage in pipeline views and merge request diffs by extracting it from the job's stdout using the coverage: regex key on the coverage job. The regex is language-specific and matches the output of the respective coverage tool:

Language	Tool	Stdout format	GitLab regex
C++	gcovr	`TOTAL 20 17 85%`	`/^TOTAL.\s+(\d+\.?\d%)/`
Go	go tool cover	`total: (statements) 85.5%`	`/total:\s+$statements$\s+(\d+\.\d+%)/`
Java	JaCoCo (via XML)	`Line coverage: 85.0%`	`/Line coverage:\s+(\d+(?:\.\d+)?%)/`

The threshold is enforced inside make coverage, not in the CI YAML, so running make coverage locally also fails fast on insufficient coverage.

Directives¶

Pipeline stages in order: Pre-Check → Build → Test → Package & Publish/Deploy; never recompile between stages
Merge only if the branch is up to date with main and its pipeline is green; no direct pushes to main
MR tier builds/tests Debug only; Merge-to-main tier additionally builds Release and Release-tests it, producing a commit-SHA-tagged intermediate image
A tag promotes (re-tags) the Merge-to-main tier's already-tested image/package — never rebuilds or re-tests it
git-cliff regenerates CHANGELOG.md at tag time, before the docs build, and commits the update back to the default branch
Inject all configuration via environment variables; never bake config into images
GitLab Pages job must be named exactly pages; artifacts must be in public/; expire_in: never
A job named pages always deploys — building the docs without publishing needs a separate, differently-named job
Use entrypoint: [""] for CI images with custom entrypoints (Trivy, Kaniko, etc.)
trivy image jobs set TRIVY_USERNAME/TRIVY_PASSWORD and --image-src remote; the image-building job's credentials do not carry over
Image scans run twice: report everything with --exit-code 0, then gate with --ignore-unfixed --exit-code 1 — never block on a vulnerability that has no published fix
trivy fs excludes the cache dirs from default: cache: via --skip-dirs — they are restored into the scan job too
Maven scan jobs point Trivy at $CI_PROJECT_DIR/.m2/repository via ~/.m2/settings.xml
Never pass --offline-scan: on a cold cache it turns unresolvable dependencies into a silent pass
Maven scan needs a job that resolves dependencies first (e.g. arch-test); same-stage jobs run in parallel
Quality reports go to test_reports/ with when: always and expire_in: 7 days
Enforce minimum 80% line coverage via make coverage against the Debug build; fail below threshold
Never install tools in pipeline jobs — all tools must be pre-installed in the CI build image