Evaluation of the Template Effectiveness (2026-07-15)

An evaluation of the effectiveness of the project templates has been done. What we wanted to find out is: "How well do the project templates support an agentic AI coding assistant in implementing a given set of specifications, while at the same time fostering spec-driven development, testing, and clean code/architecture?"

The evaluation is not about the code generator itself but about the effectiveness of the project templates it generates.

This report summarizes the results.

Goal & Success Criteria

The generator's own goals are to accelerate development through ready-made CI pipelines, Makefiles, and Dockerfiles; to push both human developers and AI assistants toward spec-driven development (ADRs, use cases, BDD feature files); and to push both toward clean code and clean architecture (guidelines, linting, coverage as a quality gate).

"Efficient" cannot be judged without a reference point. As a baseline, an experienced developer (25+ years) estimated that setting up a simple backend service by hand — repo, reading the spec, writing the code — takes ~3 days. Success was defined as an AI implementing the same specification in a few hours, ideally minutes.

Per language, the proof of practicality was considered established if:

• the AI implements the specification in ~2h,
• the pipeline is green (lint, architecture test, coverage),
• line coverage ≥ 80% is reached,
• all BDD tests (native and the language-neutral acceptance suite) pass,
• manual verification through the frontend succeeds.

For each run, the following metrics were captured against the 3-day baseline: wall-clock time (or number of AI sessions) until all success criteria were green; the number and kind of manual interventions (correction, clarification, restart); and the size of any manual post-hoc correction diff relative to the AI-generated codebase.

Approach

1. A functional specification for a simple Task Management application has been written.
2. Use cases.
3. Business rules.
4. Domain model.
5. A technical specification has been provided.
6. System context.
7. Architecture decision records.
8. OpenAPI specification.
9. A simple React frontend has been implemented. This helps human developers to quickly assess whether the application is working as expected. This web frontend has not been written using the project templates. As of the time of this writing, the code generator did not support web frontends.
10. End-to-end tests for the frontend have been developed in the form of BDD tests, so that a single acceptance-test suite runs unchanged against all three implementations.
11. Then, three repositories have been prepopulated by the code generator. These repositories are supposed to host the Task Management backends which are to be implemented by Claude Code, each scaffolded fresh via repo init so the CI pipeline of each backend runs in true isolation. The identical specification has been copied into all three repositories at the time each was implemented.
12. Finally, Claude Code has been tasked with implementing the three backends, in the fixed order Java, then Go, then C++.
13. Claude Code used the aforementioned end-to-end BDD tests to verify its own work.
14. Claude Code also implemented unit tests and BDD tests against the service API.

Two additional points matter for how the results should be read:

• The specification itself was authored together with GitHub Copilot, not Claude Code, so that specification and implementation deliberately sit with different tools — closer to a real hand-off between spec authoring and implementation than to Claude Code grading its own homework.
• The generator/template state was frozen across all three runs: no template fixes were fed back in between languages. No later run could "learn" from an earlier one, keeping the three languages fairly comparable. This is also why the bugs listed below recur across languages instead of being silently fixed after the first occurrence.
• ADRs were not only prepared upfront: real unknowns are rarely all known in advance, so the implementing AI was expected to make its own architecture decisions during implementation — guided by the architecture guidelines, or by asking the human — and document them via ADR, exactly as a human developer would. This was itself part of the hypothesis being tested, not just a setup step.

Summary

• Evaluation conducted on: 2026-07-15 (Java), 2026-07-16 (Go, C++)
• Version of the code generator: v0.8.0 (frozen for the duration of all three runs)
• Claude Code mode: Sonnet 5
• Subscription plan: Claude Pro

Aspekt	C++	Go	Java
Time needed to implement the specification (minutes)	54	40	45
Overall impression	Successful; most issues found and fixed of the three runs, but still 0% manual code corrections needed	Successful; same timeframe as Java, but with fuller verification (both native and language-neutral BDD suites run, manual frontend test completed)	Successful; comfortably under the 2h target, highest coverage margin of the three
Number of bugs in templates	2 (Drogon dependency not resolvable via CMake out of the box; Drogon's built-in CORS preflight handling silently orphaned a custom OPTIONS handler)	3 (.go-arch-lint.yml unusable against the pinned tool version; Makefile lint target used outdated golangci-lint flag syntax; CORS headers missing entirely)	2 (unused scaffold files not removed; CORS headers missing entirely)
Number of shortcomings in templates	3 (no real Gherkin execution available for C++; arch_check.py possibly duplicated per project; no musl/Alpine build verified in this run)	2 (unused scaffold files not removed; no generator-side smoke test to catch tool-version drift)	2 (ArchUnit default setting too strict for real dependencies; OpenAPI-codegen guideline impractical against the newest Spring Boot/Jackson)
Number of decisions required during implementation	5 (not captured in an ADR this run)	7 (ADR-009)	6 (ADR-009)
Quality of the architecture	Clean; arch_check.py/clang-tidy green, 5 layering decisions made and documented in this report (no ADR filed)	Clean; go-arch-lint green after config repair, 7 decisions documented in ADR-009	Clean; ArchUnit green after config repair, 6 decisions documented in ADR-009
Idiomatic code	Idiomatic Drogon/Catch2 usage; BDD scenarios hand-transcribed rather than run, which is a structural gap versus the other two languages	Idiomatic Go (sync.RWMutex, zap, listener injection for testability)	Idiomatic Spring Boot/JPA
Test coverage	80.5% (352/437 lines, gcovr) — narrow margin, needed a second, targeted round of app-layer tests to get there (45.1% initially)	81.7% (statement coverage) — narrow margin	91.9% (181/197 lines, JaCoCo) — comfortable margin

Results for C++ Service Templates

Summary

54 minutes end to end, well inside the 2h target. make lint, make arch-test, and make coverage all passed, though coverage only reached 80.5% after a second, targeted round of tests for the app layer (45.1% on the first pass). 51/51 Catch2 scenarios passed, and 13/13 language-neutral Behave scenarios passed against the real C++ server and the real React frontend via Playwright/Chromium. There is no real Gherkin execution for C++ (no cucumber-cpp in the template or devbox.json), so the "native BDD" scenarios are hand-transcribed from the .feature files into Catch2 SCENARIOs rather than executed directly — a structural difference from Java and Go that matters below. No new ADR was written for this run's implementation decisions, unlike Java and Go.

Template Bugs that Need to be Fixed

• A freshly generated C++ service repository fails to configure with CMake out of the box: find_package(Drogon) cannot find DrogonConfig.cmake/drogon-config.cmake because the dependency isn't resolvable via the template's CMake/Conan setup as shipped.
• Drogon answers every OPTIONS request itself as soon as drogon::Options is registered together with a route's other methods — a separately registered, hand-written OPTIONS handler is therefore never reached and silently does nothing. The first CORS-preflight implementation attempt fell into this trap; worth documenting as a known Drogon gotcha in the template/guidelines so future implementers don't lose time rediscovering it.

Template Shortcomings that Should be Improved

• The template provides no real Gherkin-execution tooling for C++ (no cucumber-cpp), forcing scenarios to be hand-transcribed into Catch2. This isn't just an inconvenience: a known off-by-one bug in UC002-list-tasks.feature (also found in Java and Go) was "fixed" here silently and unintentionally, because the hand-written Catch2 scenario used the mathematically correct values without ever being checked word-for-word against the feature file. A real Cucumber/Godog-style execution would have caught the mismatch instead of masking it. Worth investigating whether a maintained C++ Gherkin runner exists that could be added to the template.
• scripts/arch_check.py may be identical across all generated C++ projects — if so, it should be a single shared template artifact rather than duplicated per project (this was already flagged separately as a general C++ architecture-check backlog item, see below).
• guidelines/tool_chain.md requires C++ to be built and tested on both musl (Alpine) and glibc (Debian) images; this run only built/tested locally, so that requirement remains unverified for this template as of this run.
• No new ADR was filed for this run's implementation decisions (Java and Go both produced one). The decisions were made just the same but only live in this report — the pattern of "the AI documents its own architecture calls via ADR" broke down here and should be re-checked in a future run.
• Health endpoints (/healthz/live, /healthz/ready) required as a minimum by monitoring.md were not implemented — same open tension between the monitoring guideline and the ADR scope exclusion seen in Java and Go (see Cross-Cutting Findings below).

Results for Go Service Templates

Summary

40 minutes end to end, matching Java's timeframe but with fuller verification: both the native Godog suite (30/30 scenarios) and the language-neutral Behave suite (13/13 scenarios, against the real Go backend, the real React frontend, and Playwright/Chromium) were actually executed, and a manual frontend test was carried out — both were still open items after the Java run. Line coverage reached 81.7% (Go's -cover measures statements, not lines), a narrow margin above the 80% gate. make lint and make arch-test only turned green after repairing broken tool configuration (see below).

Template Bugs that Need to be Fixed

• .go-arch-lint.yml does not work against the go-arch-lint version actually pinned in devbox.json (1.14.0), and fails in a freshly generated repository with no user changes at all. Four separate problems in the same file: in: globs were written as full import paths instead of the module-relative paths this version expects (so no component was ever matched); domain.mayDependOn: [] was rejected as incomplete by the validator; allow.depOnAnyVendor: false means every component with a third-party import needs an explicit anyVendorDeps: true, which the template sets for none of them even though app (logging, HTTP test frameworks) will need it in practically every real project; and the v3 default allow.deepScan: true produced a false-positive layering violation for ordinary constructor injection in cmd/main.go — the same category of problem as Java's ArchUnit issue below.
• Makefile's lint target uses golangci-lint 1.x flag syntax (--out-format ...), which no longer exists under the version 2.10.1 actually pinned in devbox.json — make lint fails before any linter even runs. v2 uses --output.<format>.path instead.
• CORS headers are missing entirely, identical to the Java template. This was invisible to make test/make coverage and to the native Godog suite (HTTP-level, no browser) — it only surfaced as 12 of 13 failing Playwright scenarios once tested against a real browser.

Template Shortcomings that Should be Improved

• pkg/greeter.go, pkg/greeter_test.go, and the /greeting demo endpoint in cmd/go-service/main.go should have been removed but weren't — the same pattern as Java's Greeter.java/GreetingController.java. This is now the second language in a row where the generator leaves working-but-irrelevant example code behind for the AI (or a human) to notice and delete by hand. Recommend having repo init simply not write this example code when no example domain is requested, rather than relying on manual cleanup afterward.
• Recommend a smoke test in the generator itself (or its CI) that scaffolds a fresh Go repo and runs make test && make coverage && make arch-test && make lint against the actual devbox.json-pinned tool versions — not just go build — so tool/version drift like the two bugs above is caught before it reaches a user.
• Health endpoints (/healthz/live, /healthz/ready) required by monitoring.md were deliberately not implemented, citing ADR-008's exclusion of a dedicated metrics pipeline/advanced observability for this demo backend. Java flagged the same gap as an open question rather than a deliberate decision — the guideline-vs-ADR tension itself still needs to be resolved by the user (see Cross-Cutting Findings).

Results for Java Service Templates

Summary

Implemented in roughly 40–50 minutes (estimated from log timestamps, no exact stopwatch measurement), well inside the 2h target. make lint, make arch-test, and make coverage all passed, reaching 91.9% line coverage (JaCoCo) — the most comfortable margin of the three runs. All 30 native Cucumber scenarios passed against the real HTTP layer and H2. The language-neutral Behave suite was not yet run against the Java service in this pass, and manual verification through the frontend was not performed because curl against the running service was blocked by the sandbox's network restrictions — though the Cucumber suite already exercised the exact same endpoints against a real running server via RestTestClient.

Template Bugs that Need to be Fixed

• Greeter.java and GreetingController.java — leftover scaffold files — should have been deleted but weren't.
• CORS headers were missing entirely, blocking the frontend (http://localhost:3000) from reaching the backend (http://localhost:8080). Fixing it required a new CorsConfig.java (infra layer, analogous to the existing ClockConfig) allowing /api/** requests for GET/POST/PUT/DELETE/OPTIONS, plus a new app.cors.allowed-origin property in application.yml (defaulting to localhost:3000); 127.0.0.1:4173 also had to be allow-listed so the BDD tests could run against the server.

Template Shortcomings that Should be Improved

• ArchitectureTest.java used ArchUnit's .consideringAllDependencies(), which produced 278 false-positive violations against real code (JDK, Spring, and JPA calls). Switching to .consideringOnlyDependenciesInLayers() fixed it. This is the same category of problem as Go's go-arch-lint deepScan false positive — an architecture-linter default that hasn't actually been validated against a real, compiling project.
• The guideline recommending OpenAPI code generation isn't practical against Spring Boot 4/Jackson 3, which are too new for reliable openapi-generator support; DTOs and controllers had to be hand-written instead. Worth revisiting once tooling catches up, or softening the guideline's phrasing for bleeding-edge framework versions.
• TaskRepositoryAdapter could not be marked final as oo_design_and_programming.md requires for leaf classes, because Spring Data JPA's exception-translation proxy (CGLIB) needs a non-final class. This is a legitimate, spec-worthy exception to the guideline rather than an oversight, but it should probably be called out explicitly in the guideline itself so it isn't rediscovered as a surprise each time.
• It looks like the monitoring endpoints status/alive/status/ready are missing — flagged here as an open question rather than resolved (see Cross-Cutting Findings).
• An off-by-one bug was found and fixed in UC002-list-tasks.feature (scenario UC002-S03): the expected page slice contradicted the zero-based paging semantics documented in tasks.yaml. This is a specification bug, not a template bug, but it was found here first and never made it back into the shared spec source (see Cross-Cutting Findings).

Cross-Cutting Findings

These issues appeared in more than one language run and point to the generator/templates rather than to any single implementation:

• CORS handling is missing from the generated backends. It occurred identically in Java and Go, both only surfacing once tested against a real browser rather than via unit/BDD tests at the HTTP level. Recommend adding a CORS guideline (in api_design.md or a new frontend_integration.md) and baking a working default into the generator templates themselves — the same way health endpoints or graceful shutdown are already expected — so a fourth backend doesn't have to rediscover this through its own browser test.
• Unused example/scaffold code is left behind. Java's Greeter.java/GreetingController.java and Go's pkg/greeter.go//greeting endpoint both should have been removed but weren't. Recommend having repo init skip generating this example code entirely when the user doesn't request an example domain, instead of relying on the implementer to notice and delete it.
• Architecture-linter defaults are too strict for real code. Java's ArchUnit .consideringAllDependencies() and Go's go-arch-lint deepScan/import-path defaults both produced false positives the first time they ran against real, compiling code. Recommend validating each template's default architecture-linter configuration against an actual compiling project — e.g. in the generator's own CI — rather than trusting a config that was only ever written once and never executed.
• A shared pagination spec bug keeps recurring. UC002-list-tasks.feature (scenario UC002-S03) has an off-by-one bug relative to tasks.yaml's documented zero-based paging. It was found and fixed in the Java repo, but the fix was never carried back into the shared demo/specs/ source, so Go rediscovered and re-fixed it independently, and C++ — lacking real Gherkin execution — silently masked it without anyone noticing. This should be corrected once, at the source, before any further language run.
• Health endpoints remain an open, unresolved question in every run. monitoring.md requires /healthz/live and /healthz/ready (or equivalent) as a minimum; the demo's ADR excludes "advanced observability" for this backend, but it's unclear whether basic health endpoints are meant to be covered by that exclusion. Java flagged this as an open question, Go decided explicitly to skip it citing the ADR, and C++ skipped it too — three runs, the same unresolved tension each time. This needs a specification-level decision (clarify ADR-008/ADR-009 or monitoring.md), not three independent per-language judgment calls.

Other Backlog Items (noted alongside this evaluation, not specific to it)

TODO.md also records a few items that surfaced during or around this evaluation but aren't part of the three-language practicality proof itself:

• README/onboarding: document post-init steps (e.g. go mod tidy), write a complete setup walkthrough, cover .gitlab-ci.yml/GitHub Actions parity, note that the Conan registry isn't supported on GitHub, and add a frontend=web option that also generates JS/TS test and lint tooling for service projects.
• C++ architecture checks: evaluate Include-What-You-Use (IWYU) and a clang-tidy-based include check as an alternative to the current Python script, and check whether arch_check.py is identical across all C++ project types (in which case it should be a single shared template, not one copy per project).
• This repository's own .claude/ has drifted from the generator template it is supposed to demonstrate (found 2026-07-14, e.g. update-versions/SKILL.md). Recommend a make target or check that diffs this repo's own .claude//specs/ against a fresh bootstrap run, so this kind of self-hosting drift is caught automatically rather than by accident.

Conclusion

All three runs met the practicality hypothesis comfortably: every language finished well within the 2h target (40–54 minutes), reached the 80% coverage gate, passed its architecture and lint checks, and required zero manual code corrections — the small number of human interventions were clarifications or explicit user decisions (e.g. the allowed CORS origins in C++), not fixes to AI-written code. The recurring bug categories — missing CORS, leftover scaffold code, and over-strict architecture-linter defaults — are template-level, not implementation-level, problems and should be fixed once in the generator rather than three times per language. The shared pagination spec bug and the health-endpoint question should both be resolved in demo/specs/ and the guidelines before a fourth language is attempted.