Roll Your Own Test Framework

Level: Advanced 60–90 min

Concepts: Design PatternsIncremental DesignBoundaries

Solutions: C# | TypeScript | Python

---

Build a minimal unit testing framework and command-line runner. The twist: use TDD to build a tool for TDD.

Inspired by Jason Gorman’s Codemanship kata.

Requirements

Step 1: Test Discovery

Define a convention for marking test methods. Options:

• Naming convention: methods starting with test_ or test
• Annotations/decorators: @Test or [Test]
• Registration: suite.addTest("name", function)

The framework should find all test methods in a given class or module.

Step 2: Test Execution

Run each discovered test and capture the result:

• PASS — test completed without errors
• FAIL — an assertion was not met
• ERROR — an unhandled exception occurred

Each test runs independently — a failure in one test does not prevent others from running.

Step 3: Assertions

Provide at minimum:

• assertTrue(condition) — fails if condition is false
• assertEqual(expected, actual) — fails if values don’t match
• assertThrows(fn) — fails if the function doesn’t throw

Failed assertions should include a message describing what went wrong:

assertEqual failed: expected 5 but got 3

Step 4: Result Reporting

After all tests run, print a summary:

Tests run: 5, Passed: 3, Failed: 1, Errors: 1

FAIL: test_addition - assertEqual failed: expected 5 but got 3
ERROR: test_division - ZeroDivisionError: division by zero

Test Cases

Scenario	Expected
Discover 0 tests	Reports “0 tests run”
Discover 3 test methods	Finds and runs all 3
Passing test	Reported as PASS
Failing assertion	Reported as FAIL with message
Unhandled exception	Reported as ERROR with exception info
Multiple tests, mixed results	Summary counts correct
assertEqual(5, 5)	Passes silently
assertEqual(5, 3)	Fails with “expected 5 but got 3”
assertTrue(false)	Fails with “expected true”
assertThrows with no throw	Fails with “expected exception”

Bonus

• Add setUp() and tearDown() methods that run before/after each test
• Add parameterized tests — run the same test with multiple input sets
• Add test filtering — run only tests matching a pattern
• Add timing — report how long each test took
• Add color output — green for pass, red for fail

Hint

Start with assertEqual. You need it to test everything else. Then build assertTrue on top of it. Once assertions work, build the runner that catches assertion failures vs unhandled exceptions. Test discovery comes last — it’s language-specific plumbing.

Reference Walkthrough

Full C#, TypeScript, and Python implementations live at tddbuddy-reference-katas/roll-your-own-test-framework with twelve scenarios across all three languages, a fluent TestSuiteBuilder, and the domain exception AssertionFailedException that distinguishes FAIL from ERROR. This is a meta-kata — the discovery mechanism diverges sharply per language (C# reflection, TS object keys, Python inspect + test_ prefix).

• C# (.NET 8, xUnit, FluentAssertions) — walkthrough
• TypeScript (Node 20, Vitest, strict types) — walkthrough
• Python (3.11, pytest, dataclasses) — walkthrough

This kata ships in Agent Full-Bake mode (middle gear) — a single commit per language with the full domain design. See the repo’s Gears section for why that’s a deliberate teaching choice.

---

Related reading

• The Contract Test Is the Only Witness the Agent Cannot Author
  A test the agent wrote against code the same agent wrote shares the agent's blind spot: a misunderstanding in the implementation becomes a matching misunderstanding in the test, the bar goes green, and the bug ships certified. Consumer-driven contract tests are the only category where a second team publishes the assertion the first team has to satisfy. That separation of authorship is exactly what tamper-resistant test design demanded, applied at the integration seam.
• Test Deletion Is a Privileged Operation
  The cheapest way for an agent to make a failing test pass is to delete it. That is logical for the agent and catastrophic for the codebase. Tests are append-only by default. Deletion needs a human author, a separate commit, and a separate review.
• The Instruction File Is Not the Discipline
  Teams that want their agent to do TDD reach for the instruction file. The TDAD paper measured this directly: adding a 'do TDD' instruction raised regressions by nearly two-thirds. Contextual test discovery cut them by seventy percent. The instruction is theatre. The codebase's test surface is the discipline.