Testing Guidelines

Tests are created after the README, release model, onboarding, core functionality, integrations, and canonical examples have defined the product's observable contracts. Tests defend those contracts; they do not invent the product indirectly through implementation details.

The fact that testing is the final creation stage does not make it optional. A product is not complete until its promised behavior is defended by meaningful, repeatable evidence.

Required outcome

The test suite must demonstrate that:

• the README's product promises are achievable;
• releases identify themselves and obey compatibility rules;
• a new user can complete onboarding from a clean environment;
• core workflows succeed end to end;
• failure, cancellation, retry, and recovery contracts hold;
• integrations preserve the core boundary and fail safely;
• every supported example reaches its documented result through the public interface;
• security, ownership, cost, and data-integrity invariants are enforced;
• supported upgrades and rollbacks behave as documented.

Derive tests from product contracts

Each test must identify the observable contract it protects. Sources include:

• README use cases;
• release and compatibility policy;
• onboarding steps and expected results;
• public commands, APIs, schemas, and configuration;
• state-transition definitions;
• integration capability declarations;
• the examples coverage matrix, commands, expected results, and failure paths;
• security and recovery invariants.

A test without a plausible product failure that it would catch should not exist.

Test hierarchy

Use the narrowest level that proves the contract without bypassing the real boundary.

Unit tests

Use unit tests for deterministic domain rules, parsing, validation, state transitions, calculations, and pure transformations.

Unit tests must not substitute for workflow tests when the contract crosses persistence, process, network, or interface boundaries.

Component tests

Use component tests for a complete subsystem with its real storage, protocol, or process boundary where practical. They should verify public subsystem behavior rather than private helper calls.

Integration tests

Use integration tests to prove adapter contracts, authentication shape, serialization, retries, idempotency, compatibility, and failure isolation.

A local protocol-compatible service may replace an external dependency when the test still exercises the production client and boundary. A mock that bypasses the adapter is not an integration test.

End-to-end tests

Use end-to-end tests for the most important user journeys:

• first-run onboarding;
• primary core outcome;
• result retrieval;
• a representative failure and recovery path;
• production-critical integration paths;
• representative examples from each supported risk and interface class.

End-to-end tests must begin at the real user or machine interface and assert the final observable result.

Operational and recovery tests

Exercise upgrade, rollback, migration, backup, restore, failover, fencing, cancellation, cleanup, and restart behavior when those are product contracts.

A recovery procedure that has never been exercised is only a hypothesis.

Test design

A strong test has:

• a clearly named user or system scenario;
• an explicit initial state;
• one documented action or workflow;
• observable success or failure criteria;
• assertions on durable outcomes and relevant side effects;
• deterministic cleanup;
• a plausible defect that would make the test fail.

Prefer assertions on outputs, state, artifacts, protocol messages, and externally visible transitions. Avoid assertions on incidental call counts, private method order, source text, or formatting that is not a public contract.

Behavior to cover

For each contract, consider:

• normal success;
• boundary values;
• invalid input;
• missing prerequisite;
• unauthorized and forbidden access;
• duplicate request;
• concurrent request;
• timeout;
• dependency outage;
• partial completion;
• cancellation;
• retry;
• restart and resume;
• stale state;
• upgrade and rollback;
• cleanup and retention.

Do not generate a matrix mechanically. Select cases that defend meaningful invariants and realistic failure modes.

Onboarding tests

Onboarding must be tested from an isolated, clean environment. The test should control home directory, configuration paths, credentials, current directory, and background services.

It must assert:

• first output is actionable rather than an unexplained error;
• installation selects an exact version;
• minimum configuration is sufficient;
• the first workflow produces the documented result;
• onboarding does not create hidden mutable state before confirmation;
• missing prerequisites produce corrective guidance;
• cleanup returns the environment to a known state.

Release and compatibility tests

Verify:

• all artifacts report the same canonical version;
• artifacts map to the expected source revision and digest;
• published versions are immutable;
• configuration and state migrations preserve documented behavior;
• supported mixed-version combinations work;
• unsupported combinations fail before mutation;
• rollback restores a supported state when promised.

Core workflow tests

Exercise complete state transitions through the public interface. Assert both the final result and the absence of invalid intermediate or duplicate state.

For asynchronous work, use bounded waits based on observable state. Do not rely on arbitrary sleep durations as proof of completion.

Integration tests

For each integration, verify:

• capability discovery;
• configuration validation;
• credential scope and selected identity;
• request and response translation;
• pagination and limits;
• rate limiting and bounded retry;
• idempotency;
• normalized errors;
• unavailable dependency behavior;
• core operation while the optional integration is disabled;
• compatibility with supported external versions.

Real external environments should be used for a small set of credentialed acceptance scenarios when local substitutes cannot prove provider behavior. Such tests must be isolated, budgeted, non-destructive, and explicitly controlled.

Security and privacy tests

Defend:

• authorization at every mutation boundary;
• tenant, user, workload, or namespace isolation;
• least-privilege credential selection;
• rejection of unsafe paths and inputs;
• secret absence from logs, durable state, artifacts, and error output;
• audit and provenance records;
• safe handling of malformed or hostile external data;
• destructive-operation confirmation and ownership checks.

Do not place real secrets or production personal data in test fixtures.

Determinism and isolation

Tests must:

• own their state;
• use unique identifiers and paths;
• avoid dependence on execution order;
• avoid shared mutable global configuration;
• use bounded timeouts;
• clean up resources they create;
• make time, randomness, and external responses controllable where needed;
• report enough evidence to diagnose failure.

Flaky tests are product defects. Do not hide them with broad retries or automatic reruns.

Fixtures and mocks

Fixtures must be minimal, readable, versioned, and representative of real contracts. Golden files are appropriate for stable serialized formats and user-visible output.

Mocks are permitted only when they preserve the boundary under test. Do not mock the behavior being claimed as verified.

Prefer:

• real local filesystem over an in-memory storage fake for persistence contracts;
• real child processes over mocked execution for process-lifecycle contracts;
• a protocol-compatible loopback server over replacing the production HTTP client;
• generated provider fixtures captured from documented schemas over invented responses.

Test execution

Define named execution groups such as:

• fast local contract suite;
• full local integration suite;
• credentialed provider acceptance suite;
• migration and recovery suite;
• release qualification suite.

Document prerequisites, duration class, external side effects, cost, required credentials, and cleanup behavior for each group.

A narrow passing suite must not be presented as proof for unexecuted layers.

Failure reporting

A failed test must identify:

• the scenario;
• expected contract;
• observed result;
• relevant state or artifact location;
• whether external dependencies were involved;
• cleanup status.

Preserve bounded diagnostic evidence without leaking secrets.

Prohibited patterns

Do not:

• test source text instead of behavior;
• assert private implementation details without a contract reason;
• call a mocked helper and label it end to end;
• use unbounded waits;
• depend on test order;
• share mutable fixtures across parallel tests;
• use production credentials or data by default;
• retry flaky tests until they pass;
• ignore cleanup failures;
• claim full product verification from unit tests alone;
• add tests that can never fail on a plausible defect.

Completion gate

The testing stage is complete only when:

• every promised primary outcome has end-to-end evidence;
• release and onboarding contracts are covered;
• core success and recovery paths are defended;
• integration boundaries and outage behavior are exercised;
• canonical examples and their cleanup/failure paths are exercised;
• security and data-integrity invariants are covered;
• suites are deterministic, isolated, and diagnosable;
• credentialed or destructive suites are explicitly controlled;
• test documentation states exactly what each executed suite proves.