THE AI-NATIVE SOFTWARE DEVELOPMENT PLAYBOOK
Chapter 06
Quality, security and governance
Treat autonomy as a risk decision.
Use independent checks, least privilege and accountable release gates.
Verification must be independent of generation
AI-generated code can be plausible and incorrect. A model may overlook an edge case, misunderstand authorisation or produce insecure dependencies. The remedy is defence in depth: executable tests, static checks, dependency analysis, threat modelling, independent reviews, controlled credentials and monitored releases. Having an AI agent review its own output is useful feedback, but does not amount to independent assurance.
Treat autonomy as a risk decision
| Action class | Default permission | Evidence or approval |
|---|---|---|
| Read approved code/docs | Allow with access controls | Log access to sensitive systems |
| Modify isolated branch | Allow for bounded tasks | Automated tests and visible diff |
| Call external systems / write data | Restrict | Explicit scope, least privilege, action logging |
| Change IAM, secrets or controls | Require approval | Security review and independently enforced policy |
| Deploy or mutate production | Require controlled authorisation | Release checks, monitoring and rollback |
The ServiceFlow permissions failure
Suppose an agent produces an approval endpoint that checks whether the caller is authenticated but fails to confirm that the caller is the assigned approver. The feature appears to work in happy-path testing. A negative authorisation test must prove that a requester cannot approve their own request and that a manager from another department cannot approve it. A server-side policy check, not merely a hidden UI button, must enforce the rule.
The team fixes the vulnerability, adds regression tests and documents the control. The change does not ship merely because a generated test suite reports success. An authorised reviewer checks the design, permission model and test evidence.
Prompt injection and tool security
Agent systems can encounter instructions embedded in repository files, issue tickets, webpages, log output or retrieved documents. Such content may attempt to redirect the agent into leaking data, changing settings or executing unauthorised actions. Prompts and policies reduce confusion but are not access boundaries. Tools should enforce authentication, authorisation, validation, allowlisting where appropriate, secret isolation, resource limits and audit logging. OWASP describes prompt injection and sensitive information disclosure among core LLM application risks [8].
Minimum release gate
- Acceptance tests and meaningful negative cases pass.
- Code review addresses risky changes, dependencies and architecture.
- Security scans are interpreted, not merely reported as green.
- Sensitive data, credentials and agent permissions have been checked.
- Deployment is attributable, observable and reversible.
- Known residual risks have a named decision owner.
IN PRACTICE Increase autonomy only when tested boundaries, reliable rollback and observable failure modes justify it. High impact work merits stricter controls regardless of agent capability.
Research basis: Anthropic [1], OpenAI [6], OWASP [8], NIST [9].
Threat model the workflow, not only the generated code
Review the agent’s capabilities: which files can it read and write; which secrets are mounted; which remote systems can it call; can it install arbitrary packages; can it trigger CI jobs; and can it write directly to production? Attacks may arrive through malicious repository instructions, poisoned documentation or compromised tool outputs. Good controls include tool allowlists, credential minimisation, dependency verification, approval for privileged actions, audit logs and limits on execution and spend.
Incident learning
If a generated change creates an incident, preserve the diff, agent transcript where permitted, tool calls, approvals and test evidence. Determine whether the failure arose from unclear requirements, inadequate context, excessive permissions, deficient checks or review overload. Improve the systemic safeguard, and add an executable regression test whenever possible. The goal is accountable improvement, not assigning blame to an opaque model.