Contents

THE AI-NATIVE SOFTWARE DEVELOPMENT PLAYBOOK

Chapter 06

Quality, security and governance

Treat autonomy as a risk decision.

Use independent checks, least privilege and accountable release gates.

Verification must be independent of generation

AI-generated code can be plausible and incorrect. A model may overlook an edge case, misunderstand authorisation or produce insecure dependencies. The remedy is defence in depth: executable tests, static checks, dependency analysis, threat modelling, independent reviews, controlled credentials and monitored releases. Having an AI agent review its own output is useful feedback, but does not amount to independent assurance.

Treat autonomy as a risk decision

Action classDefault permissionEvidence or approval
Read approved code/docsAllow with access controlsLog access to sensitive systems
Modify isolated branchAllow for bounded tasksAutomated tests and visible diff
Call external systems / write dataRestrictExplicit scope, least privilege, action logging
Change IAM, secrets or controlsRequire approvalSecurity review and independently enforced policy
Deploy or mutate productionRequire controlled authorisationRelease checks, monitoring and rollback

The ServiceFlow permissions failure

Suppose an agent produces an approval endpoint that checks whether the caller is authenticated but fails to confirm that the caller is the assigned approver. The feature appears to work in happy-path testing. A negative authorisation test must prove that a requester cannot approve their own request and that a manager from another department cannot approve it. A server-side policy check, not merely a hidden UI button, must enforce the rule.

The team fixes the vulnerability, adds regression tests and documents the control. The change does not ship merely because a generated test suite reports success. An authorised reviewer checks the design, permission model and test evidence.

Prompt injection and tool security

Agent systems can encounter instructions embedded in repository files, issue tickets, webpages, log output or retrieved documents. Such content may attempt to redirect the agent into leaking data, changing settings or executing unauthorised actions. Prompts and policies reduce confusion but are not access boundaries. Tools should enforce authentication, authorisation, validation, allowlisting where appropriate, secret isolation, resource limits and audit logging. OWASP describes prompt injection and sensitive information disclosure among core LLM application risks [8].

Minimum release gate

IN PRACTICE Increase autonomy only when tested boundaries, reliable rollback and observable failure modes justify it. High impact work merits stricter controls regardless of agent capability.

Research basis: Anthropic [1], OpenAI [6], OWASP [8], NIST [9].

Threat model the workflow, not only the generated code

Review the agent’s capabilities: which files can it read and write; which secrets are mounted; which remote systems can it call; can it install arbitrary packages; can it trigger CI jobs; and can it write directly to production? Attacks may arrive through malicious repository instructions, poisoned documentation or compromised tool outputs. Good controls include tool allowlists, credential minimisation, dependency verification, approval for privileged actions, audit logs and limits on execution and spend.

Incident learning

If a generated change creates an incident, preserve the diff, agent transcript where permitted, tool calls, approvals and test evidence. Determine whether the failure arose from unclear requirements, inadequate context, excessive permissions, deficient checks or review overload. Improve the systemic safeguard, and add an executable regression test whenever possible. The goal is accountable improvement, not assigning blame to an opaque model.

top