Autonomy without governance is not ambition. It is exposure.
Agentic systems need more than mitigations for the core risks of LLMs — they introduce fundamentally new surfaces. Persistent memory raises context poisoning and unauthorised state retention. Autonomous planning opens the door to misaligned objectives and recursive loops. These are design-time decisions, and by runtime the decision has already been made.
Four surfaces the LLM threat model does not cover.
Security is not an afterthought and it is not architecture-dependent. Whether the system is a single agent, a multi-agent workflow or a swarm, these surfaces exist the moment the system can remember, plan and act.
A rule we do not negotiate
Security integrated through the lifecycle, not appended to it.
Our agent development follows the OWASP GenAI Security Project's agentic developer security guidelines. The structure below mirrors their lifecycle approach — secure design and development, secure build and deployment, then runtime — because proactive controls at each phase are what create a resilient framework against threats specific to agentic systems.
3.1Secure design & development
These early decisions shape every downstream control strategy. Getting them wrong is not a bug to patch later — it is a property of the system you then have to live with.
3.1.1Threat modelling for agentic systems+
Traditional and LLM threat models are the floor, not the ceiling. We model the agentic surfaces on top: what the agent remembers, what it can reach, what it can trigger, and what happens when it is wrong. Safe-failure states are defined here, at design time, rather than discovered during an incident.
3.1.2System prompt engineering and hardening+
Explicit boundaries and safeguards with a predetermined allowlist of topics and an implicit deny for everything else. Clear delimiters separating instructions from input, structured placeholders instead of raw concatenation, few-shot examples that demonstrate refusal, and deterministic controls limiting the agent to expected actions and data sources.
3.1.3Secure coding practices+
Validate every input — not just tool arguments, but user prompts, API responses and anything retrieved from memory. Error handling that does not leak. Secrets from a secrets manager, never hardcoded. Least privilege for every component, enforced rather than documented.
3.1.4Content moderation, designed in+
Identify the checks the domain actually needs — PII, toxicity, disallowed topics — choose between rule-based filters, classifiers and external services on requirements rather than fashion, and design what a moderation result triggers: block, flag, sanitise or alert. Then test the moderation against the policy it claims to enforce.
3.1.5Human-in-the-loop placement+
Risk-based identification of the actions that require approval before they happen: sending mail, modifying data, executing code, spending money, updating critical memory. A review interface that makes clear what is being approved, the human decision logged, and rate limiting so a flood of approvals cannot become a denial of service against your own reviewers.
3.1.6Memory security design+
Access control by least privilege, encryption at rest and in transit, validation and sanitisation before anything is written, dangerous instruction tokens stripped from recalled content, PII detected and redacted before storage, and human oversight to approve and audit memory entries. Persistent memory is the surface most often left entirely ungoverned.
3.1.7Input and output validation+
Guardrails on both directions, whether integrated into the agent, exposed as a tool, or inserted as a proxy. Strict schemas for tool arguments and expected outputs. Allow and deny lists. Output neutralised before rendering or forwarding, with suspicious payloads dropped rather than passed along — so an injection cannot propagate to the next agent in the chain.
3.1.8Authentication and authorisation+
Identify permission boundaries for every agent and tool. Use real identity providers and authorisation servers rather than shared API keys. Decide deliberately whether an agent acts as a business system or on behalf of a user, and issue the matching identity type. Use decentralised identifiers for remote agents.
3.2Secure build & deployment
Development environments sandbox agent chains so failures stay local. Deployment pipelines do more than package and release: they snapshot capabilities into labelled artefacts, log audit trails, and embed signed constraints on tool use and memory schema.
3.2.1Static analysis in the pipeline+
SAST on every commit and merge, focused on insecure API use, hardcoded secrets and unsafe patterns — including in the code the agents themselves generate.
3.2.2Dependency scanning+
SCA in the pipeline, with advisories mapped to the actual call paths so the BugFix Agent can propose the minimum safe upgrade rather than a blind bump.
3.2.3Environment hardening and sandboxing+
Agent-generated code and interpreter tools execute in isolation — containers, microVMs or sandboxed interpreters, chosen on isolation need versus overhead. Read-only filesystems where possible, restricted network egress, minimal OS privileges, and the deployment pipeline explicitly unreachable by the agent itself.
3.2.4Secure configuration management+
Dedicated secrets management with rotation, fine-grained scopes on external API access, IaC templates scanned for misconfiguration before deployment, and verification that no secret is ever written to a log.
3.2.5Pre-deployment testing+
Prompt-interface fuzzing with malformed and adversarial input, plus penetration testing against the agent's tool surface. Capabilities are snapshotted into clearly labelled artefacts, tool and API constraints are signed, and the release protocol supports rollback and runtime enforcement of behavioural boundaries.
And then runtime, continuously
Guardrails on inputs and outputs, moderation in line, PII redaction, traces and cost attribution, and active watch for drift and context poisoning. Plus the control most often missing entirely: a rehearsed path to revoke an agent's credentials and stop it, quickly, without taking down everything around it.
This section follows chapter 3, Agentic Developer Guidelines — Agentic AI Developer Security Guidelines: A Lifecycle Approach, in the OWASP GenAI Security Project's Securing Agentic Applications Guide 1.0, alongside Agentic AI — Threats and Mitigations and the OWASP Top 10 for LLM Applications. Section numbering above maps to that guide so a reviewer can check our work against the source. The guide is the work of the OWASP GenAI Security Project and its contributors; PhaiAI is an independent implementer, not an author.
The Agent Control Standard, on the wire.
Design-time controls constrain what an agent is built to do. A Guardian constrains what it is permitted to do, at the moment it tries. Every PhaiAI agent runs behind one, and we deploy them in front of agents we did not build.
Not every verdict is yes or no
The modify disposition is the one that changes how security feels to a development team. An unbounded query does not have to be blocked with an apology — it can be rewritten with a row limit before the tool ever runs. Controls that let work continue are controls that survive contact with delivery pressure.
Fail closed, and prove it
A Guardian that crashes, hangs or becomes unreachable and lets the host proceed has stopped governing — silently. The reference implementation ships with a fail-open default precisely so that the choice is visible. We set the failure posture to deny, then verify it with a test that kills the Guardian and asserts the agent stopped.
An audit trail that reconstructs
Every hook and every decision is recorded as an OpenTelemetry span and an OCSF event, so your security team reads the trail with the tooling it already runs instead of learning a new console. When someone asks why an agent did something last month, the answer is a query with a reason code attached.
{
"jsonrpc": "2.0",
"id": "7f21",
"result": {
"type": "final",
"acs_version": "0.1.0",
"request_id": "b3e1c9f0-3b21-4b7a-9f2e-6a6d2e6a9a11",
"decision": "deny",
"reasoning": "Tool call targets the deployment pipeline. Development
agents are scoped out of ci_config by policy.",
"reason_code": "scope_violation_pipeline_write",
"policy_ref": "payments@v4.1.0#agents.development.scope"
}
}
// ... and the same agent, asking for something it may modify
{
"decision": "modify",
"reasoning": "Unbounded SELECT against customers exceeds the row-limit
policy. Query capped at 100 rows.",
"modifications": {
"parameter_overrides": {
"query": "SELECT id, name FROM customers LIMIT 100"
}
}
}The wire format, the five dispositions, the deterministic-first decision chain, the sixteen lifecycle hooks and the Instrument / Trace / Inspect pillars are defined by the Agent Control Standard, an open project of the OWASP GenAI Security Project. Its reference Guardian runs Microsoft's Agent Governance Toolkit behind the ACS contract. ACS v0.1.0 names its own open gaps, including an unauthenticated reference wire and a fail-open default; we track those issues and configure around them rather than pretending they are closed. Decision payloads above are illustrative, in the shape the specification defines.
Four ways we are usually brought in.
Often after something has already gone sideways — an agent with credentials nobody scoped, a fleet nobody can inventory, or an audit question nobody can answer. All four work as standalone engagements.
Agentic threat modelling
A structured review of one agent or one fleet against the OWASP agentic threat model: memory, planning, tool access, output handling, identity and blast radius. Output is a findings register with named controls and owners, not a maturity score.
Guardian deployment
Stand up an ACS-conformant Guardian in front of your existing agents — including ones we did not build. Policy bundles as code, failure posture set to deny, and a verified deny before anyone declares it working.
Agent red teaming
Adversarial testing of the actual deployment: injection through retrieved content, memory poisoning, tool-argument abuse, cross-agent propagation, and escalation through legitimate capability. We report what we got the agent to do, with transcripts.
Estate inventory and AgBOM
Answer the question most organisations currently cannot: which agents exist, what models and tools they use, what they can reach, and who authorised them. Serialised as CycloneDX, SPDX or SWID, and kept live rather than surveyed annually.
Can you name every agent in your estate and what it can reach?
Most organisations cannot, and the number is usually higher than expected. We will inventory one business unit, model the surfaces, and hand back a findings register with named owners — and a Guardian in front of the riskiest agent before we leave.