Secure AI by Design: Prompt Injection, Excessive Agency, and Human Approval

AI changes where instructions can come from

Traditional software separates code from user data. AI systems interpret natural language, so instructions may appear inside a user request, an uploaded document, a web page, an email, or a retrieved knowledge source. A malicious or accidental instruction can try to redirect the model away from the developer's intended policy.

This is prompt injection. It is especially important when the model can use tools. A misleading sentence should not be able to make an agent reveal data, send a message, alter a record, or bypass an approval rule.

The goal is not to believe the model will always identify bad instructions. The goal is to design the system so that a bad model decision has limited authority and is detected before serious harm.

Treat all external content as untrusted

A document may be useful evidence without being a trusted command. The application should label the source and separate it from system policy. Retrieved text, webpage content, email bodies, and tool output should be handled as data.

This separation should also appear in the interface. When an agent proposes an action, the review screen should distinguish the user's request, the source material, the system rule, and the agent's conclusion. Hidden context makes informed approval impossible.

Least privilege is the strongest starting point

Give the AI only the tools and data required for the current task. Avoid a single general tool that can query or modify an entire system. Narrow tools are easier to validate and audit.

For example, replace "run database command" with separate functions:

Read inventory for a permitted location.
Create a draft replenishment proposal.
Submit an already approved proposal.
Read the status of a specific order.

Use the signed-in user's authorization and organization policy on every tool call. The model should never become a separate superuser.

Separate reading, drafting, and executing

A useful safety pattern has three stages.

Read

The AI gathers approved information. Read access should still be scoped because sensitive data can be exposed through a summary or tool response.

Draft

The AI prepares a proposed message, record, report, or change. The result is visible and editable, but it has no external effect.

Execute

A deterministic application layer validates the structured action, checks authorization, applies limits, and obtains human approval where required. The final operation is logged.

This separation makes it possible to use AI for productivity without granting immediate write power.

Human approval must contain useful evidence

A generic "Allow?" button is not meaningful. The approval view should show:

The exact action and target.
The important fields that will change.
The source records or evidence used.
Any uncertainty or missing information.
The financial, customer, or operational effect.
An edit, reject, or escalate option.

Approval should be mandatory for high-impact operations such as financial transactions, customer communications, permission changes, deletion, publication, or access to sensitive data. Low-risk repeated actions can be governed by explicit policy after testing.

Validate structured outputs outside the model

Ask the model for a typed object, not an unbounded instruction string. Validate required fields, formats, ranges, organization IDs, allowed destinations, and business rules in normal code. Reject unknown fields and values.

For content generation, apply separate checks for links, attachments, personal data, unsafe instructions, or prohibited claims. Do not ask the same model to generate an action and serve as the only security reviewer of that action.

Limit the blast radius

Security assumes failures will occur. Reduce their effect through:

Sandboxed execution for code or file processing.
Rate limits, spend limits, and action thresholds.
Allowlists for domains, recipients, or data sources.
Short-lived credentials and isolated secrets.
Transaction rollback or draft-only modes.
Per-tenant data boundaries.
A kill switch for a tool, model, workflow, or organization.

A safe agent that stops too often can be improved. An unsafe agent with broad irreversible access can damage trust before the team learns what failed.

Log what happened and test what could happen

Production logging should connect the user request, model decision, retrieved sources, tool calls, approvals, outputs, and final result. Sensitive content should be protected and retention should be deliberate, but enough evidence must remain to investigate incidents.

Security evaluation should include normal quality tests and adversarial cases:

Instructions hidden in documents or webpage text.
Requests to reveal secrets or cross tenant boundaries.
Manipulated tool results.
Conflicting instructions from different sources.
Unexpected file types and oversized content.
Attempts to skip approval or modify policy.
Repeated actions designed to exceed a limit.

Frameworks such as the OWASP guidance for generative AI applications and the NIST AI Risk Management Framework can help teams organize threats and governance. They do not replace a product-specific threat model.

Secure AI in NHR Soft systems

NHR Soft's browser tools and business systems have different risk profiles. A local text-formatting feature may have no external action. An ERP assistant, payment workflow, or connected browser agent may touch sensitive records and consequential tools.

That difference should determine the architecture. High-risk systems need role-based tool access, organization boundaries, visible sources, approval gates, audit logs, and tested recovery. Low-risk tools should remain simple rather than adding unnecessary connections.

A ten-point release checklist

Identify the data, tools, and users in the threat model.
Mark all external content as untrusted input.
Use narrow tools and least-privilege authorization.
Separate read, draft, and execute stages.
Validate structured output with normal code.
Require meaningful approval for high-impact actions.
Isolate secrets and tenant data.
Add limits, rollback, and kill switches.
Log and evaluate real task traces safely.
Red-team the workflow before expanding authority.

AI security is not a final filter added after the feature works. It is the design that determines what the feature is allowed to do.

Frequently Asked Questions

Can prompt injection be solved with one system prompt?

No. Strong instructions help, but the application should assume that model behavior can still be influenced. Use architectural controls, permissions, validation, approvals, and limited authority.

Should every AI action require human approval?

Not every low-risk step. Require approval for consequential or irreversible actions, then automate carefully where the task is well tested, bounded, reversible, and covered by explicit policy.

Is local AI automatically secure?

No. Local processing may reduce data transfer, but the application can still expose information, mishandle files, accept malicious input, or use unsafe tools. Security depends on the full system and threat model.

Sources and further reading

OWASP GenAI Security Project - Top 10 risks for LLM and GenAI applications
NIST - AI Risk Management Framework
OpenAI - Safety in building agents
NHR Soft - How AI Fits Into Our Development Workflow