LLM application security: injection and tool abuse

The question that opens any security review of an AI application is different from the traditional one. It is not "where does user input come in", it is "what can the model do if it believes the wrong thing". The answer defines the size of the possible damage.

The attack surface moves

In a traditional web application the boundaries are reasonably sharp: input, validation, query, response. In an LLM application, instruction and data travel on the same channel, and the model has no strong separation between "this is an order" and "this is content to read".

The OWASP Top 10 for generative AI applications maps that territory, and reading it should be the first step for any team putting an LLM into production. The item at the top of the list is the hardest one: prompt injection.

Prompt injection has no definitive fix

This is the point that causes the most frustration. SQL injection has a solution: a parameterised query separates code from data structurally. Prompt injection has no equivalent, because there is no layer separating instruction from content on the same channel.

Simon Willison has tracked the problem for years and keeps landing on an uncomfortable conclusion: filtering by a list of forbidden patterns fails, because the space of phrasings is infinite. Every new filter is bypassed by a different wording, another language, an encoding.

The practical consequence is that the defence cannot be "prevent the injection". It has to be "limit what the injection can reach".

The indirect kind is the frightening one

Direct injection, typed by the user into the chat, is the best known variant and the least dangerous, because the user is usually attacking their own session.

Indirect injection is another story. It arrives inside content the system reads: a fetched page, an uploaded document, a processed email, a comment on a ticket. The malicious text enters the context without anyone typing it there, and the target is not the attacker, it is another user or the company itself.

Any application that combines reading external content with access to tools has to handle that scenario explicitly, in the design.

The damage happens in the tools

A model that only writes text has limited reach. A model that can query a database, call an API, send a message or delete a record has the reach of the most powerful tool you handed it.

Three controls cover most of this:

• Least privilege per tool. Queries with read-only credentials; writes restricted to the minimum scope; no shared, broad credential.
• Authorisation outside the model. The user's permission is checked in code before the action runs, never through an instruction asking the model to respect limits.
• Human confirmation for irreversible actions. Sending money, deleting data, writing to an external customer. The cost of one click is low compared with the cost of a wrong action.

Microsoft's report on lessons from red teaming more than a hundred generative AI products reinforces that framing: successful attacks tend to exploit integrations and permissions, not sophisticated weaknesses of the model.

The output path leaks too

Two leak routes show up often and are easy to forget.

The first is retrieval without permission control. If the index holds documents at several access levels and search does not filter by the user who asked, the system becomes a leak with a nice interface. The filter belongs in the query, not in the instruction.

The second is output rendered without escaping. If the model's answer goes into HTML untreated, it can return executable markup, and you have a classic browser injection problem from a new origin. Treat model output exactly as you treat untrusted user input.

Red teaming has to be routine, not an event

The risk profile of an AI system changes when the prompt changes, when a new tool is added and when the vendor updates the model. A security test done once at launch ages in weeks.

What works is keeping a set of adversarial cases in the same pipeline as your quality evaluations: direct and indirect injection attempts, privilege escalation requests, system instruction extraction, tool abuse. It runs on every change, alongside everything else.

The generative AI profile published by NIST, a companion to its risk management framework, helps assemble that list of scenarios without starting from scratch.

What this means for your team

Start with the question from the first paragraph: list every tool the model can invoke and, for each one, write down the worst possible outcome if it is invoked with bad intent. The list is usually shorter and scarier than people expect.

Then move authorisation and permission filtering into code, if they are still in the prompt. That is the change that reduces the most risk per hour of work.

And add five adversarial cases to your CI this week. Five already change the conversation, because they turn AI security into something the team measures rather than something the team hopes is fine.