It, Not They: The Security Cost of Treating LLMs Like People

Security Mental Model Deep Dive

Language influences control design

The article’s strongest point is that metaphors become architecture. When teams describe an agent as a colleague, they are more likely to grant it colleague-shaped access and to treat its output as judgement rather than computation. Using workload language keeps attention on identity, scope, ownership, expiry, and revocation.

Accountability cannot be delegated to a model

Phrases such as “the agent decided” create a phantom actor. A useful incident record instead identifies the human or service principal, the credentials used, the approved scope, and the owner of the resulting action. That framing makes remediation possible because it points to an actual control boundary.

Prompt-level safeguards are insufficient

The article connects anthropomorphism to misplaced trust in loyalty or scepticism. Its practical alternative is architectural: least privilege, deterministic authorization, egress controls, approval gates, and treating retrieved content as attacker-controlled input. These protections remain enforceable even when the model is confused or manipulated.

Evaluation should replace trust

Model behavior changes across versions, prompts, and tasks. Benchmarking on the organization’s real workflows, recording failure modes, and re-evaluating after changes is more reliable than assuming a fluent system has stable judgement. The final principle is simple: usefulness does not make a process a person.