The client runs an AI-driven property-management platform: agents that process utility bills, triage maintenance requests, run owner onboarding, move work orders through their lifecycle, draft rental-market reports, and request reviews. An LLM decides; deterministic code executes. Money movement always requires a human approval.

Four channels are open to the public — a shared service mailbox, an SMS number, a website quote form, a tenant portal — and text from any of them reaches a model whose output drives a write to the system of record.

The client asked for a security review before anything went wrong: static analysis, design review, and specifically the AI-native risks — prompt injection and related techniques. Not an incident response.

What the review found

A live vulnerability, one day old. A feature had shipped the day before that routed an inbound message to a work order by quoting its six-character reference — built to fix a real problem, where replies from unrecognized senders (a contractor's office manager, say) were being silently dropped. But the reference set the target record regardless of who sent the message, and that same reference travels in every email subject and vendor text the system itself sends out. Anyone who had received one of those messages could have emailed the shared mailbox quoting it and moved a work order to COMPLETED — which in turn proposed closing the tenant's original request and scheduled it for archiving.

Behind it sat a chain of gaps that made a mistake like that possible in the first place:

  • No security gates in the build. Pre-commit ran lint, type-checking, and tests — no static analysis, no dependency scanning, no secret scanning, and no CI at all.
  • Deploys ran from a laptop, authenticated as a human account holding subscription Owner — a role that can delete storage, read every secret, and rewrite the access control protecting all of it, when publishing the function app itself only needed Contributor on one app. Forty-eight commits of running production code existed on that laptop and nowhere else.
  • No separation between instructions and data in prompts. Untrusted message bodies were interpolated directly into prompts, with no delimiter and no statement that the content was data rather than instruction.
  • Model output flowed unescaped into email subjects and chat cards — a newline in a model-generated summary is header injection; a control character breaks an Adaptive Card.
  • Unauthenticated cost amplification. Every inbound message triggered a paid model call. Nobody had to breach anything to run up the bill — only to keep sending.

What we did

The routing fix was worth keeping — a quoted reference is still a reasonable way to say which record a message concerns. What it can't do is stand in for who sent it. We separated the two: only a sender matched to a known vendor, owner, or tenant earns a comment the lifecycle reads as progress. Everything else is logged "Unverified," stays visible with the sender flagged unknown, and is excluded from anything that changes state.

Alongside that:

  • Security gates went into the same command developers already run — SAST, dependency CVE scanning, and secret scanning next to lint, types, and tests, so a scan can't be skipped by forgetting. The first pass found a weak hash in active use and two dependencies with published advisories.
  • Prompt fencing. Every prompt that reads untrusted content now wraps it in explicit markers stating that nothing inside is an instruction — with the markers stripped from the content first, so a sender can't include a closing marker and have the rest of their message read as if it came from the system.
  • Output sanitizing strips newlines and control characters from model summaries before they reach an email header or a card.
  • Rate limiting on all four public channels — per-sender and global hourly caps, checked before the model call, keyed to the true sender rather than a shared gateway address, and failing open, so a tenant reporting a real leak is never blocked because a counter had a bad day.
  • The deploy path was rebuilt. CI now runs the full gate on every branch and pull request. Deployment moved into CI via OIDC federation: short-lived tokens, no stored credential, an identity scoped to a single application instead of the whole subscription, and trust bound to one repository's production environment.

Verified, not assumed

Every control was tested against the thing it was built to stop, not just reviewed on paper. A fake AWS key confirmed the secret scanner breaks the build. A real injection attempt — forged fence markers, a system-override instruction, a request to leak the prompt — was sent to the deployed classifier: it classified the message correctly and described the injection attempt in its own summary, while three independent controls held underneath — the fence contained it, enum validation in code rejected the invented category, and the unverified-sender rule meant it couldn't have changed anything regardless. Twenty-two messages against a limit of twenty got the last two refused, in production. And the new deploy path was proven by using it: deploying through it, then calling the live endpoints, including one that exercises the restored configuration end to end.

What the work turned up along the way

Three findings said more than the checklist did. CI's first three runs each caught something that would have shipped from a clean machine but passed on the developer's own: a linter that had floated to a newer version with rules the local pin didn't have, a runtime dependency that was present locally but never declared, and two development dependencies carrying published CVEs that had quietly been patched by hand months earlier. None of that showed up locally. The value wasn't "CI runs the tests" — it was "CI is not this laptop."

Two defects had passing tests written against a mocked API path that had never once worked in production — one mocked a lookup whose real filter silently matched nothing, the other used a field name the API doesn't have. And five separate writes to a vendor API were returning HTTP 200, saving a record, and silently dropping part of it — a unit link, an assignee, a due date, a priority — with nothing thrown and nothing logged. One had been quietly demoting the priority of every task it touched, including an urgent habitability complaint; thirty-four records were reconstructed from the platform's own audit trail.

Where it stands

At completion: 350 automated tests, five documented silent-write traps, three silent-drop routing defects fixed, and a deploy identity cut from subscription Owner down to one scoped application. A hosting-plan upgrade to enforce deploy approval, a proper pre-production environment, and credential rotation remain open.

None of this required slowing the product down. The lesson was the same one that shows up in most systems that let a model touch production: a reference is not an identity, model output is never authority on its own, and a security gate nobody has watched fail is not a gate.

If your product routes public input through a model into a system of record, this is the shape of review worth running before an incident forces it. Talk to us about an AI risk assessment, see how our penetration testing team approaches AI-native attack surfaces, or talk to an expert about your own pipeline.