When AI Writes Code, Who Writes the Guardrails: Addressing AI Security Risks

As LLMs get embedded deeper into developer workflows- writing code, generating configs, summarizing incidents, teams are moving fast to adopt AI without fully understanding what it breaks. The tools feel great at first, but behind the productivity boost lies a growing set of blind spots. These are being overlooked by most development teams. Their effectiveness is becoming more apparent.

The security surface is changing. Old attack patterns are resurfacing in new form. And most teams are shipping AI features without the right architecture, oversight, or even a basic threat model.

This episode of the AI x DevOps podcast tackles these issues head-on. Rohit, Co-founder of Facets, spoke with Nathan Hamiel, Director of Research at Kudelski Security and the AI/ML track lead at Black Hat. With decades in offensive security and recent focus on AI threats in production systems, Nathan shared a grounded, technical view of how AI is actually performing and where it’s quietly introducing risk.

What you’ll learn:

• Where AI-generated code adds overhead instead of saving time

• How prompt injection works and why it's difficult to fix at the model level

• How to incorporate LLMs into security tooling without breaking trust boundaries

• ​Why architectural decisions matter more than model accuracy in production setups

Key Takeaways:

• AI shifts development effort downstream; saved time in coding often leads to more time in integration and debugging.
• Prompt injection remains a core vulnerability.
• Most vulnerabilities in AI tools stem from poor containment, missing guardrails, or overexposed data.
• Using LLMs to augment deterministic tools like fuzzers or static analysis works well, when scoped carefully.
• ​RBAC, execution isolation, and sandboxing are critical for AI agents that trigger actions or access internal systems.
• Risk must be evaluated per use case.

Building with AI? Start with a Secure Mindset

We’re in a phase of AI euphoria. LLMs are being woven into products at record speed—writing code, handling data pipelines, powering agents. The results are flashy. But behind the scenes, we’re shipping old problems at new velocity.

“We’re seeing a monumental number of old school vulnerabilities coming back into production because of AI development.” — Nathan Hamiel, Director of Research, Kudelski Security

"

This post isn’t about future AI safety or speculation about AGI. It’s about what’s happening now- how security is quietly regressing in the rush to integrate AI.

1. Same Flaws, Shipped Faster

The most common issues we’re encountering aren’t new. They’re not even unique to AI. But the speed and volume of AI-assisted development are resurfacing conventional vulnerabilities in bulk.

Security is about discipline. LLMs don’t replace that. If anything, they require more of it because the illusion of productivity can mask what’s going wrong underneath.

2. AI-Native Risks Still Matter

Prompt Injection

​Source: https://www.promptfoo.dev/blog/prompt-injection/

The flagship example: users embedding instructions in input that override system behaviour. It’s hard to detect, harder to defend, and totally alien to traditional application threat models.

Hallucinated Dependencies

LLMs can invent package names. Malicious actors can register them. It’s not widespread, but it’s a new class of exploit that didn’t exist before.

These risks don’t replace traditional ones but compound them.

3. Architect for Failure, Not the Demo

Many of the worst vulnerabilities Nathan's team found weren’t clever exploits. They were architectural oversights:

• Secrets stored alongside agents

• Unrestricted internet access

• No validation of what code gets executed

The antidote is architectural discipline. At Facets, we’ve leaned into a deterministic pipeline model where the AI can assist in generating outputs—but the actual execution path remains reviewable, predictable, and safe.

“We don’t let AI make production decisions. It generates outputs that pass through a deterministic pipeline with validations and context-aware safeguards.” — Rohit Raveendran

"

This approach doesn’t remove all risk; but it contains it. And that’s the point.

4. Hallucinated Data: A Silent Threat

One overlooked risk isn’t in the code. It’s in the data. AI-generated output is already leaking into public datasets. Mistakes, inaccuracies, half-truths—all of it gets scraped and recycled.

“The hallucinated data of today may end up being the facts of tomorrow.” — Nathan Hamiel

"

Once it’s embedded in training data, it becomes harder to correct. Garbage in, model out. ​Read Nathan's blog​

5. No, We’re Not Seeing Exponential Progress

There’s a pervasive belief that AI progress is exponential. The tooling feels impressive—until you scratch beneath the surface.

Each model iteration feels magical at first. But deeper use reveals familiar problems: fragility, unreliability, bloated output, shallow reasoning.

The takeaway: be practical. Ground your decisions in what the model can do, not what the market says it might someday do.

Beyond the Hype Cycle: A Pragmatic View of AI's Future

This blog only scratches the surface of the conversation. In the full episode, Nathan Hamiel dives deeper into:

• Why model hallucinations are a long-term reliability risk

• The growing security blind spots in MCPs and AI agent stacks

• How teams can use existing DevSecOps workflows to safely scale LLM adoption

• Why “safe enough” is not a valid approach when AI output affects production systems

▶️ Listen now on [Spotify], [Apple Podcasts], [YouTube] or wherever you get your podcasts.

And if you found the episode useful, share it with your team, especially your security and platform engineers.​