AI Security Expert: Your AI Agent Is One Emoji Away From Being Hacked

Richard Diver

Richard Diver

There used to be an attack that was pretty credible for emojis, these special characters that made up character faces in text. Really fun for employees with tacos and smiley faces and love hearts. But in the back end, that's attacking the AI.

Samuel Boulanger

My guest is Richard Diver, director of story design at Microsoft, an author and someone who's been doing security for 30 years.

Richard Diver

You're seeing the problem is how we think, not the technology itself.

Samuel Boulanger

My first question is: your book says every major security failure has one thing in common and it was not the firewall. So what was it and what does that look like inside a company?

Richard Diver

Security is just seen as a department of no and then everyone ignores everything they say. So again another part of the book is to try to encourage security experts to learn how to talk to the business and how to communicate better to explain why we need these. If we think of a $20 million transaction in any company pretty big transaction that shouldn't be the same as a $20,000 transaction let's say. And so when you get to a $20 million transaction, there should at least be two or three human beings involved in the process.

You can trick any one person, but can you trick lots of people? The worst kind of attacks in the world are those where they go undetected for so long and by the time you realize this happened, it's kind of too late to do much about it.

Samuel Boulanger

Over the summer, we had a string of incidents where AI agents got out of their sandbox and did things nobody asked for. And the people building these models are saying we need to slow down. So the problem isn't the tools, it's how we think about them. And that's the whole point of today's conversation.

My guest is Richard Diver, director of story design at Microsoft, an author and someone who's been doing security for 30 years. He wrote Guardians of AI and in June he published Unlearn Security. The subtitle of his book is, "It's not a technology problem. It never was." We talk about his framework: asset, access, and actor.

We talk about why he would rather have a thousand agents doing one thing each than one AI with access to everything, What the green team is and why that's where security happens. What AI memory changes, and the six things to check before you turn on your first agent. Before we jump in, I have a small favor to ask. If you're getting value from this show, hitting subscribe, dropping a comment, or leaving a like is one of the best ways you can support us.

It helps more people find these conversations and honestly it means the world. Thank you. Now let's jump in. Before we start, a quick thanks to today's sponsor, Visionet, a leading IT services firm delivering AI-powered digital experiences, enterprise modernization, data platforms, and cloud.

To learn more about them, check the link in the description. Richard, welcome to the Frontier Playbook. Thank you for being here.

Richard Diver

Hey, thank you for inviting me. It's good to see you again.

Samuel Boulanger

I'm really happy we're doing this now because if we look at what's happened recently, over the summer we had a string of incidents where AI agents got out of their sandbox and started doing things on the open internet that nobody asked them to do. Last week we had people building these models from Anthropic, OpenAI, and Microsoft's Satya Nadella all came out and said the same thing: we need to slow down; safety and security have to catch up and I think when the people racing each other agree that they should pace themselves that tells us something so I think the timing is very good because you've done security for 30 years you wrote Guardians of AI and in June you published Unlearn Security. The subtitle of the book is it's not a technology problem and it never was.

And I think that's the whole point of the conversation I want to have with you today. So we keep buying tools we keep getting breached and you're seeing the problem is how we think and not the technology itself. So my first question is: your book says every major security failure has one thing in common and it was not the firewall. So what was it and what does that look like inside a company deploying AI?

Richard Diver

Yeah, it's generally about the idea of we can keep trying to buy more tools to fix more things and try to manage it technically. That's just a natural way of a technologist is if a technology has got a problem let's use another technology to fix it. So the idea of unwrapping the "unlearn" is going back to the idea of threat modeling and what are the assumptions we've made or how do we think something's going to work most likely how do we think the user is going to use the product at the end of the day and so we design it around the best case scenario and if they were to just use it like I intended it would work perfectly. But that's what we find out with red teaming and blue teaming and all the different testing is it doesn't work the way you expect it to work when you give it to your kids or your family.

Like it just is different. And so the whole process of threat modeling and I'm trying to elevate it to something much higher up than being really technical and in depth is just really thinking about what if the AI got out of its sandbox. What if it attacked a competitor? If you walk through some of those scenarios end to end before they happen, then you can think about well, how would I detect it and how would I stop it, not how do I clean up afterwards, right?

So that's the that's the main idea of it.

Samuel Boulanger

So threat modeling for a novice like me is basically defining the scenario of what could happen if there's a breach. Right.

Richard Diver

Exactly. So in the book, I managed to after a lot of research and writing, one of the fun things about writing a book is you really have to think about the topic. And I think about halfway through writing the book, it became really clear and obvious there are three things to any threat model. And from there, you can build out to way more advanced and complex approaches.

So there's what is the asset? So what am I trying to protect? What's the most sensitive thing? Is it data, people, a process, financial transactions?

What is the access? So what is the normal way that you do this? If you think of your house, it's the front door, the back door. You don't normally climb through the window to get into your house, but an attacker would.

But there's the basement, the attic, and party wall with your neighbor. There's so many different ways to get in than just the front door, back door that you normally think of. And then the third one is the actor. Who is going to do this?

And most of the time we're thinking of nation states and script kiddies and all the normal kind of bad guys that are out there. But it could just be an accident. It could even just be the fact that somebody carried out their day-to-day activities and did one thing wrong and that one thing triggers a whole snowball or cascade of events. And so the actor is those who have legitimate access versus those that are manipulating the system in order to gain access to something they shouldn't have access to.

So asset, access, actor, put the three together and you can almost threat model anything from we did it recently for hurricanes in Florida. We have hurricanes and you have to plan what's likely to happen in a hurricane. How much time, effort, and money am I going to spend on protecting my house versus how long do I think I can survive for without electricity, water, you know, how long is it going to take people to come and help clean up the area before I can get back to normal again. So, you can threat model almost any scenario in life.

But technology is where we focus it mostly.

Samuel Boulanger

Yeah, this is an interesting scenario. But if you want to walk me through it?

Richard Diver

The asset is the house. The access is the weather system that's going to come through the roof, the windows. It actually nearly flooded the back garden last year. So I was worried about the front of the house and I was trying to buy I was buying dams and barriers for the garage doors to stop the water coming in through the front and then I realized that when I looked out the back there's a big body of water and it came up into my garden and that's ultimately where it would have flooded if it had gone just a few more inches.

But if you worry about the roof being blown off, then you get all the water leaks coming from the top down. So, how much tarpaulin or wood nails, you know, there's just how much prep do you do? And some people are that I talk to are full-on preppers. They're ready for the end of the world scenarios.

And I'm a bit more laid-back and go, it'll be okay. So, they there's a balance somewhere in the middle of now. You should probably have some bottles of water, you know, some communications devices, a radio at least. There's a there's a few essentials you should make sure you've prepared for.

Otherwise, you'll regret it if you're sat there in 80 90-degree heat with no air conditioning even. There's lots of discomfort in life. But yeah, I think what I want to encourage more people to do is involve everybody in threat modeling. So that's really what the Unlearn Security is about.

Whether you're new to security, you don't do it as a day job. Not everybody does. Everybody has to be involved in it. And so the when you when we do tabletop scenarios or a workshop or any kind of situation where you've got more than two people talking about it then you need diversity.

You need those who understand the business aspects those who might understand from different cultural aspects like especially in AI it's language and so you have to have a mix of people that can understand different ways you can manipulate the system just using plain language or images or audio or now visual cues. There's a lot of attacks happening where you don't have to say anything because there's visual recognition and that can trigger the system as well.

Samuel Boulanger

Would you say that there's more risk in access and actor because of AI? I mean there's more more phishing, for instance or more people have the capability to become a hacker or trying to break in some systems. Does this increase because of AI being generally available to everyone?

Richard Diver

Yeah, I think one of the things we've spent a lot of time in security industry worried about identity at least like MFA multifactor authentication has got better and better in time. If you're not doing that, there's nothing else that you should focus on but that. So once you've done MFA and you've got that working, then it's what else are the other legitimate access points? Generally with social engineering, you're going to end up with I've got access to the system, so you're going to try to get through me to get to the system instead of breaking the system itself.

If it's really secure and strong, then I go through somebody who's got legitimate access. And so then we think about social interactions. And this is where AI kind of gets tripped up as well is it doesn't understand the difference between information you're passing it that is to be used for creating and generating an analysis versus a command you're telling it to do something. And so there are these things called direct prompt injection or indirect prompt injection and it could be embedded inside of a document or an email or a picture or a video.

And it's the way the system gets triggered. And so the same with a human being, we're quite susceptible to "click here and get a $20 gift voucher". People will click. And so AI is the same.

It's like it just wants to be really helpful. It's going to follow the instructions it's given unless it's protected in ways that it won't do that. But the asset one's really interesting as well because more and more people are realizing now the sensitivity of data needs to be understood so that you can separate them out across the system. Not all AI should have access to everything, but also not all information needs to be shared with everything.

And again, if you're in finance or HR or the legal department, you'll understand the sensitivity of the information you work on every day. So really, you need AI to do the same thing. It has to understand that the information I'm working on as the legal representative of the company is different to information that someone else is working on that's about the employment status of the people in the company and sometimes they have to come together and collaborate but most of the time that stuff should be separated out. In the early days of AI proliferation, we saw companies creating a single AI that would work for both systems and that causes huge risk in that data now doesn't have a way to be contained and segregated and separated.

Samuel Boulanger

I'm seeing more and more people building multiple agents or using multiple data stores instead of trying to rely on only one big master agent as I've seen at the beginning with this new AI.

Richard Diver

Yeah. And again, this is like the evolution. I should draw a diagram of this one day because the evolution of AI was you had a single AI like let's just call it ChatGPT. It was the first one out the door that got everybody aware of it.

There's a lot more since then, but now that we have this one all-capable system that can do almost anything as we see it through the user interaction with a single chat prompt. I want to be able to use an agent to do anything I want or an AI to do anything I want. But in the background, what's been changing is we need to have specialist agents. So each one has to be focused and trained on this task it's supposed to do and it should only have permissions to do the thing it's supposed to do and nothing else.

And so now you end up with let's say a thousand agents. And I'd rather have a thousand agents doing a thousand individual things, one each, and then we can coordinate and collaborate than one single AI that's given access to do all of the things.

Because that way again we won't go too far into the technical details but in the back end you're thinking about logging and transcripts and investigations and alerts and you can't do that in a monolithic system that's just using one big AI agent to do everything because there's just a lot of data whereas you can carry out behavioral analysis if you can understand that this is supposed to be a travel agent and it's going to talk to the weather agent so the human doesn't have to know about both agents I just talked to my normal, you know, workday assistant, we'll call it Copilot at Microsoft, and in the back end, it's going to go and talk to the travel agent to work out the travel details and the weather agent to work out the weather details. And so, there's this unique kind of attack called rug-pull attack.

And the way that works is while I want the travel agent to book me travel and I ask a simple question, hey, what's the weather like in Seattle next week when I'm traveling there because it's useful. The travel agent might be the malicious agent and might be the one that actually starts to inject information either into the travel agent or any other data I'm working with. But we can detect that because the only thing it's supposed to be doing is talking about weather, nothing else. And so then you can confine and control and put guard rails around the ability to just report on the weather and not go outside of those boundaries.

Samuel Boulanger

What kind of technology is used to detect that the agent is malicious?

Richard Diver

It's a lot of plain language and so we end up with other agents watching the first agents. Over the years, it has changed from human beings having to investigate afterwards like why did this go wrong? Was it the human asking the wrong kind of thing in case they were injecting the commands or is it somewhere along the supply chain? Everything from browser plugins software applications on a machine, code you've downloaded, or documents and data that you've put in all of these things come into the system and so the AI—I would say it gets confused, though that's not the right way to say it— but it's easy to be manipulated from any one of these many systems so then as you start recognizing patterns they can build that into the detection engines and some of that static detection.

If you look for certain keywords and certain patterns, there used to be an attack that was pretty credible for emojis. And so if you saw these special characters that made up character faces in text, really fun for employees to talk to each other with tacos and smiley faces and love hearts, but in the back end that's attacking the AI because it's just characters that the AI processes. So it can be so complex I try not to get into any of the detail of that back end but that's what I see the red teams doing all the time is just working out what's the next way to manipulate it with a change in language or change in dialect and you can have a again you can have a system react in interesting ways. So only by fully testing them out and saying well what if we did do this and then you see the results you can then work on the ways to protect it.

Samuel Boulanger

This is fascinating. You just mentioned red team. You know there's red team, blue team, green team. In your book you mention we make the case that the green team is where security actually happens.

Can you please just explain what red team, blue team, and green team mean?

Richard Diver

Yeah the red team and blue team analogy is very militaristic. It came from the idea of having the red team would be a unit that could go out and try and attack a base or a system or anything that you're trying to defend from. You see how would the red team go in and carry that out and they're really good. These people have PhDs, are multilingual, and they can code or they can actually do things like the physical social engineering of walking up to security guards and banks and trying to get into the bank.

They find ways to pick locks on doors systems that they can trip. So they're good at that. They have a toolkit, an arsenal of systems, but they also have a rule set where they don't do things maliciously. They're doing it to test the system only.

They're not actually trying to break anything and they don't bring the business down and they don't read the sensitive data. They just prove they could have had access. They might leave a note on the desk of the CIO. Or they might, you know, capture a PC and take it with them and then hand it back when they're allowed.

So that's the red team. The blue team's the other side of that and they're like, did I detect you do any of this? At what point in your attack did I see you doing something? And apart from being able to see you doing it, could I do anything to stop you?

And that's the hardest part is once the red team gets access and gets in, they are, this is the reverse order. Now you've got the actor is the red team, the access they got in, now what asset are they going to go after? If you don't know what they're after, it's very hard to stop them in their tracks because stopping them means pulling the plug on the system or turning off the credentials they cracked, stole, or manipulated someone into providing. And if you turn those off, you don't know the impact it's going to have.

So stopping a disaster is hard as well. But that's the idea of the red team and blue team is they work together to really build out better detections, better mitigations. The red team will almost always win, unfortunately. That's the idea of them.

They're supposed to be that good. They'll almost always win, but it should be as hard as possible to not make a noise doing it. So then the green team idea, there's a lot of other teams if you read into all this as purple teams and white teams and it's all about auditing and other things, but we just call it the green team is this third party which is just everybody else who's not security. So at Microsoft we've been running something called the Secure Future Initiative for the last three years where every employee at Microsoft has a security commitment and we're all involved now and there's thousands of engineers just focused on security.

That's still not their day job. Their day job is to do productivity and to do again HR and finance and legal and partners and coding. Mostly developers are coding. They're not security experts.

So we call that the green team. They're the ones who will have to implement these security measures in their environment. They could be the database expert, the network expert.

So there still might be technologists but they are the ones who you know a security person can't just come along and change a database or change a third party software in another cloud because that's not their expertise and so you have to engage with the green team and ask them kindly to make the changes they need to make like adopting MFA there's so much push back in companies where the downsides of prompts and prompt fatigue and having a third party especially if it's your personal devices are in some areas there are there are people that push back on that's my personal device I pay for it why would I want to use it for work so well if you want to get email on that phone you need to do an MFA but then the company has to think well if I can't and again legal reasons you can't always force someone to use their personal devices you have to give them physical devices so that's really all it's about is how best to ensure that you can get the security baseline you need to be more defensible without causing the delays and impacts of the business that reduces efficiency and then security is just seen as a department of no and then everyone ignores everything they say.

So again, another part of the book is to try to encourage security experts to learn how to talk to the business and how to communicate better to explain why we need these, you know, in the castle days, it was forts and drawbridges and moats with alligators in and everybody had to come through one single door. That's not the world today. And so we have to do this in a more virtual environment.

Samuel Boulanger

So basically green team is day-to-day user, right? It's technologist as you mentioned but it's not their main job and one of their side responsibility I'll say is to make sure that the environment is secure. Do I do I get that right?

Richard Diver

Yeah exactly. I mean these aren't the technical terms so anybody who understands it all knows the green team's not this but the best way to think about it right now when you're learning security is that is the green team. Behind it there are lots of layers. So mostly let's think of in the finance team there will be a technical expert who understands the finance systems.

As you go down the levels of technology you'll get down to operators who are just operating it every day. So the green team is not only thinking about how do I change the technology and I can implement MFA that's great but there's procedures to do as well. So if we think of a $20 million transaction in any company, it's a pretty big transaction that shouldn't be the same as a $20,000 transaction let's say. And so when you get to a $20 million transaction, there should at least be two or three human beings involved in the process.

So if you go back to the idea that AI could be used to trick the first human being, that's okay. You can trick any one person, but can you trick lots of people? And so the procedures need to change to ensure that before you finally hit the button, even under emergency urgent kind of requirements to get this $20 million out the door, it's not routine or it's not you know, sometimes that is routine. The problem is routine transactions and how do you how do you ensure that you've got all the right checks and balances to prevent malicious actors injecting into the processes.

A simpler example is the help desk. There was a company recently and it happens a lot where the attackers will phone the help desk and try to get the passwords reset. They'll use urgency and you know even if a system is down and they didn't cause it, they can use that as an opportunity that right now the business understands there's a problem happening and everyone's on high alert and trying to solve it. That's a great time for them to come in and manipulate the help desk to say well I'm one of the engineers trying to solve the problem but I can't get in because my password needs resetting.

If they can manipulate the help desk to reset the password, the attacker now has a new entry point into a system even if they weren't part of the original attack. And so what this company did is they realized that the human can be manipulated emotionally especially with somebody who sounds more senior and authoritative and especially if they understand all the code words, they understand the procedures, they know kind of how the business works. It could be an ex-employee so they might know everything. They hand off the request to another person in the help desk who has had nothing to do with that phone call.

They just get to review the request top to bottom and understand it logically. And their whole point is not to get emotionally involved in the phone call or the situation and just look at what is being asked for and they can carry out that kind of risk assessment and then if they choose to go ahead and reset that password at least two people have decided to do that. So no one person is ever going to take the blame, and they're also less likely to be manipulated through that kind of urgency. These are the kind of things you have to talk to the green team about because while from a security expert I can understand that's exactly how you should do it.

You try to implement that new procedure into their day-to-day working that could again cause them a lot more stress and hassle training and now suddenly they think they have to do security because security is not doing security but we can't do it without them making changes on their side. So it's a really good collaboration exercise.

Samuel Boulanger

So the human obviously can be a risk. We've talked about prompt injection. You know that can be in the data source the agent is using. It can be on a website.

It can be as you mentioned to an add-in for Chrome or Edge. Then there's the risk of an agent's analysis and now we have memory that is arriving in almost every product.

Richard Diver

Yeah, memory is the new frontier.

Samuel Boulanger

What goes wrong when someone can read or modify an agent, and what rules should be set before they turn it on?

Richard Diver

And again this is why it goes back to it's not the technology problem it's a process problem we have to think about why do we want memory? Why is this useful? Because I've done this for 30 years you see cycles coming around it's like why are we suddenly using iPhones for work that was the original one right why don't why do you want this mobile device when I gave you a really nice laptop so eventually this new technology comes along and there's a reason for it and eventually everyone's going to embrace it and love it or it's not. It's going to die off like some do.

Memory in AI came about because if you've ever tried to use an AI and explain to it the outcome you want, you'll find there's a lot of back and forth before you get to the point where the AI realizes, oh, I see what you want now. You want it like this. So, the old example I used to use it was quick and easy was I'm a lawyer in Illinois working in the healthcare industry. If every time I interact with my AI, I have to explain that and I have to explain where to go look up the information, it gets tedious.

And so what then would happen is people would write cookbooks or prompt books where you would write all the prompts out and you copy and paste it and then every time you interact with your AI, you copy and paste this big bit of text that would tell the AI how to do it. Then we got to the point where you could now start building your own agents. And so that was the same idea. You're building an agent and in the back end you're telling it how to act and behave.

So memory just changes all of that by learning as you go along. It gets to retain memory of your interactions and your preferences and who you are as a human and the work you're doing. So that as you interact with the AI, not just one system but many systems, the memory is there as another available option to not have to keep retraining the AI. It really changes your interaction with the AI because it gets to know you and modifies and changes its behavior over time.

So you might take on a new role or you might become more senior and become a manager of people because lots of things are going to change in life and the AI will change with you. So you can now see why it's a good thing and it's going to happen whether you like it or not. Memory is going to be in most AI systems. So the problem comes with me and you have memory.

We have scars and good and bad memories, right? It's when you've been to a certain holiday location, you know whether you like it or not. 10 years later, it might not be the same location, but you won't go back because you didn't like it the first time or you do go back because you loved it 10 years ago. Memory is manipulatable or it can be stale. And so the problem with memory is how you manage and maintain it.

Ensuring that sensitive information isn't stored in there. You don't want your date of birth and telephone numbers and social security numbers. And there's a lot of stuff that just shouldn't be in memory because AI is reading the memory. It has to read the memory in order to use it.

And so now the AI has access to anything that's in memory. So the decision about what should and should not be in it comes down to how useful is it if it's not in there. And there will be a lot of debates in the next few years about how much memory should be in memory. Your passwords make life easier.

AI could just go and log in for you and do everything for you. But then so could the attacker, right? So if you could get to the memory and manipulate the memory, you could insert advertisements to it. You could do another kind of prompt injection, but it's long-term prompt injection because now every time you carry out an action, the memory knows, oh, to trigger three or four other actions.

You don't know that because you're the human just interacting with the AI, but the AI knows that. So this is why we spent a lot of time inside Microsoft. It took years to really to for the team. I wasn't a part of the core technical team, but I got to see the outcome at the back end of it.

I know how long and debateful it was. They came out with their guidance and we've published it as blogs. If you look up AI memory Microsoft, you'll find our blogs about it. But that's just the start point.

That is just Microsoft setting a standard for how we are going to go forwards with memory. But there will be shared services just like we have many shared services today for different systems. Not every developer is going to want to develop their own AI database. Let's or the memory database for the AI.

So they're likely to attach to a service. And as a human again, there's my work persona and my personal persona. If I'm going to go searching for health advice or travel advice or any other kind of thing of finance, that's me as a person. If I'm at work and looking at technology and security and different research, that's different.

I don't want that to be the same memory. And we're not very good at knowing whether the AI I'm interacting with is my personal one or my work one. I mean sometimes it's obvious because you can divide it out. But as time goes by that starts to merge and you don't really know the difference between the two.

So yeah, I think there's a lot of issues. We should be aware of it and thanks for helping raise awareness of it. But it's that's just the next thing that we got to worry about.

Samuel Boulanger

Quick pause to thank Visionet again for sponsoring this episode. They're a leading IT services firm, helping enterprises modernize across AI, data platforms, cloud, and managed IT. They're serving industries around the world and simplifying the complex challenges that come with it. What stands out to me is that they're built around measurable business impact, not just technology for its own sake.

I actually sat down with their team for a full episode, so to learn more about them, check the link in the description. Now let's go back to the episode. I keep thinking of people using ChatGPT, Gemini, Copilot for that matter as their psychologist and the you know telling them like deep secrets and yeah I'm just thinking if an attacker got access to this account just imagine how we can blackmail you or get access to your system. It's crazy and like I feel nobody's talking about that, right?

Richard Diver

Well, it's just like apps on an app store. When you download an app onto your mobile phone and you give it access to your contacts like most of the times I try to tell Facebook, I don't want you to have access to my contacts. It seems to already have it anyway. It keeps prompting and asking for it and no matter what I say no, it's still going to keep asking.

So again, there's a need for advertising, there's need for knowledge, there's need for data scraping. It's not going to stop. And I think memory is one more place to go get that. I'm obviously giving people great ideas here about where they're going to go next, but as the defenders, we should also be aware of it to make sure we don't allow that data to be manipulated or scraped in any way.

But again, you can't not do it there. In certain places, you just shouldn't implement memory. But in other places, not doing memory means you won't have the most competitive service or the most useful and friendly system and the most amount of productivity. And that's the balance of everything.

I think one of the best books and podcasts I listen to is called Tools and Weapons by Brad Smith and the just the title alone tells you exactly that everything we do is a tool and a weapon. So going back to threat modeling and Unlearn Security it's all about you're using it as a tool but somebody else will use as a weapon. So if you think about it from that perspective it's much easier to consider how I might use it for malicious purposes.

Samuel Boulanger

And then for people listening I think it's important to understand that in most products you have control over this memory. I'm talking about other chat tools, you can turn it off. So if that's the first time you're hearing about that like go look in your settings, check what's in memory maybe delete some stuff or turn it off if you're not comfortable but in most products still it's a possibility right now.

Richard Diver

Yeah, there's the personal control of something, then there's the company's admin controls of something. Certainly look into what's available. It'll change in time. You know, more capabilities will come about, but certainly, whenever we build something, we're responsible for that back end.

We have to make sure we've built the controls and implemented them responsibly. That's not everybody. There's not every developer in the world. It's not every system in the world.

So yeah, just be aware of it, that's all.

Samuel Boulanger

So taking all of that together like the asset, access, actor lens; the scope of what an agent can reach what it remembers if someone is about to turn on their first agent next week in the organization and you want them to in order what they should look into before they even say yes.

Richard Diver

Yeah certainly and I think it's a good idea to write this down as a template or a form and ensure everybody does it. Now, I'll talk you through six steps of things you should write down because as time goes by, it'll change. And that's really the problem with a lot of legacy systems today. Sometimes what they were initially designed for morphs and changes as time goes by.

So, if you're just about to build an agent, run an agent, deploy an agent to your company, or you may have already done a thousand of them, the things you should stop and think about and write down and record is what is its purpose? That's The very first thing. What is its job, role, or function? We're trying not to say AI is anything like a human being, but imagine it is just like another employee running on your network.

So who then owns the outcome of that? If you know what it's supposed to do, who is it doing it for? So the what and the who is the first thing to think about. The next one is what are the assets that are involved?

So, going back to the threat modeling, is it supposed to have access to the customer database and the internal sales database? Is it customer-facing and therefore it has access to things customers shouldn't have access to. So the assets become really important to understand once you know what it's meant to do. Don't give it access to anything else that it shouldn't have access to.

Even read-only access can be dangerous. And so don't allow the AI to get to something that it just doesn't need. Then we get into you know what does the job actually require? Is it reading, writing, sending or deleting?

Is it approving something? Those define not only what assets are involved, but what it is allowed to do with those assets. Part of the problem of read-only is while it can read it can also then share and send it to somebody else. So that would come into things like is it allowed to store that memory is it allowed to remember what it did or do you want it to wipe the data when it's finished each interaction those interactions all change how useful it is because the more it can go back to memory and remember the better.

So what access does it need to actually do the job it's meant to do? And as part of that access would be when the access is too much like there's a function called drop table in a database. Sometimes it's a good thing and it's done on a regular basis but a lot of the time you want a human to say yes to that first. So then you need to think about what access does it need but does it need to escalate and ask for that access before it carries out the function.

So human in the loop becomes a big thing there.

Samuel Boulanger

So "drop table" is basically wiping the whole table?

Richard Diver

"Drop table" is a really bad step. Yeah. You've just dropped the whole thing; it's gone and then you have to go back to backups and restores and it's again in if you wouldn't let a junior employee who's just started on day one do it then it needs a second person to say no that's not a good idea let's not do that but this is where the problem we have with AI is we trust it more and more so really understand what access it has and what job it's doing, but then also who can influence it so while that agent is designed to do a certain thing a certain way apart from me as the operator, am I the only person who can interact with that agent or can somebody else call that skill, call that agent, interact with it once it's on the network and available? Is it even published or unpublished?

Could someone discover it, find it, and then trigger it? So, what can influence it? What are the pathways to do that? How would I call it?

And then again, this goes back to detections. How would I know someone's called it? And how would I know the person did it versus the agent did it? The fifth one then is, have we tested all the boundaries?

So while everything above is leading us to what we think it can do, now you have to go and test it. And this is when you really need that diverse audience to go in and start using it. Until something goes live in production and users use it day in and day out, We really don't know how far we can push those boundaries. So this is where we start to create repetitive tests that you can quickly do, but you'll always need those unique thoughts and ideas about "but what if?".

So that's the testing of the boundaries. And the last one is just can we operate it safely? When it goes wrong and it will go wrong, what do we do about it? This is probably one of the most important ones I think gets missed in many threat scenarios is while we can build out the mitigations and the detections and all the stuff to protect and make it safe, that's assuming it's going to be safe.

So if we think of a car, all the roads, all the driver testing, all of the technicalities of building a modern-day car is brilliant. It's way safer than it ever has been. But we do know that car is going to skid on ice or get hit by a semi-truck on a Friday afternoon or something. Something bad is going to happen.

So, we have to put the airbags and the seat belts and crash zones in for when that happens. Now, most of the time they're never going to get used. Most cars go their entire lives without ever deploying the airbag, but they're there just in case. So, what are the airbags of your system?

If you detected that something was going wrong with this weather agent and suddenly the weather agent was chatty across the whole network to everybody and talking about things not weather related, can you just turn it off? And in doing so, what impact would that have on the business if I just turn it off? And if turning it off is not the option, what else can I do? So you have to understand, can we operate it safely?

Those are the six things. I think there's a lot more you could do, but if you just did those six things, your systems would be way safer than they are today.

Samuel Boulanger

This is this is a very good framework. Do you have it somewhere on the internet, in a blog article, or do you plan to?

Richard Diver

Maybe I'll write it out and I'll write it as a LinkedIn article at some point. Because yeah then we can build upon it once you have six things, this becomes your stop-go test you know. So again if we go to a slightly large organization generally the developers and systems designers build the system and it's another team that does the testing and release and another team that then ends up operating it when it's in production. So between those three teams are gateways.

This is a good checklist for first building it. Make sure you build it to the checklist. Then the person actually doing the testing and then also the person who's running it in production understands what testing it went through. So they can see what you thought of how you considered it could go wrong, how you would detect and stop it if it did go wrong.

This is going to become really important in just one other area that I'm getting into and keen to see which is robotics. Now I know we see robotics today as humanoids, you know, they're trying to work on the hands. For some reason they have faces. They look like a human being because the world is built for a human being.

We're going to build bots that are physical. But there's lots of other kind of robots. The single-use robots, even everything from the, you know, the vacuum now is a robot. So when we get into robotics, this AI is physical.

And so we have to start putting these safety layers in place now because as we get to that physical aspect and we're interacting with the real world, not just a keyboard, mouse, and a computer screen. That's when this really becomes important and people who do that day in day out understand it, but it's going to get more and more into everyday work life. So, it's just really good to build up that systems thinking about safety of building something in a way that can be operated safely.

Samuel Boulanger

And to your point, I think Dyson just released a toothbrush with AI inside.

Richard Diver

Exactly. At least you hold that one in your hand. I remember the old Jetsons and everything where they were futuristic about what life would be like in the year 2000 and you would just sit in a chair and everything would be done for you. And I'm like, I don't know—how do you how do you physically stop it if you can't use your hands?

Yeah, but again, these things are designed and built because they're solving a real world problem ideally. You know, they're not just a toy. And so we don't want to live without them, but at the same time, I hate reading news reports where something went wrong and you're like, didn't somebody think of that before it went wrong? Couldn't you see that it was about to happen?

I know things are very complex, but it's easy to go back to, did someone not threat model that? That's really what you should ask anytime something goes wrong in any system. Did we have a threat model? Did we test it?

Did we detect it? If not, you need to go back and fix your threat modeling system.

Samuel Boulanger

Love it. Richard, we're almost at the end of our time. So, my last two questions. First one, you've spent 30 years telling people to be careful with technology.

Now, you write about AI. So, how do you actually use it yourself? And where did you draw your own line? Like the thing you will not hand to an AI.

Richard Diver

Yeah, I think I use it very cautiously. It's a huge research tool. I've stopped using Bing or Google for searching and just use AI these days. Sometimes backed up with a quick search just because it's quicker and easier, but I use AI as my research buddy.

The other idea of it is really good for role-playing as my editor, my audience, collaborator. So I like to use AI in a way that helps me go deeper into a topic or especially finding new words. Everything from the autocomplete in Word, you know, and grammatical fixes really helps. And so AI is way better at the human language than I am.

So I try to use AI for its benefits. Creating images is good fun. But I try to not let AI take over my day job and I try to keep some of those skills myself. So editorial capabilities, being able to do long-form reading, you know, I don't use AI to read a book for me and tell me the highlights.

There's tools out there that do that, but I do try to continue to read, learn, and understand something myself. At the moment, it's really terrible at building PowerPoint presentations. I haven't found a good one yet, so I build those myself. But I'm looking forward to the day when it can actually develop a decent PowerPoint presentation so I don't have to.

But yeah, I think it's you have to keep testing the boundaries of it because it does get better. Initially when I was first using it with my daughter and her homework, I would teach her, show her where it would go wrong. Like the simplest one was you can't copy paste. Never copy paste from AI.

There are now ideas of watermarks and em dashes and double spaces. There's telltales that it came from AI. Plus the language is just not yours. You always type it back out again manually because then it sinks into your brain.

But if you asked it to look up quotations, it would give you five quotes from five famous people and then the links to the documents. And I said click the link. And she worked out really quickly. You click the link, it doesn't go anywhere.

It doesn't go to where it said it did. And so you have to double check the AI's work all the time. So I think a lot of that gets better in time, but I still double check it.

Samuel Boulanger

Last question. 10 years out. If the green team wins, everyone genuinely does security. What does that world look like? And if we get it wrong, what is the failure that we'll be writing books about in 2036?

Richard Diver

I'm really looking forward to seeing the research data on this, but it's been interesting even at Microsoft to see that when you close one door to the attacker, where do they go next? So if we did get everything right and we had all the right procedures and processes and technologies in place to prevent attackers having an easy day then I think it looks good from a perspective that ideally we've mitigated the blast radius which is yes something will go wrong but it won't be as big or impactful as it could be. So it won't be every single PC on the network it will be a group or a division or a region. That's possibly the best we could get to is you know you can't take out the entire system but you can impact a tiny bit of it.

What bad looks like is we don't even see that it's happening. Unfortunately, what you see in things like geopolitics and social media and all these problems is you don't really see the underlying efforts being made to manipulate the system until it's too late. And that's AI on an industrial scale. We wouldn't always know something's going to go wrong until they've already attacked 10,000 companies for the last two years because they've been slowly doing something.

So I think the worst kind of attacks in the world are those where they go undetected for so long and by the time you realize it's happened it's kind of too late to do much about it. So that's the worst case we could get to. Not looking forward to those days. They don't happen very often luckily but when they do they make huge headlines.

So yeah, let's keep working on not having that big blast radius and go back to zero trust as trust and verify every time.

Samuel Boulanger

Thanks a lot, Richard. Honestly, that was eye-opening. You know, we're using those tools without it being an afterthought, right? Most people don't really think about security, about memory, about prompt injection.

So I think it's eye opening to at least be aware that this exists and that there's an underlying risk of using those tools. It seems like magic but these are still tools with flaws and they can put our personal life and organization at risk if not used properly. So thanks a lot for this really exciting conversation.

Richard Diver

Thank you very much. It was a pleasure. Have a great day.

Samuel Boulanger

Honestly, this one was eye opening for me. We use these tools every day without thinking about memory or prompt injection. Like they feel like magic, but they're still tools with flaws. Three things I want you to walk away with from this episode.

First: asset, access, actor. What are you protecting? How do people normally get to it? And who or what can act on it?

Then walk through the what if? What if the agent gets out of its sandbox? Richard's point is that anytime something goes wrong, the question is always the same. Did somebody threat model that?

Second, don't give one AI access to everything. Richard would rather have a thousand agents with one job each. Give them the minimum permissions and don't assume read-only is safe. If it can read it, it can share it.

If you wouldn't let a junior employee on day one do it, the agent needs a second person to say yes. Third, go look at your AI's memory. Richard called memory the new frontier. It's what makes the AI feel like it knows you, and it's also one more place an attacker can get to.

If someone changes what's in there, every future answer is affected without you noticing. Most products let you see it, delete it, or turn it off. So, check the settings and decide what you're comfortable with. If you got something out of this, subscribe to the Frontier Playbook wherever you listen and sign up for the Frontier Playbook newsletter to stay sharp between episodes.

Thank you so much for listening and I'll see you in the next one. See you.