Welcome to ACP

Tom Morgan 0:36

It's time for another ACP. Alex is with me. Hello, Alex. How are you?

Alex Baker 0:40

Hello, Tom. Yes, I am good, thank you. What about you?

Tom Morgan 0:43

Good. yes, very good,

Recent Cyber Incidents

Tom Morgan 0:44

thank you. it's been a busy couple of weeks, not just... I mean, it's always busy, but it's been busy in the IT world particularly, I think, and we're gonna be talking about that a little bit today. we're gonna be talking about the recent cybersecurity incidents that have happened, how different organizations have responded to them, and how that affects contact centers or how contact centers can play a part in how organizations respond. 'Cause, yeah, I was thinking about this the other day and I thought this would be sort of topical. we've had quite a lot going on recently.

PHP Vulnerability Fallout

Tom Morgan 1:15

in the last couple of weeks we've had the big PHP to Shell vulnerability issue. I started off, I dunno about you, Alex, but I kinda started off, I was on holiday so that didn't help, but I thought, actually I didn't think it was quite so much of a big deal as it eventually turned out to be. I thought it was just, "Oh, it's another vulnerability in Node or in PHP or whatever. Yeah, we'll patch it and we'll carry on." And then it was sort of national news and it was big organizations, government organizations were turning their websites off, and I was like, "Oh, I... This is a bigger deal than I thought," right?

Alex Baker 1:45

Yeah, agreed. it made me think of, I'm sure there was a, a linked news story where there was, it came up like a cybersecurity expert, said he'd only just heard of PHP or something. I saw that. Yeah, kind of, kind of overstating or understating the importance of it to the internet everywhere.

Tom Morgan 2:03

Yeah. Yeah, definitely. And, and, and it's interesting as well because different people have got had different responses to it because, certainly I run some PHP, but it's all done through, like, there's a company that hosts it, right? So, they reached out to me before I reached out to them to say, "You don't have to do anything. This is fully hosted and managed. We'll fix it. Don't worry about it," which is great, you know. it turns out that maybe not all government organizations do that. It looks like quite a lot of people are hosting their own PHP. no particular comment on that, but it does mean that when something like this happens, you are the person responsible.

Outages Hit Contact Centers

Tom Morgan 2:33

and I think it was a kind of a big deal because, you know, government organizations turning their websites off is like, you know, the primary purpose of a government organization is to be reachable by your public, and if you turn off the, you know, the one way, people can reach you, I mean, it's, it's not great. And also the knock-on effect for things like your contact center and your email inbox are huge, right? People can't reach you.

Alex Baker 2:56

Yeah, I'm sure there were some busy days at contact centers across government organizations. But I remember being on a, a call, I guess it was thinking back when it, when it was all, all kicking off. But, what the customer that we were talking to sort of having, just jumped from a, an emergency incident call. cause probably 'cause that very reason, you know, people can't get on to do something in a portal, and then the phone lines are suddenly getting hammered.

Tom Morgan 3:18

Yeah. Yeah,

IoT Attacks New Normal

Tom Morgan 3:19

yeah. And then recently, certainly in the UK and Europe, we've had, a resurgence of the refrigerator problem, with sort of IoT, temperature monitoring devices being misused and heating food everywhere up so it's unusable. and so it just, it just sort of feels like these kind of targeted attacks are on the rise, which makes sense. You know, we're kind of at the point where all of the mythos, that kind of level of model that went through the protected organizations, it's either leaked, which is very possible, or it's got to the point where those models are smart enough that even like, you know, the open source, open weight models that you can get can be taken and abused much more easily than the hosted ones, because you can, you know, obviously remove all your guardrails and things. and so we're starting to see those exploits n-now tick up And it's hard to know, you know, are they actually on the increase or is our monitoring of them on the increase? but certainly in terms of outages, it does feel like it's becoming more regular and even, I don't know, e-even sort of the people on the street, if you like, non, non-nerds, like normal people, have started to attribute small periods of downtime to, "Oh, maybe it's a hack," or an AI, you know, a-an AI hack or something, rather than just systems down or, you know, some, you know, something, something non-hack related, which I thought was interesting. It's becoming like the new normal.

Alex Baker 4:41

Yeah, yeah. It's where people think of that first 'cause it's so prominent in the news, I guess, isn't it? All the models breaking out of sandboxes and things

Tom Morgan 4:49

Yeah. So it was kind of in that environment of like this is becoming ever more common. And so I think it's, I think it's a thing that lots of organizations are thinking about right now. Uh, you know, what is our security response? What would happen if this was to happen to us? and that's a challenge because if you think about,

Contact Center in Response

Tom Morgan 5:05

Well, I, I thought this was interesting because obviously the the, the biggest challenge is that you can't, you can't get your message out to customers that something's gone wrong if you just turn everything off. but also you can't get your message out to your people. So many of us work from home now, and we don't work in offices. Offices are easy. You just, you know, walk up and down the floor chatting to people, telling them what's going on. But when everyone's working at home, and then they're all connected by your systems, your communication systems, if you have to turn those communication systems off, how do you reach those staff? And so that got me thinking about actually the contact center could be a really or should be a really important part of your cybersecurity response. and that's not just working out how the contact center updates to tell people that we're closed because of a, you know, some cyber incident, but your data's safe, don't worry about it, to actually how do you inform your staff what to do next. So I thought that was interesting. I think there's probably an interesting argument to have, which we should have, around whether this is me jumping on the bandwagon of news and trying to force contact center into a solution.

Minimum Viable Company

Tom Morgan 6:05

but I do think there is a place for a contact center in your... lots of people now, well, I say lots of people, CTOs, CEOs are talking about this idea of a minimum viable company. Like, what's the minimum amount of technology you need to keep your organization running? and having that sort of ready to go, if you like, or a version of it or a, a clone of it or, you know, or a twin, available elsewhere. But I think, I think a contact center, both an external and/or an internal contact center, is important and should be part of that minimum viable company. But I don't know what you think. Maybe I am just sort of ambulance

Modern Continuity with Connect

Tom Morgan 6:38

chasing,

Alex Baker 6:38

I think, I think also, yes, it's, yes, it's important, but also it's so much easier to make it happen these days as well. Thinking back to, you know, maybe 20 years ago probably, and you talk to companies and I remember doing some work for a big insurer in the UK, and they had business continuity contact centers. and I do remember sort of going into, you know, a huge, completely empty office and installing phones that probably nobody would ever go and sit at and, and use. But that it was there so that they could ship people out of a particular physical contact center and move them to this other building if necessary. and of course the associated cost and maintenance and licensing that goes with that, it must've been, must've been huge. Whereas it's so much easier to stand it up, at the drop of a hat or have something there ready to go, without all that initial outlay. I think it is, it's a much more real prospect, I guess.

Tom Morgan 7:36

Yeah, definitely. especially sort of, well, Amazon Connect with its consumptive charging, it will literally cost you pennies a month to maintain a number and no usage costs until you need it. so, so bit of a no-brainer, I suppose, really, to have one ready to go. but I think the trick though is having it ready to go. It's no good waiting until you need it and then be like, "Great, I'll stand one up and somehow get this phone number out to everyone." You know, you've got to have done that before, and made sure that all your people have got the phone number, whether that's, you know, you put it in the welcome pack or you print it on the ID card or whatever it is, you know. yeah, email it to them once a month or something. But yeah, you've gotta make sure they have it and they know, they know what it's for and they know when to use it. so there's some processy stuff there. But yeah, I think definitely, like Connect makes that affordable in a way that wasn't before. but it's It's still challenging though because... Well, if we think about the sort of customer-facing side of things, that doesn't work for customers, right? It'll work for your internal staff, you know. If you can't log on, if there's a problem, then call this number, but it doesn't really work for your, for your kind of, for customers because they only know the, they only know your number.

SSO Single Point Failure

Tom Morgan 8:39

and even on the sort of staff one, I flip-flopped a bit because we have so many different ways of communicating. We have, you know, email and Teams and, Slack and all these other things, right? but actually in a funny sort of way, we've been trying to do a good job in, on the security side of things and be good, you know, corporate individuals by signing up to SSO wherever we can. But it, actually, what that's done is it's made Microsoft 365 or Entra ID, you know, your SSO, whatever that, whatever your SSO provider is, has made it like a single point of failure, which is, which is a challenge because if you, you know, you, you might need to turn that off for whatever reason. and, and that might be that because you're under some sort of attack, and actually the easiest thing to contain the attack is to just disable all the accounts 'cause you've no idea where it's coming from until you work it out. So you just need to, you need to do that. But then the challenge is everything stops working, so you have to have this like totally separated communication method out there somewhere, and it seems like using a phone line is quite a good way of doing that. but yeah, no, it is a it is a challenge for customer facing because what are your options then? If you've got a, you know, you've got a contact center set up somewhere on a phone number and then something bad happens, what should you have done beforehand to make that process of switching over to like something completely different? Is there something you can do ahead of time? What does that even look like? You know, do you... Yeah, I don't even know.

Rerouting Numbers in Crisis

Tom Morgan 10:00

Cause number porting's a thing, right? But it's, that's quite a drastic

Alex Baker 10:03

Yeah, I'd, I get, I'd say number porting probably due to the time constraints isn't something that you wouldn't, you wouldn't just kind of go and port your numbers, in the event of a, a crisis unless it was a, I guess, a longer running one. but yeah, your lead times are

Tom Morgan 10:18

Because it takes a long time.

Alex Baker 10:20

a-around that.

Tom Morgan 10:21

Right. Okay

Alex Baker 10:22

Yeah. so what you could look at i-is some sort of, carrier level diversion potentially. it, that does rather assume though that you have your phone numbers, held in a, in a different place maybe to, let's say you had a, a, a sort of production Amazon Connect instance, but you need to repoint to one which is, maybe not SSO in your example. you'd have to have the numbers hosted at a, at a carrier rather than sort of in Amazon Connect. if you, you know, you... What I'm trying to say is you, you wouldn't be able to sort of log into the main instance to do a, a redirect from there. so yes, it, then if you, if you held them at carrier level, a lot of the carriers have, have tools that you can use to log in and do the redirects, assuming of course, that they're, they're not kind of linked to your corporate email, which maybe you can't get into.

Tom Morgan 11:11

Right. Yeah, absolutely. Which, almost certainly they will be like you, you know, at the point at which whoever it was in your company set up the account with the carrier. And again, these tend to be very murky details in an organization, don't they? Like from our experience, it can take a while to find the one person that knows how the carrier dashboard you know, who, who the carrier even is, let alone like who is the admin for the carrier. and that is something that's worth finding out ahead of time, not, you know, when you can't reach anyone

Test Your Continuity Plans

Tom Morgan 11:40

anyway.

Alex Baker 11:40

And it's a great reason to think about, perhaps we're going slightly away from topic, but, in, within your BCP, your business continuity planning, you should be doing regular checks and runs through what does happen and who owns what and, and where do we, where do we go to, to, to activate these contingency plans.

Tom Morgan 12:00

Hmm. Yes. Yeah, it's got me thinking actually. I don't think something like Microsoft 365 has a... Actually, I don't even think it has a top-level kill switch particularly. I know you can disable accounts, but it'd be really good for continuity testing, to have a kind of like just, you know, like set times, you know, to set up a thing where you're gonna disable everything for six hours and then re-enable it, you know, including yourself, because it's no good everything being fine because you can keep going because you've, you know, you have to, you have to plan for the worst. that might be an interesting thing to look into. But yeah, like, yes, that's definitely a thing people should test because it's... Well, it's just like backup and restore. It's the same thing, and we were equally as bad with doing backup and restore in the old days before it all went to the cloud as well. Like, you know, your backups are fine, but how's your restore? You know, don't know. Never tried it, so yeah.

Alex Baker 12:50

Yeah, definitely.

Global Redundancy Options

Alex Baker 12:51

probably worth also mentioning in terms of, in terms of general resiliency, just 'cause I know it came up in, in one of the, the recent, announcements, but that's, Amazon Connect global redundancy. and I know it tends to, tends to be the bigger companies that consider it. and there is a cost associated with it, but it does, you know, a-as a, an out of the box or relatively speaking, feature of Amazon Connect. It does let you have a, a sort of hot standby contact center in a different region that you can have. and it was the numbering thing that made me, made me reference this. but you can have the numbers kind of, across the two regions, so w-with, with little in the way of sort of number redirect. So if it's Thinking about the problems that, AWS have had in the Middle East recently, where they've, they've lost a couple of the availability zones there. If you had a, a bigger event like that and you weren't able to access Amazon Connect in the main region, you do then have a, a, a ready-to-go hot backup

Tom Morgan 13:52

Yep. Yeah, that's true. Yes, and that, it's really, yeah, and I know this comes across as scaremongering. I don't really want it to, because it's really easy to look at the uptimes for Connects and AWS and think, "Well, actually, they're so good that I don't think I need to worry about this, right? Because, you know, it's really unlikely they're gonna have an outage that is gonna impact our contact center." It's like, that's probably true. That's not actually quite what I'm talking about. I'm talking more about a target, you know, a targeted attack to you, the organization, that means your staff can't log in, therefore they can't use the contact center that is perfectly fine and running okay. It's just no one can log into it, you know? So it's, You have to sort of plan all this stuff out and look at every part of the puzzle and work out where the weak links are. and annoyingly, it's probably not your cloud providers that you can go and beat up for a better SLA. It's probably other stuff, annoyingly. So,

Alex Baker 14:45

great. And like, yeah, as I said, probably a bit of a, bit of a tangent, but I thought it was worth

Tom Morgan 14:50

totally worth talking about. No, it is, it is totally worth talking about because actually that is a totally valid, you know, scenario for some organizations though that have points of presence across the world. so yeah, no, totally.

Alex Baker 15:01

I think probably the over-overarching point is make sure you've had a think about all of these scenarios, I guess. So what happens if, an AWS region fails? Also, what happens if, like you say, your SSO fails and nobody can log in? All these different failure points, it's, it's worth thinking through what happens, i-in the event of each having a, a failure

Tom Morgan 15:23

Yeah, totally. Yeah, totally.

Internal Hotline Demo

Tom Morgan 15:25

sort of riffing on this idea and sort of knocking around some ideas about, well, how would that look like for your internal, like, you know, customer-facing is relatively straightforward to work out. I think you you come up with your message, you publish a number to it, you divert the number, you know, off you go. the internal one is interesting, though, because, you, yeah, you, you sort of gotta get a number out a-ahead of time, and then it's a it tends to be quite a fast-changing picture. You know, things that over the course of a day, the message might change from, "Please don't log in anywhere," to, Only use the VPN," or, "Don't use Teams and email," or, "Just use Slack," or whatever it is. so being able to update people inside your company, in a sort of timely manner is an interesting sort of thought experiment. I did a demo using Connect of how that might work, and sort of really briefly, there's two numbers. It's a very, it's a very basic kind of Connect concept. It's a single flow and then two phone numbers. but if you're the admin, you dial one of the numbers, and there's a pin code that it walks you through, and there's an AI agent on the other end of it, an Amazon Connect AI agent. you validate the pin code, and then the agent will get you to just sort of say in your own words what the, what the current state is. You know, what do you want your staff to do? Is it, like, don't use laptops, don't use a VPN, don't use Teams, just sit on your hands for a bit, and I'll give you an update, you know, in three hours' time. And it'll turn that into a concise, fairly professional-sounding message, and you can okay it, and then it will hold that. And then there's another number that anybody can call that's not protected in any way. I suppose you could pin protect it if you wanted to. You know, if you're a big enough, important enough company that, like, people from the press and stuff might get hold of the number and start phoning it, you might wanna pin protect it. but that just reads out, essentially reads out the message. and then there's also an AI agent there that you've preceded with a knowledge base, for answering common questions. so things like, you know, the questions everybody has in an outage like this, you know, am I still gonna get paid, or what do I do now, or all that sort of stuff. So put that together as a, as a demo. I've done a short video on it as well, so that'll be out at some point, and we'll link to it in the show notes as well. but it wasn't a lot of work, I guess, is the big takeaway. Like, and it's sort of worth having in the back pocket and crucially making sure all your staff and all your IT people that you want to make those key decisions, making sure they have those numbers and those pins and everything before you need it. I think that's probably the biggest thing that I've said three or four times now 'cause it's no good having it, and then no one knows it's there.

Alex Baker 17:55

Another big sort of DR and BCP planning piece, isn't it? Make sure everyone actually knows what they've got to do, what they've got the number they've got to call

Tom Morgan 18:03

Yeah. And, and actually, funnily enough, like if you're not super worried about AWS uptime in regions and stuff like that, you could put it on the same AWS account as a separate Connect instance, with a phone number and everything. Because once it's set up, you don't need to configure it, configure it, so it doesn't matter if you've lost your SSO access, it'll still run. You still, you access it via a phone number, you know, a PSTN phone number with a PIN number. So actually you can use it to update it and everything. you don't need to sort of change how it operates. So it's fairly low admin overhead as well in terms of something that might be just be worth setting up on the side as a, as a Connect instance. And also you don't have to have Connect today as well. It's completely separated from your existing live contact center. This is just a completely separate add-on, that you could put on the side. Might be worth, worth people thinking about.

Alex Baker 18:51

Yeah. The, the whole, the whole sort of dial-in thing, that again re-reminds me of, you know, w-working in contact centers quite a few years ago where we used to do that probably a bit more than w-we do these days. But having it, having an admin line where they could, they could call in, they could close phone lines remotely from there. They could put ad hoc messaging on the, the lines. I guess because everything's a bit, you know, so much more a-accessible these days and, and probably a bit more, freedom for, for, contact center admins to, to go in and change things on the, the website, maybe we do that a bit less in general.

Tom Morgan 19:27

Yeah. And I suppose, yeah, maybe that's a Maybe that's a, a top question that if you run a contact center today, you should be asking, Do we have that feature? Is it enabled? Do I know what the PIN is? Do I want to keep it enabled?" It's a risk factor, obviously. It's a risk vector, sorry. but actually could it be quite useful if I can't log in to make the changes I need to make? You know, if there's an outage, can I change the welcome message to my customer-facing one? Yeah. Because there's a sort of another side to this as well, that if you, if you are unfortunate enough to be the victim of some sort of attack, and you think there's a risk around, data breach and that, you know, people's, people's data might have leaked out and all that sort of stuff, you do have a responsibility to tell people that that's happened, and that's really hard if you've lost all your communication channels. so actually, you know, it's supposed to be as soon as possible and where feasible within 72 hours. So if it's a multi-day, you know, if you're out for a couple of days, maybe putting something on your contact center welcome message i-is a good idea because people are gonna start, you know, wanting to know, have that as an answer or wanting to know what's going on. so that's also, that's also worth talking through. Like I say, it didn't take me very long to put this demo together, and there's definitely other things you could do. There's a bit of a balance between not making it too complicated, and keeping it sort of simple to use. But I think you had a really good idea actually that I thought was worth talking through

Proactive SMS Outreach

Alex Baker 20:47

Around, yeah, so I guess when we were talking it through before the, before the recording, we were talking about the possibility o-of, of making it more of a sort of proactive communications method, maybe through outbound campaigns. so if you were to have a, a, a sort of fairly basic outbound campaign set up to do outbound SMS, possibly if you were to maintain a, a campaign list with all of your, your staff mobiles in, you, you could, you could feasibly, and I, I don't see why you couldn't do it as a sort of a, a, a dial-in option like you've set up already. but you could, you could phone in, talk to the AI agent and say, "I want to send this text message out to the staff," and have outbound campaigns do that text-based outreach

Tom Morgan 21:33

Hmm. Yes. I, and I'll tell you another reason that's a good idea actually, because in any of these scenarios, in order to reach your staff, you need to have a list of their personal mobile numbers, right? so that's challenging anyway because how are you gonna store that securely according to GDPR, like, safely, you know, data protection, all that sort of stuff. Normally, you put it on your corporate systems, but obviously that's out of play. So what are you doing? Like, your options are fairly limited there. You know, you either sign up completely separately with another account somewhere and keep it secure, you print it out and put it in a safe or something. but actually storing it as an outbound campaign list in Amazon Connect is a pretty secure place to keep it, because you can set it up under your SSO, all the rest of it, like you say, you know, when the times are good, but it's there accessible in a safe way when you need it. So, actually as a data store for personal mobile numbers, it's not a bad place.

Alex Baker 22:26

Yeah. It could be, could be quite a, a quite a cool way to approach it. I guess the, the, the usual, you know, caution around things like service quotas that, that might, even in a, a general outbound campaign's deployment might catch you out. But it's, again, back to that, that careful planning, isn't it? Make sure you've tested it. You, you've, you know, you know all the staff know what they're, they're doing in order to make this happen, and you do test it probably a couple of times a year just to, to make sure it's still there and working like you think it's going to be.

Tom Morgan 22:57

Yeah, no, definitely. Definitely.

Homework and Checklists

Tom Morgan 22:59

We don't always end the episode with a homework section, but this one probably does have some fairly good homework sort of takeaways and actions. definitely go and have a conversation, you know, with your IT team. If you are your IT team, have a conversation with yourself or your CEO about resiliency and about what the backup plan looks like, what the minimum viable company looks like, and, and consider that messaging for internal as, as just as much as sort of customer-facing. consider this idea of building out a mini Amazon Connect instance. and we've done a couple of, PDFs as well that might be useful to sort of help, just bring a bit of structure to the planning. the first one is called Prepare Now, and it is literally that. It's a list of sort of checklist items to go through and things to think about now whilst the sort of times are good to try and work out and play out some of those scenarios, for what's going on. And then the other one, which hopefully you won't need, but it's worth downloading now, is a sort of the first 72 hours. You know, what do you do in hour one? What do you do in hour two? What do you do in hour three? Like, it literally sort of walks you through, because I don't know, Alex, you probably... As someone who's been on the receiving end of having to manage or be a part of incident responses, you like, the checklists really help because everybody's running around like a headless chicken, shouting at everyone and unless you... It's so easy to forget stuff, I think, is the, the thing, isn't it? Because you're trying to think of, you're trying to do, like, parallel streams in your head and think of so many things that, yeah.

Alex Baker 24:29

And, and if it's a, a bit, you know, off the, off the beaten track in terms of you don't, you don't deal with it day in, day out, these kind of things, yeah, you do sort of... They, they, they slip your mind, don't they? So it, it helps to have it all structured to work, to be able to work through.

Tom Morgan 24:44

I'm, I'm absolutely gonna forget the reference, but I think it was a hospital. So, checklists as a thing, even for things that you do every day. So like, pilots do this, right? Pilots have a checklist before they take off, and pilots take off hundreds and hundreds of times a year, right? But they always They still run through a checklist because accidentally forgetting one thing in the checklist could be really bad. and I think it was a hospital went through the same process for A&E or something, like something really similar. They, they looked around at different industries, and I think it was the aviation industry they picked up on, and they implemented this for, I think, for handovers, maybe, like A&E handovers. Whatever it was, but, and they saw like a, a, a positive change in, you know, in their stats as a result of it. So, it's sort of... It's one of those things that makes sense when you say it out loud, but it's, it's worth thinking about. Like, a checklist is a really good idea for things where if you miss something out, it's really bad. Mm-hmm. Yeah.

Alex Baker 25:36

advice

Tom Morgan 25:36

Yes. So, so yeah, go find those PDFs, and, we'll put a link to them in the show notes, so you've got them. And, consider them and just, yeah, just have a think about, you know, what will hopefully not happen to you, but does seem increasingly to be happening to more and more people.

Wrap Up and Subscribe

Tom Morgan 25:51

All right. Well, that's a fun note to end on, isn't it? we could, we could talk about this all day. but it is time to bring this episode to an end. Thank you very much, Alex, and thank you all for listening. Be sure to subscribe in your favorite podcast player. That way you won't miss the next episode. Whilst you're there, we would love it if you would rate and review us. If you have colleagues that you think would benefit from this content, please let them know. To find out more about how Cloud Interact can help you on your contact center journey, visit cloudinteract.io. We're wrapping this call up now, and we'll connect with you next time