Episode 06 is here, with Akamai's Dave Lewis and host Thomas Fischer talking global security advocate to global security advocate on the underlying issues that enabled WannaCry's success and the steps organizations should take to prevent similar attacks in the future. Tune in via SoundCloud below or listen and subscribe via iTunes.

Highlights from this episode include:

• 0:35 - Recap of the WannaCry ransomware outbreak and the patch management challenges that left many industries vulnerable to attack
• 3:25 - The technical debt problem facing organizations reliant on legacy systems
• 7:20 - How third parties like vendors or suppliers can drive risk and the shared responsibility for protecting against future attacks
• 12:45 - Best practices for securing your supply chain

Intro/outro music: "Groovy Baby" by Jason Shaw, licensed under CC BY 3.0 US

Transcript

[0:00:08.8] TF: Good morning and afternoon everybody. This is episode 6 with the Digital Guardian Podcast. We’re here today with Dave Lewis. I’ll let him introduce himself in a few minutes. I’m Thomas Fischer, a global security advocate for Digital Guardian. Today, as I said, we have Dave Lewis. If you’d like to introduce yourself, Dave.

[0:00:30.4] DL: Sure, yeah. Hi, I am Dave Lewis. I am also a global security advocate, but in this case for Akamai Technologies.

[0:00:36.4] TF: Dave, it’s been an exciting week. Well, at least an exciting seven or eight days, hasn’t it? We’ve basically launched cyber war–this is the biggest cyber-attack history.

[0:00:47.8] DL: You almost said cyber-W word there.

[0:00:50.0] TF: Yeah, I did almost. It was a slip of a tongue (laughter). It’s been an interesting few days, at least for me. As we were talking earlier, it’s insanity, because we’re just repeating the same mistakes we’ve done over and over again. I don’t know what you were doing back in the day, but when SQL Slammer hit, I was basically managing teams trying to get them to basically close off every SQL server that we had in their network.

[0:01:16.3] DL: Yeah, I remember that. I actually stood in the room not more than seven feet away from David Litchfield when he gave his presentation that ultimately led to Slammer. I remember seeing him, he was at a whiteboard. He’s writing all over the whiteboard. I was like, “Oh! This is not going to end well.”

[0:01:33.6] TF: Yeah. It’s funny enough, I was catching up on Paul’s Security Weekly Podcast this morning. Back in April, Paul and Carlos Perez were discussing exactly what happened on Friday almost to the tee talking about the SMB vulnerabilities and that we should be turning it off.

[0:01:51.7] DL: It’s frustrating because on one hand we can jump up and down, yell and scream. It’s like, “Oh my God! Why didn’t you do that?” The reality is it’s far more complicated than that because all these organizations, they may require or they may not have the skillset to make sure that these things are turned off. They may require it because of some other dependency on ancient software that’s been running in their organizations. Those are a couple of examples, but that’s one of those problems where it’s like, “Oh my God! You should have had-” “Yes, but…”

[0:02:21.7] TF: Exactly. Some of the biggest industries in Europe including where I live in U.K., which was the NHS. The issue too you have systems that are dependent on not being updated. I used to work in the oil and gas industry as well and some of the production control systems, you basically didn’t touch because any touching of the system would mean a lot of downtime. Go ahead.

[0:02:47.5] DL: Sorry. I was going to say I worked in the power system for nine years so I understand that fully. One particular system that our guys scanned with Nmap of all things hit one of the – The name is called lockd, and that thing panicked and a caused a cascade failure up the stack and took the whole system down. That’s on a simple scan.

[0:03:07.1] TF: Yeah, I know. It can be ridiculous. Unfortunately, some of these — You just don’t have the choice right, and what are you going to do? You can segment off that equipment but then that means segmenting not just that equipment, but a whole infrastructure because people have to access it to actually manage it and to use it or to recover information that they’re getting from that device.

[0:03:28.6] DL: Right. The thing is, if they built it correctly from the outset and did implement network segmentation, it would limit these sort of problems. Yes, you may have to require those particular pieces of software, or whatever it is, that are beyond repair at this point to be still running, but if you have it segmented it off, at least you’re limiting the risks to the rest of your organization. This is one of those frustrations I have that it’s like we’ve gotten to the point now where it is almost cheaper to tear down and rebuild than it is to continually manage some of these applications or software or whatever it happens to be, because you’ve gone from, from technical debt perspective, you’ve gone into a depreciation where you’re in negative value. So now it actually cost you more to maintain it than it does to replace it.

[0:04:15.1] TF: Agreed. What about equipment that’s massive, like MRI scanners? Something you just can’t tear the whole thing down and put it back up. It’s a function of millions of dollars, isn’t it?

[0:04:27.7] DL: Exactly. That’s why I was going to back to earlier when I was saying it’s not as simple as we would like it to be. Working in power systems, we had devices that were in the field that had been there for 30 plus years and people are like, “Oh, just patch it.” I’m like, “This is not going to end well. There are no backups for these systems. You have to deliver them on an 18-wheeler …” That’s one of those things where there are levels of complexity that people don’t really appreciate with this sort of thing.

[0:04:53.2] TF: Yeah. Some of those systems you just can’t bring down, because they’re vital to production carrying on or they’re vital for the chain to keep running. There’s little, very, very, or no downtime.

[0:05:07.0] DL: Yeah. Every organization I’ve ever worked in, with the exception of the current one I’m in, has always had the summary intern that’s built application-X that’s running on an old beige desktop that nobody knows how to maintain, update, or patch, let alone get into it, because they can’t remember the password, but it’s mission critical.

I ran into that over the years so many times, and this is one of those things where this leads into this technical debt where we have problems exactly like this that not only affect production and patching and all the rest of it, but it can ultimately affect the supply chain because if something gets introduced into one of those systems that are accessible, it could have a cascade failure across your network.

[0:05:46.1] TF: You mentioned technical debt, but how do you deal with technical debt when – One of two things. One; you just don’t have the budget because you can’t, you’re five or six billion pounds in debt anyway. Also, some of the infrastructure that you know you need to upgrade, you can’t upgrade because upgrading it will create too much downtime and make the actual – In the case of hospitals, if you’re taking downtime, you’re potentially risking the life of patients or life of injured people coming into the hospital. How do you balance that technical debt versus constraints like that?

[0:06:25.7] DL: It really is that you have to look at what are the risk exposures you have to your organization and then start at the top of the list, with the worst one of the bunch, and work backwards. There’s no easy answer to this sort of thing and that’s one of the real difficulties is exactly that. There is a frustration level because there is so much debt in some of these organizations that it is almost insurmountable, but I’m never one to give up.

Also, when you’re dealing with something like a hospital type of organization – I spoke at HIMSS this year down in Orlando and a lot of these conversations were around this, “How do you do it?” When I was in power systems, when we were deploying new systems, we would actually do it in parallel and cut over when we sure that after all the testing we did that it would actually be resilient. We’d do it in an off-peak situation and make sure that when we did cut it over, that we had actually a proper fallback plan in the event things went completely spiffy.

[0:07:19.7] TF: That’s the best strategy, isn’t it? In that technical debt, we come up to the vendor, or the supplier. If you think about part of the problem with the technical debt in the field of medicine is that hospitals are running equipment that cost them millions and millions of dollars to buy and very rarely get upgraded.

I saw recently that a few of those vendors are actually announcing that they’re going to be pushing and they’re going to providing updates to these old computer systems, most of them running Windows XP – one of the biggest targets for last week’s outbreak. How do you see that fitting in at that supply chain aspect, because it is part of the supply chain, right? Somebody who’s supplying you a device or piece of software or a piece of equipment. How do you see that fitting into managing that technical debt?

[0:08:09.8] DL: As a consumer, and in this case, those are organizations are the consumer, they actually have to hold the vendors accountable and they have to make sure that as the consumer in this particular scenario, that they are completely spelling out what their requirements are. They have to be crystal clear with what they’re trying to do, because a lot of vendors will come in with their shiny box and say, “Oh, look! It has blinky lights.” It’s not going to help the situation in that case.

It’s really incumbent upon the customer to say, “Look, we need this, this, and this.” If those vendors aren’t able to deliver that, there are vendors that will, and that’s just it. There’s no good answer and it’s a little bit frustrating in that regard.

[0:08:48.8] TF: Yeah. I agree. You get into situations where you can put as much pressure as you want on your vendor, but at the end of the day, you need to manage that pressure. I saw a lot of – There was a lot of blame throwing after the proverbial crap hit the fan. There was a lot of blame throwing. Was it Microsoft’s fault? Was it the NSAs fault? For me, it’s just – Go ahead.

[0:09:15.2] DL: It’s all of our faults.

[0:09:15.9] TF: Exactly. That’s what I was about to say.

[0:09:19.0] DL: Oh, sorry.

[0:09:20.9] TF: No. It’s great. Because if you think about, it’s like we know – I think we both know – we’ve been playing at this game for long enough. You can’t pass that buck, right? You’ve got to take responsibility for your technical debt and you’ve got to take responsibility for managing the systems that you can and can’t upgrade.

I remember when I was part of – I was lead. I was the equivalent of SME for the desktop infrastructure for the company I was working for. We got set targets at that time where we had to basically a pool of over 80,000 PCs. We had to have monthly update rate of over 85% to 90%, and we were tracked on that. We had that tracked in terms of our objectives and in terms of the CIO dashboard.

When I hear what went on last week, I’m like, “Don’t we do that anymore? Does companies just not bothering with their patch rates anymore?”

[0:10:14.1] DL: You know what it is? Is I think that somewhere along the lines, we actually forgot how to be adults in the room. This is really something that is a bit troubling, because we look back 10, 20 years ago when we were starting out in this whole field, there was a limited number of things that were available for us to work on. As time has gone forward, all the applications has sprung up, that there are so many different angles that you can look at it.

The base of people that are available to work on these things is not kept pace with the number of opportunities in different types of software, different types of hardware, all that sort of thing. There is a necessity for more people to work in this field, and that’s one of the things I worry about is that in addition to the technical debt, we’re actually losing the base because I got some gray in my beard now and there’s more every day. Eventually, I’m going to go off and own a bookstore somewhere.

[0:11:11.6] TF: A bookstore?

[0:11:14.2] DL: Something non-technical. One day, that will come.

[0:11:21.3] TF: Yeah, I’m running right behind you. I’m looking at islands right now. It’s like I just want to go to an island where there’s no technology. Just get away from it all.

[0:11:32.8] DL: That’s actually pretty much where the bookstore would be.

[0:11:39.4] TF: There is another aspect to the supply chain that I was thinking about when we were mentioning SQL Slammer, because this was an incident I can’t really discussed, but I worked with – A long time ago. I worked with a customer who told me a story, it involved SQL Slammer essentially, but it was part of that supply chain issue. We have customers, they’re part of a supply chain or they have a very extensive supply chain in the back to be able to support their business and to be able to do what their business does.

We have a situation here where does our IT security policy, does our audit reach need to go beyond our own closed loop? Do we need to take precautions and ensure that those suppliers are practicing the same level of security that we expect for ourselves? What’s your kind of vision on that? How do you think a company should proceed to ensure that that whole line of – Because we’re no longer just one closed entity in terms of a business. A business now relies on so much more. What do you think a best business practice – Like security practice company should have to ensure that that threat doesn’t come through the supply chain?

[0:12:58.7] DL: This actually goes right back to what I was saying earlier about spelling out your requirements. For example, if you’re onboarding a partner company to allow them access to your own network, you want to go through those steps and make sure that they had done X, Y, and Z before you ever connect them to your network.

One company I worked at years ago, we actually had about 300 plus vendors that were tied into our network. I made this silly mistake of putting on my hand going, “Where is all the documentation for this?” I found out that maybe if we’re lucky, 20% of them were actually documented at all in any way, shape, or form.

We had companies that had untethered access into our network, and this is a real problem. Then when you look at the software they buy and purchase and put in to your own environment, you want to make sure you’re vetting it because if you are outsourcing your code to country X, Y, or Z. My thing for the day is X, Y, Z. If you’re outsourcing code development, you want to make sure that you’re vetting that code that is being delivered. You want to make sure you’re getting exactly what you asked for and not having either by happen stance or by negative intent, that they’re not introducing something into your environment, whatever it happens to be that could have a malicious purpose either by accident or by happen stance.

[0:14:13.6] TF: Making sure the whole thing works as you expect it to work.

[0:14:17.9] DL: Exactly, yeah. If you’re running a financial institution, you’re developing your code offshore, are you just going to blindly take it and install it? It wouldn’t be very swift, wouldn’t it?

[0:14:26.3] TF: No. Then, again, you never know it’s swift. It’s quick.

[0:14:33.3] DL: [inaudible 0:14:33.5] problems.

[0:14:39.7] TF: The things that we say. The things that go on in the world. One of the aspect that really got to me was the fact that we still don’t know what was the exact point of entry or what the vector for infection was on this thing, but one of the things that got to me is that it is definitely moving around over the internet via RDP or via SMB. Most people have confirmed that.

Akamai does a lot of tracking of trends of network activity. Why do you think people are still opening those ports on the internet? What’s your vision on that kind of – Why would you have SMB open on the internet? I can understand RDP because support people want to connect and things like that. SMB?

[0:15:38.6] DL: Even then, there’s no good reason for it. For example, when I did a quick search on Shodan, when this all broke. I found 1.3 million instances of port 445 open to the internet. Now, there’s no guarantee that that was SMB itself. Still, I’m willing to say it was better than one.

You know what it is? There is where I worry about going back to technical debt. It applies to skill sets as well. We have people that are very good at doing – Maintaining systems or whatever it happens to be, or writing code, but they won’t necessarily have any inkling towards security, because they’ve never been taught it. Never been exposed to it until something like this happens and then they become very acutely aware of it.

[0:16:21.7] TF: Yeah. The other thing that I see too is that our systems have become so complex and something might have been brought up. We’re back to the technical debt, yes, but you might have brought up something, some application, or some new system up at one time or some POC that you started to run and you opened all your firewalls and you tried to make everything work properly. You have to make it work quick, because you had to start because the business was requiring it quickly and you just forget about it because maybe you don’t follow the proper change of management protocols. Maybe you don’t follow the – It’s just a test, so you just put it aside and you get submerged with other problems that are more serious and you just completely forget about it. Then on day, as you say, you wake up because it’s that vector infection. You’ve just been completely hosed because you left that port open at one point in time.

[0:17:14.1] DL: You’re saying the firewall rule, allow any is bad?

[0:17:18.3] TF: No. I have that on my firewall, don’t you?

[0:17:22.8] DL: Yeah. You really nailed it. There has been so many instances where organizations where we had POCs in the past, and I would go through the firewall rule set and I’ll be, “What’s this?” “Oh, that was a POC we did four years.” “Ha? Why is this still here?” “Oh, yeah. We forgot to take that out. We meant to, then we went for lunch.”

[0:17:45.7] TF: Then it was the weekend.

[0:17:47.6] DL: Yeah, exactly. It’s frustrating.

[0:17:50.7] TF: Yeah, I did appreciate it, but they launched it right on Friday. It’s just like –

[0:17:56.7] DL: The thing that actually really caught my attention was they launched it on Friday. Didn’t Donald Trump sign his cyber-security Order on the Thursday?

[0:18:03.6] TF: Yeah, I mean it’s not even really a cyber-security order. It’s order to actually investigate what needs to be done for cyber security, isn’t it?

[0:18:10.2] DL: I know, but I watched X-Files too many times, so I enjoyed that too much.

[0:18:17.7] TF: Yeah, oh God! The days we live in.

[0:18:24.2] DL: I know, right? I broke him.

[0:18:27.6] TF: I wonder what’s next.

[0:18:30.3] DL: Oh, never ask that question. That, and never do the math.

[0:18:34.1] TF: I can’t do math. That’s why I have computers.

[0:18:37.0] DL: People say, “How many years have you been doing X?” I start doing the math and then I cry.

[0:18:42.7] TF: God! I think I’d be doing this longer than my son has been alive.

[0:18:49.6] DL: Yeah, no. I think you have to.

[0:18:51.7] TF: Yeah. I think the first time I got into security was probably in late 90s, or mid 90s – No. Mid 90s. That’s 1990, not 1890.

[0:19:05.7] DL: Oh, my goodness! I actually have you beat. Oh, wow!

[0:19:08.4] TF: Really?

[0:19:09.5] DL: I may or may not have been participating in bootlegging video games in the 80s.

[0:19:14.9] TF: Yeah, okay. Yeah, I was doing things that I will not talk about when I was still in high school as well. It involved dialers and things like that, and CompuServe, and CompuServe access nodes. Do you remember those?

[0:19:30.9] DL: Yes, I do.

[0:19:33.8] TF: Those were great, man. Anyway, we’re coming close to the half hour. It was fun and entertaining, Dave. I know you need to – You’re at a conference right now, aren’t you? Something like that.

[0:19:45.9] DL: Yeah, I’m actually speaking at NolaCon right now.

[0:19:48.6] TF: Oh, cool. Well, that’s a great thing to do. Keep sharing that knowledge. It was good to have on board on this podcast, Dave. Thanks a lot for your insights. I hope to see you and hear from you soon.

[0:20:04.6] DL: Will indeed. Thank you, sir.

[0:20:05.9] TF: As for our listeners, if you remember, last podcast on episode five, a little contest was launched to win a BSides London ticket. If you’re a U.K. listener and you want to come to BSides London, go back to podcast episode five and listen in and find out what you need to do to win the ticket.

Have a good day everybody, and thanks a lot, Dave, again. I hope to have you on board in the far future.