How to not make a power glitch turn milk into a very expensive problem | Ep 5

Mikkel (00:06):
Today, we are talking about operational resilience. And I think we just had a little pre-talk now. And I think this is going to be a really interesting topic, because when you have machines out there, you invest in having machines that work. You invest in having new machines in your production floor. You also invest in having software on those machines. And thinking operational resilience, thinking, how can I keep things running? That is absolutely a topic that then should be top of mind, simply because introducing software, introducing digitalization, it opens not just a door, but it opens an entire wall of new possibilities or new threats, I could, say that may interrupt your production. And I think that’s very important. So, I’m really glad that you’re here, Erik, to talk about this. And for you listening, this is the Food Tech Podcast. And I’m Mikkel Svold and I’ll try to be your host throughout this interview. And I think one place to start is probably what possible blockages are we actually seeing in food and beverage production today. What can block production?

Erik (01:34):
There’s several things that can block a production. But if you look on the software, that can be perhaps a bug, but it’s tested a lot today. But what we see that is the system needs to run it. And some of the areas, there’s talking a lot of cybersecurity, and NIST 2, and so on. But one of the most thing is that there can be a problem with the power for a short time or a little longer, but there can be change in the power and the government says we should get used to it. So, it’s to say, okay, do we have a power supply, a UPS on the servers?

(02:20):
Because the servers, when you’re digitalized, the software is running on PC or servers. So, how can we be sure that it will work, even if for a very short time? So, that’s one of the areas. And also, if you have an iPhone or Android, you used to get all these updates. And some of the updates is just going on and then the phone is up again. But you cannot do that on servers, because if you are making a cheese production, and it needs to install, and be out in 20 minutes, then the process from milk to cheese has started and then you have one big cheese in the tank. So, you don’t want that.

Mikkel (03:05):
Yeah. You explained this. It’s a quite literal thing, right?

Erik (03:09):
Yeah.

Mikkel (03:09):
So, when you say you have one big cheese in the tank, it’s actually like a big, creamy Havarti sitting inside of your tank and you’ll have to-

Erik (03:18):
Have to dig it out.

Mikkel (03:19):
… shovel it out.

Erik (03:20):
Yes. Yes.

Mikkel (03:22):
And that’s because you delayed the process because of a software update, for instance.

Erik (03:27):
Yes. And that’s also one of the difference, because you’re talking… A lot of people have heard the word IT and then they hear the word OT.

Mikkel (03:38):
Operations technology.

Erik (03:40):
Yes, exactly. So, this is in the production. So, if you’re working in an office and something happens, you cannot make an invoice or print a document in five minutes, okay, I can do something else, get a cup of coffee. But in production, where you have a production ongoing, it is very important that you don’t have this. So, one of the way to secure your app running is that the software needs to be updated, but in a controlled way. So, we see a demand from the market that they want redundancy. So, instead of one server, you have two server with a software on. And then you can take one server out, make it update, put it back, and take the other out, and do it.

(04:31):
So, just when you are on an airplane, it’s not only one engine, you have two engines, you have two pilots and so on. It’s the same. So, to have that on the server. But then we’re also looking, okay, what’s the next step? Because if you now have power supply, of course, two and perhaps get the power from two different areas, two suppliers, but you also need to look on the cables. So, where are these cables go on and to see, okay, how far to be secure that it will be up running.

Mikkel (05:08):
Yeah. I think before diving too much into all the, nearly, I was just about to say technical details of securing this, I think let’s step one step back, take one step back. What does operational resilience even mean? Because I think we need to get that established first before we ask people to mitigate it. What does it mean? And if you were floor production manager, what does operational resilience mean?

Erik (05:44):
I would say that, yes, I know I can continue production.

Mikkel (05:48):
That’s what it means.

Erik (05:52):
No matter what or whatever. Of course, if no energy at all, then I cannot produce. But to secure that I can run a production. And if I cannot run production, then my system will be ready when I can again. And I have collected all data so I don’t have to throw some of my product out or at least only the few of them.

Mikkel (06:16):
And why is this so important?

Erik (06:19):
It is because it cost a lot of money. The product cost a lot of money. The machines can be defect. If you are running a process on very much heat and then you certainly cannot cool it down or there could be so many effects, explosions. The labor people working there can get the accidents and so on. So, it has a much bigger impact on a production than-

Mikkel (06:56):
Than if it was an office building?

Erik (06:59):
Yes.

Mikkel (07:00):
Yeah. In your experience, how often do these incidents actually occur? How often do production halt without being purposely interrupted or stopped?

Erik (07:19):
You see, a lot of companies has this redundancy. But there is analysis of what is the typical cause. And that is actually that they’re happening something about the power. That is the main issue. But then we’re looking on cybersecurity or there can be that the hard disk fails, or there can be… because hard disk is like on a vehicle, something that you use all the time. So, you set up systems today that you have more disk in. So, even if one is corrupt, then you can take it out and put new in. So, when we are visiting a potential customer, we see, okay, where are the weakness, typical, and how are they secured? And of course, you can start with a basic one and then you can move upwards. And that’s where you look, okay, how secure do we want to be, just like an insurance in your home or on your car, and find ways, the fix point.

Mikkel (08:27):
I think that’s a very pragmatic way of looking at it. So, instead of saying we need to be completely resilient against any attack, any incident, anything that might go wrong. And of course, there’s not only being cyber attacks. It can also just be a power pole that falls in the wind. That could also just interrupt production. How do you assess the risk? How do you know how resilient you want to be? Because it feels like one of those things where you can keep on investing forever.

Erik (09:04):
Yes. And if you talk to a customer, say, how much will an hour cost if you not can produce? Or how much will you lose if the power or the system goes down? Bang. Now, is there anything that will need to be fixed before you can start up again, for example? And then you’ll have this amount of money, which impact will it have on your customers and so on? So, to evaluate the risk. And that’s actually also one of the things in the NIST 2, to make a risk analyze and say, okay, how secure do we want to be? And when you have that compared to the investment, say, okay, do we have a backup? And if the backup can restore and we could be up running tomorrow or four hours, is that okay? Then it’s that way you should secure.

(10:04):
But if it’s cost money, then you’ll perhaps go to next step and say, okay, if you have one server with a critical, how much redundancy can we put into one server? Or say, okay, we want two servers, we fail over the next step. So, we have a template where we go through and say, okay, how much does it cost on each step with how much capacity and so on? Because as users today, you don’t see it. You have perhaps a laptop and you save the documents in OneDrive or Google Drive, and then it’s Google Drive and Microsoft’s problems to have all this redundancy. But at food repair and food production, you need to have it local, because of the speed and also security. So, then this task gets moved to local.

Mikkel (10:54):
Is this the responsibility of the production manager or is it IT department?

Erik (11:01):
It is a mixed. If you look under these two, it’s a board responsibility to have calculated the risk and say, okay, this risk is… we don’t think it’ll happen, so it’s okay, we do like this. We know it’s a risk, but the cost will be too high.

Mikkel (11:20):
To mitigate the risk?

Erik (11:21):
Yeah. But when you have made that kind of risk, and as you mentioned, we have perhaps some equipment that is very old and we can perhaps not buy a new one and it’s something we want to use. Of course, you can use it. But from a security perspective, you need to isolate it in the production area on the network. Just like if you have a room around it and only one person can go in, so you isolate it. And that’s okay. But if you then need to have one person to go in and look for it, it needs to be away from the internet. Because if you open that one up for the internet, then you can have a vulnerability, or what it’s called.

Mikkel (12:14):
Vulnerability.

Erik (12:14):
Yeah, vulnerability. And if something happens, then if a company, typically you have a cyber insurance and they actually get an accident, it’s just with a fire, so then you have no insurance. So, it is very important that if you want to set up a secure system and you think, yes, now it’s secure, and we go out, and then the everyday starts. And in a production, everything happens, also what you don’t expect. And you need to fix it and fix it now. And then suddenly, you perhaps do something. And that is very important that you close that door again or are aware of it. So, it’s not enough just to say, okay, we have done it now. It’s like being a-

Mikkel (13:13):
Yeah, like we’ve done security now?

Erik (13:14):
Yes.

Mikkel (13:15):
That’s not enough.

Erik (13:16):
You get a report, yes, we have the security now. And then every quarter, walk through, have a still. Is it looking the same way? If you’re working on an office and say, everybody, please put your dishes in the dishwasher and if you come one week later, perhaps there is something on the table, then you see it. But in your network cable, you don’t see it. That’s the difficult one.

Mikkel (13:46):
I want to ask you, when you talk to companies, do you typically experience that people know this? Do you typically experience that they have already given this some thought, how to secure your operational resilience? And hereby, I mean the software bit of it. So, I would imagine that, say a production manager, I would imagine that they more or less know how smoothly the machines are running, the actual hardware. I would presume that they would probably know. Maybe data-based, maybe gut feeling based. But do they also know all of this, how smoothly the software runs? How secure is the software? Do we have the right? Who has keys to what systems? Is that a typical thing that people have already under control? This may be a leading question, I don’t know.

Erik (14:57):
Some have. Some have or also some of the big companies have. But if you are in the production, what is most important for you? People come and ask questions. What about this? Can we deliver? We have this needs, we do this and this. So, there’s a lot of requests all the time, but this one is not making the request. That’s something you need to do by yourself. And if you have a production and you’re looking on a monitor with a scale and see how it’s going, you will not look on the IT monitor. So, what we see is that a lot of companies want… just like in the office that you just save on OneDrive, could you take care of our IT? And could you come and make these visits? Because suddenly, there’s a machine not running. We need an external to come in. He is new employee. We create him as a user. Do we remember to remove the last one? And there’s a lot of things. When something goes wrong, you start, I need to do this now.

Mikkel (16:05):
Yeah. You solve the problem immediately.

Erik (16:06):
You solve the problem, because that’s what you’re used to. I have problem, fix it fast because the production needs to run, it costs money. But do you have the time to clean up afterwards? And you don’t see it. So, I have seen when I visit a customer that your typical now put in a firewall between the office and the production. They’ve done it all. And suddenly, somebody had changed a rule. So, they had a lot of rules. What can talk to what, that’s the perfect one. I mean the last rule of 34 was star, point, star. And that means everything comes and everything goes through. And I can imagine that could be from the CFO or somewhere, that they need some statistic data for a board meeting. “I need it now.” And they couldn’t find out all the rules. They set that up.

Mikkel (17:09):
So, you insert that little line of code?

Erik (17:11):
Yes. And you got the report, and ran to the office, and here it is. You didn’t clean up and you don’t see it. If you have unlocked the door from the-

Mikkel (17:22):
But then let me ask you, what do you do? How do you prevent this? Because it feels like it’s one of those things where maybe they had pragmatic… that financial or that CFO, he had to have that report. He had to be able to get those numbers, so it was necessary to do that little pathway. How do you make sure that you have processes that then deletes it again? Because like you say, then one minute later, everyday life sets in, ordinary production sets in, another production issue comes up, something else interrupts, and then you forget. And I think it’s one of those small things where those kinds of holes in the cheese, no pun intended, but those kinds of security breaches, they must be all over because of small… then we just had to do this and then we had to do that. But how do you make sure that you have a process that turns it back again?

Erik (18:28):
Yes. It should be something inside that company, but how many have the access to that? How many is educated to that, so that a few people, if you need to make some changes? But if you need this report, you don’t connect through that. There is best practice on how to do it.

Mikkel (18:52):
Because you get it in another way?

Erik (18:53):
In another way. Or you make it okay if that report… I don’t know if it was a report, but it was a scenario. But then you set up a way, how can information from one area to another goes in a secure way? And to look on what are the needs and how can we… again, on the levels, how much security do we want, because how much does it cost if something happens? Then you set that up. But one of the thing in NIST 2 is, do we have physical access and how many does have access? They didn’t know who has changed this and we don’t know how long or whatever. It could be the day before. It could also be three months before. We don’t know that. But to make some procedures and say, okay, just like a machine, when we are running a production, we know what to do. We also need to be aware of the OT or the IT, but we have procedures around that.

Mikkel (20:05):
Yeah. And I think I just want to tie it back to the operational resilience, because then what happened? Nothing happened. The little, what was it, dot star, dot?

Erik (20:18):
Yeah.

Mikkel (20:22):
Let’s just say it sat there for a year and it didn’t do anything to operations. Operations ran smoothly. Maybe even more smoothly. So, what does this have to do with keeping your operation resilient and keeping it running? How are those two connected?

Erik (20:42):
Oh, okay. Yes. If nothing happens in the plant, it was okay. But if they get an attack, then you haven’t closed your door and then it could spread to the whole area, instead of have divided into a small area and then you could fix the problem. So, it’s what you try to do, that if you are attacked, you don’t attack the whole company, but just small areas. So, this is one of the things that they do is that they isolate to small cells, so it’s only this cell that could have that impact.

Mikkel (21:38):
So, keeping things separate, basically, so you don’t make a catch-all kind of incident?

Erik (21:45):
Yes.

Mikkel (21:47):
Where would you start? So, again, this is one of those… and especially I think people who’ve heard talks about the NIST 2 and all that comes with that, it feels like a huge thing. It feels like something that… so many things that you have to change in the way that you operate right now, in order to comply. When you come in and when you talk to, it could be dairies, it could also be other, it could be breweries, it could be any food and beverage facility, when you come in, what are the basics? Where would you begin, just to give people somewhere to start?

Erik (22:35):
Yes. There are two areas. There is the one with all the paperwork. And that can be lawyers or consultants to walk through, a lot of paper that is going to happen typically in the office. And then there is practical parts. And that’s where we come in. And in NIST 2, it’s actually a lot of good advices to follow. So, the first thing I would do, even if NIST 2 wasn’t there, but to say, okay, what is the typical for have my plant up running, is, okay, what about the power? Yes, I have a UPS. How old is it? Because a lot of UPS does not have lithium-ion batteries.

Mikkel (23:34):
A UPS being?

Erik (23:34):
It’s the power supply, so it’s actually a big battery just like in the electric car.

Mikkel (23:38):
Got you.

Erik (23:39):
And if the power somehow went out in two seconds, then you have it on the battery or 10 of 15, or half an hour. And then it can close the system down on the correct way, if the power is not coming. So, that’s the first one. And then, of course, the server, how redundant is that with power supplies, disks and so on? And when it stopped running, what about my backup? I see a lot of companies, they don’t have backup, actually. So, do I have a backup? A lot of people don’t care about backup, but they like to restore.

Mikkel (24:25):
Restore from what you say?

Erik (24:26):
Yes. And do you have a restore local or do you have it in the cloud, but have you tried to see if you can restore it? So, that’s what we do a quarterly basis, to take one out, try, spin it up, and see, yes, it’s running. Because and then you can secure it several ways with a 3, 2, 1, so you have one in cloud, one in local, and one on the machine. But to fix that, that costs very small. But in the production, you perhaps start up with a PC for test, and suddenly it is in production, and you forgot to take backup.

Mikkel (25:09):
Yes. So, even those two points alone, that will really keep production running in a way where you are securing yourself against power outages and also cyber threats, basically?

Erik (25:23):
Yes,

Mikkel (25:24):
Just those two?

Erik (25:26):
Those two. And of course, with the backup, there is that you cannot change the backup if it’s in the cloud. There are some ways from different vendors, so you can do that. Yes. But that’s the basic, but I see a lot that don’t have it. And they add more servers and forget to put it in because now every day work starts. So, just to be questionnaire, and be aware of it, and get it every quarter. And then, of course, if you are in NIST 2 company, there is a lot of risk, questions and a lot of things you could do. But it also what I said that is good advices. And that is the one that we said, okay, make segmentations of your network, so if something happens in one room, it will not spread to two other areas. That is one thing.

Mikkel (26:23):
Yeah. And I think that the picture you painted to me before we started this talk. So, when you have a building, you also have fire doors between the office building and the production building. You may even deliberately have them in separate buildings with 20 meters apart, just in case a fire breaks out one place it doesn’t spread. You should think about your IT and OT systems in the exact same way. You need them to be 20 meters apart, right?

Erik (26:55):
Yes, exactly. Exactly. So, when we have written in newspapers that a company has been hit, how big impact? There is a lot of companies you don’t read about, but they could fix it in a small area perhaps. And the next thing is, okay, if I want to secure around me, who is visiting me. Just like in the corona, we should not visit anybody because then we was not getting ill. When you have a lot of equipment, you need to have your vendors to update or access remotely. How can they still access without…? Could bring anything in. So, there is a lot of software.

Mikkel (27:48):
Intentionally or not intentionally?

Erik (27:50):
Absolutely not intentionally, but they are also visiting a lot of other companies. So, if the system is that I plug in and have full access, then something could be transferred to the laptop, and transferred to the next, and wait three months to be activated. So, how can we secure that they can access the equipment and help us with support, but not transfer any files? So, there is software for that. And I think to prevent that is also if you look on the statistic, it’s about 25% of the attacks comes that way. To have a secure way of handling this, you need the vendor. And if something actually comes in, do we have what we call end-to-end protection? In old day there was antivirus, but more advanced today that it can shut down the PC or the system, because we are getting more efficient the more we digitalize, but then we are also more vulnerable.

Mikkel (29:03):
Open new flanks for attacks, basically?

Erik (29:06):
Yes. So, to secure the next level here and then you have in many ways a good place to be. And then you can say, okay, now I want to monitor. Something happens because I can monitor. As you mentioned, that if the fire door is open or not. But how do you monitor the cable? So, what we do is for the road, as you said, you are perhaps few people on the plant, everything is automized. Well, how can we monitor that the backup has been taken? How can we monitor that a disc is not corrupt? If we are making cables in a ring in the plant and one cable is-

Mikkel (30:00):
Broken.

Erik (30:01):
… broken and still running, then you will not see it unless you get an alarm that one cable is bad. So, to have the monitor, so that’s the step I go through.

Mikkel (30:12):
Yeah. And then if you were to take one last thing that you recommend. So, make sure that you have your power supply in order. Make sure that you have your backups, because even though backups are not interesting or cool to talk about, restoring is pretty cool to talk about. And then segmenting your different areas into separate bins so they don’t overlap, so you can’t get access if you shouldn’t have access. And then where do you anchor all this? Because the NIST 2 says it’s anchored at the board or at the C level.

Erik (30:53):
Yes. You’ll need to educate people to be ready. So, you’ll need to say, we have an incident plan. Who has these roles? And do we have service levels agreements with our vendors? So, if we have a problem, have we taken care of that? And you’ll need to have the management to review always in the future. Are we the right place? Will we still do what we have written down? So, you need to be aware.

Mikkel (31:32):
The NIST 2 was activated, I believe it was in June this year, right?

Erik (31:39):
Yes.

Mikkel (31:41):
So, people should already be ready, so to say they should already have this. Is that also your experience that they do?

Erik (31:51):
No.

Mikkel (31:52):
No?

Erik (31:52):
No, no. Of course, some have, but a lot of companies, it’s not fully implemented. And I saw an interview on the television that it can be very big penalties. But they said, we will not go out and write penalties now. We want to talk to companies and say please move forward. But they need to be more speed on. And the more you read about it, the more you read in newspapers, it will be moving just like the GDPR. And then there will be some standard thing.

Mikkel (32:33):
And I think also now again, before turning on the microphones, you mentioned that the number of attacks, cyber attacks has more than doubled since last year. And I think the more we hear about those kinds of incidents, and sadly, of course, that’s also connected to geopolitics and all this. And it’s a sad thing and we shouldn’t open it. But I mean, the more we see incidents like this. Just recently, yesterday or two days ago, we had the Amazon breakdown, which basically took down, well, two-thirds of the internet or something like that. So, the more we see that together with the NIST 2 placing of the responsibility at the very top of organizations, I would expect this actually to accelerate. So, it’s securing your digital parameters, securing your systems more.

Erik (33:31):
Yes, that’s correct. It will move forward. But I see if you always say, NIST 2, NIST 2, but I would like to say, how can we secure more uptime to have it in a broader way? Because that’s where I can get the money out of it. If I think, oh, I need to do a lot of papers, it’s just problems. But if you change the perspective a little and said, okay, how can we secure more uptime? How many breakdowns or stops did we have last year? Could some of those be avoid? And so, look it in that way and say, okay, then we are ready for the future.

Mikkel (34:12):
Yeah. So, basically, instead of looking at it as rules and regulations, look at it at these changes and these ways of handling your digital infrastructure. Look at it more as a way of securing future income, basically.

Erik (34:31):
Yes. And there is a catalog of good advices. And if you think you’re not that big, you could say, okay, this risk I will take because it’s too expensive, but I’m aware of it. So, it’s actually a catalog of what should you be aware of.

Mikkel (34:47):
Yeah. Erik [inaudible 00:34:50], thank you so much for joining the podcast today. It was very informative. And I think it’s also one of those things where production managers, and not even just production managers, but anyone outside of the IT department, they’re a little bit overwhelmed by the things. And I think it’s a really good point, making sure that you have your risk assessment. So, to do your risk assessment and then you know which risks are you willing to take and which are you not. And then kind of go through it from the basics. So, you mentioned the power, the backups, the segmentation, and the anchoring, and the leadership, and that sort of thing. Getting that right, because that will take you 80% of the way and then the rest will come. I think that’s a brilliant way of looking at it. Erik, thank you so much for joining.

Erik (35:42):
Thank you for being here.

Mikkel (35:44):
And to the listeners out there, if you enjoyed this episode or got something valuable out of it, please do share it with a colleague or a friend, or whoever you think might also get value from it. It really helps us spread this knowledge. It helps us spread the ideas that keeps the food and beverage industry running forward. So, please, do share it with someone you know. And of course, also, go and follow the podcast and click the like or connect button. It really also helps the podcast lift off, I would say. So, that would be a big help. If you have any questions or any topics that you think would be interesting to get our take on or someone you think we should be talking to, do reach out to us at podcast@au2mate.com. I think that’s it for now. Thank you so much for listening.

Erik (36:46):
Thank you.