Global Data Vault’s January webinar takes us back to the basics with a topic that’s actually anything but basic–ransomware and how to avoid it. Today, we’re sharing our recent client experiences with ransomware, how these companies knew they were attacked, what the common attack strategies were, and how we were able to help them based on these experiences. And, we’ll share the steps you can take to avoid becoming a victim of ransomware.
If you are a victim of ransomware, we’ll cover the steps you should take immediately following an attack, and we will provide a DR Plan Template and Ransomware Checklist. The speakers today are Kelly Culwell, our Service Delivery Manager, and our Operations Director, Steven New.
Watch the video or jump to the areas in the transcript below that most interest you:
- What is the true cost of ransomware and how prevalent is it today?
- How do you know when you have been ht by ransomware?
- Recent ransomware attack methodology
- Common ransomware attack factors
- What preventative measures can be taken to prevent being breached?
- What are some good IT housekeeping practices?
- What can you do if you get hit by ransomware?
- Read real customer stories
- Are there common factors in ransomware attacks?
- What are some preventative measures against ransomware attacks?
- Are there added protections that can be taken?
- Maintaining good IT housekeeping practices
- What can you do if you get hit by ransomware?
- Ransomware detection and analysis techniques
- How to eradicate ransomware?
- Recovering from a ransomware attack
- What do Global Data Vault do?
Kelly
00:04:45
As everybody knows, ransomware is serious. It’s more prevalent today than ever as we can see in the current statistics slides, the number of ransomware attacks nearly doubled in the first half of 2021. They say attacks happen about every 11 seconds, and whether you know it or not, your firewalls, perimeters, and businesses are constantly being bombarded. Typically, what happens with ransomware is they ask for ransom, right? That’s where it gets its name, but what people don’t understand is how great the cost can be for these incidents. In 2018, a ransom demand could be anywhere from $5,000 in 2018 to $200,000 in 2020. Then in 2021, there was a ransomware demand for $40 million, the largest ever.
Those are astronomical numbers. Obviously, it’s going to depend on the size of your organization and what the hackers think they can extract and gain from you. But there are a lot of underlying costs that people don’t always take into consideration, such as reputation, lost employee productivity, any kind of other insurance, or just plain old downtime costs that cost you money as a result of the attack.
Kelly
00:05:54
So how do you know when you’ve been hit with ransomware?
Typically, you’ll get a ransom request. You’ll see something on a screen. Somebody will send you an email telling you that you’ve been hit and give you instructions. They might say something like, we’ll contact you in 72 hours for the money and they’ll want to be paid in something hard to trace, like Bitcoin.
You will start getting alerts on systems. You know, we expect to see an alert or here or there, but if all of your systems suddenly start reporting as offline, that might be a good indicator that something else is going on.
One thing that’s pretty common is not being able to log into resources such as your computers, servers, or firewalls. Maybe people can’t get on the internet because DNS is down. Users are unable to log into their virtual environments to see what’s going on. A lot of systems are attacked and restricted from use, which obviously creates bigger problems, and that’s the whole point.
Kelly
00:06:54
Recent Ransomware Customer Stories
Recently we have had a few customers hit with ransomware and almost all were variants of Conti that got them. It’s been pretty much the same attack surface, where access is gained to a customer’s domain, admin credentials, or some type of elevated credentials. They’ve done this through spear-phishing attempts, malicious websites, to plain old “hey-click-on-me” type attacks. Once they have the domain admin credentials, the malicious software or attackers were able to gain access to the backup servers because they were on the domain. Then they’re able to grab ahold of their backup files and the cloud backups.
In addition to compromising everything in the production environment, they use this method to get backup files, cloud backups, et cetera, effectively meaning customers can’t restore their data. These customers varied in type. We had an oil and gas company. We had an accounting firm and we had a mechanical contractor, obviously being very respectful of their identities, but my point is, it doesn’t matter who you are. You are vulnerable to ransomware. Thankfully we were able to get all our clients back online with minimal delay.
So Steven, the clients above were all hit in more or less the same manner. Can you tell us a bit more about this common attack vector?
Steven
00:08:26
Common Ransomware Attack Vectors
Yes, we’re seeing domain administrator accounts being breached. The attackers come in using the domain administrator accounts, and then log in and run scripts across your network that encrypt everything. These scripts are smart enough to detect if you are running backups, and the issue that we’re running across is if the threat cannot encrypt the data or cannot read it, it will delete the file. That’s by default. So what we’re saying is it goes in and tries to read files so It can encrypt it and then delete your data. What we’re seeing on the Veeam side is the actual backup data, since they can’t encrypti, it is being deleted from the repositories.
Kelly
00:09:29
That’s pretty dangerous stuff because obviously if a company can recover their data, they wouldn’t need to pay the ransom, so they’re eliminating that as far as preventative measures go. There are a few things that we can do with that, so let’s walk through them.
Steven
00:09:50
What preventative measures can be taken to prevent being breached?
Well, first of all, you want to limit the use of your domain admin privileges. Be sure that you do an audit of your accounts to make sure that you know which accounts have domain admin rights. Domain admin is the key to the kingdom, so anyone who has access to domain admin has the key to the kingdom, and they’re going to be able to do anything on your network.
A preventative measure you can take is to maintain security accounts. You also want to use service accounts. An example is when you implement Veeam, I would suggest that you use a service account, and that is the only thing that service account is used for, just for Veeam.
Another thing is to have an anti-virus program that blocks scripts.
What we’re seeing is that these scripts are running as PowerShell via API. They’re going in and running through your system, so if you have an algorithm-based anti-virus program that is script-blocking; it may not stop every attack but that will be a good step in the right direction.
You also want to make sure that your Veeam server is not on the domain. These programs go through and scan your active directory DNS to find any systems to reach out and see if they are accessible via a CIFS share and encrypt that data. Another thing that I’ve thought of while we’re looking at this is that you shouldn’t name your Veeam server, “Veeam Server” or your backup server, “Backup Server.”
It’s pretty obvious that if I’m a hacker trying to get into your infrastructure, the first thing I’m going to look for is anything with the word backup in it, and when I find it, that’s the first thing I’m going to kill. You also want to lock down PowerShell scripts and ensure your endpoints are protected as well, even those conference room PCs, kiosks, or anything that can communicate on your network has an algorithm-based script blocking anti-virus solution on it. We’ve already discussed service accounts and only use those for a specific purpose.
Kelly
00:13:00
Those are good points. There are a couple of other things that we can do. We talked about MFA, or multifactor authentication, in our recent Cybersecurity Budget Breakdowns webinar. That’s a pretty good way, even if your accounts are compromised, to provide another layer of access. It says, okay, you have the password but now give me another form of authentication, and that will prevent that unwanted access.
You talked a little bit about endpoints and what’s called Endpoint Detection and Response (EDR). You know, it’s the anti-virus that you put on computers and endpoints. An endpoint is anything that has an operating system on it, right? So it’s not just laptops and desktops. It’s also printers, phones, switches, routers, firewalls. Anything like that is basically an endpoint. The weakest link in any organization is the endpoint typically because it has these things sitting there operating them. People in general are usually the weakest link. Now we do things a little bit differently at Global Data Vault, and we offer added protection through a feature called enhanced data protection.
Steven
00:14:20
Enhanced data protection is a solution offered by Global Data Vault to give our customers an extra layer of security. It stores the data outside of your repository, where you cannot get to it. If you cannot access it, it can not be deleted and that is actually why we store it where you don’t have access to it and you cannot see it in the Veeam console. In the last four attacks that we’ve had, it was Global Data Vault’s Enhanced Data Protection solution that saved our customers. We were able to fast clone the data back into the customer’s repository to give them the ability to restore the data from the cloud and bring the customer up in an emergency situation and get them back to a functioning state so that they can make payroll the next day and do their accounting. They could essentially run their business from the cloud while they are mitigating the threat in their infrastructure.
Kelly
00:15:31
That’s some pretty scary stuff. To describe exactly what happens, the customer logs in and thinks, “I’m going to go check my backups.” They look at the repository on-premises, but their backup files are gone. They go to their cloud repository–files are gone. Can you imagine what that must feel like? That’s when they call us and Steven says, sure, we’ve got your data right here. I’ll have that back in just a little bit of time. That’s pretty valuable stuff!
Steven
01:16:11
The importance of patch management
Yeah. So, patch management is critical to your protection because there are patches released every week that address critical vulnerabilities in software. We call it Patch Tuesday by Microsoft, but it’s not only Microsoft that releases patches. Any and all software solutions that you have in your infrastructure will be releasing critical patches regularly, so you need to make sure that have a good patch management solution in place, and that your servers and or endpoints are patched frequently.
You also want to make sure that you have some sort of cybersecurity in place. If you don’t have access to a 24-hour SOC, you may still have the ability to monitor your systems using the various monitoring programs that are available these days. But we would suggest that you have something that can watch your systems 24 hours a day, be it a SOC, network operations center, or one of the monitoring software out there.
User education and security awareness
I cannot overstress the importance of user education and security awareness. I would suggest that you have every user in your organization do security awareness training. If you have a cafeteria, even the cafeteria workers take your security awareness training, quarterly. That way they know what to look for. They know about spear-phishing email attempts.
I would also suggest that you find a program to test your users. There are several different programs out there that will send test spear-phishing emails to everybody in your organization, and then send you a report on who clicks on the links. That’s a really great educational opportunity to send those users who did click for some additional training on security awareness and maybe spam training. User education is one of the keys to help prevent ransomware.
Because most of the attacks that we have seen were all generated from users clicking a link in an email or opening an attachment, user education is very important.
Review any unauthorized systems and or user logs
The reason for that is if you review your firewall and see a bunch of hits coming in from another country and another state, that could be a brute force attack attempt trying to gain access to your system. That is another reason why it’s good to have a 24-hour SOC or network operations center. That way your firewall logs are continuously being monitored for unauthorized access attempts, and any IPs that the attacks are coming from can be blocked. You also want to check for non-standard systems and devices.
I’m going to go back to the PCs, the kiosks, the cell phones, anything that accesses your network that is not on a guest network needs to have some sort of security on them. That way, you know that Christie from accounting wasn’t connecting her iPhone when she downloaded an attachment the other day from a strange email address and infected your entire network.
Review firewall rules and logs
Maybe you’re getting an attack from a certain country. You want to put a global block on that. Another good thing is checking your domain controller, making sure that you don’t see any strange accounts that you do not recognize trying to log in over and over again.
Kelly
0:22:33
One of the things I wanted to mention about good housekeeping is sometimes we fail to think about the easiest things. One thing that attackers have done in the past is drop a USB outside a company’s door, around lunch break. All they have to do is wait for somebody to pick it up. I guarantee you that 90% of the people that pick one of these things up, the first thing they’re going to do is stick on their computer to see what’s on it. By doing that, you just unleashed whatever is on the USB onto your network. Now, if you have USB access policies or, you know, autoplay policies, those will help prevent that.
Steven talked about the kiosks and the things that you connect to the computers, but don’t forget about simple physical access devices as well.
And talking about IT housekeeping, a lot of times companies start off strong with best practices in place and everybody vigilant. But over time, people get complacent just as you do when you move into a new house. At first, you are really good at housekeeping, and everything is kept really clean for a while, but then over time you might not do the windows as often as you used to and there are dust bunnies under the bed.
The same thing goes with your infrastructure and reviewing security items on here. One of the things that we like to ask people is to keep us in the loop about any kind of business continuity plan changes. Obviously, as a provider, it’s important for us to understand what your requirements are, what changes you’ve made for BCDR, so that we can help you with that and we can recover you in the method that makes the most sense for your business.
That would include a quarterly review of your disaster recovery plan, and I’m talking about internally. Global Data Vault doesn’t necessarily need to be involved in that unless there are significant changes or questions that you might have, but we do have a series of cybersecurity webinars on how to create your plan and whatnot, If you want to go check them out.
Kelly
00:17:26
That’s pretty important. Recently, we’ve seen users calling us first thing in the morning and saying, “I think that we’re being hit by ransomware!”
It seems to be happening most often around two to five or six in the morning. Hackers are timing their attacks to when they think there is less chance of being noticed and more chance of success. That’s why it is good to have that 24-hour alerting system.
What to do if you do get ransomware?
We have to determine the scope of the attack and do what we can to reduce that. That’s number one, that’s where endpoint protection and anti-virus come into play to shut things down. But what if there was just one person present? If Steven had to go call everybody, I can’t just stay here and panic, can I?
Steven
00:24:04
What I’d recommend is to delegate each of these tasks to an individual. At the same time, you need to have one person determine the scope of the attack. The tasks are:
- unplug your switches and your firewalls so they can’t communicate internally until you do determine the scope of the attack
- delegate somebody to contact your cyber insurance company
- contact the authorities, since it is a cyber threat
- contact the FBI or local law enforcement
- contact your cybersecurity provider
- contact your backup provider
Cyber insurance companies, like car or home insurance companies, will want to know the scope of the damage, and won’t want you to repair anything until that has been established.
Your cybersecurity provider will help you determine the entry point of the attack, including what system it came from. I would suggest that you isolate that system once you find it and completely unplug it. Turn it off.
While you are contacting the authorities and determining the source of the attack, your backup provider needs to disable your account to reduce the scope of damage on their side.
Some people think that they can contact their backup supply provider after all the other tasks have been done, but all these things need to pretty much be concurrent. The backup provider needs to know immediately so that they can go ahead and disable access from your system and then start working on a plan to get your systems online, while your insurance company comes in and determines the scope of the attack.
Most of the time, the cyber insurance company is going to request that you not touch anything in your infrastructure, including restoring your machines. They normally ask that you save a copy so that they can go ahead and complete their investigation. That is why you need to contact your backup provider so they can work on getting the systems online to provide you access until we get a plan in place to go ahead and failback once your infrastructure is remediated.
Kelly
00:27:11
Detection and Analysis of a Ransomware Attack
Part of all of that is the detection and analysis of the situation. It’s really difficult at the time of the attack, or when you just discover something suspicious is going on to figure out exactly what the cause is. Your SOC might be able to say it came from this device, but it may take a while to figure out:
- what was affected
- who or what originated the event
- how it’s occurring, not only after the fact but also to reduce the impact that it could have on other systems
Even once it seems to be contained, you need to know which offices have been hit. We need to determine who, what, where, when, and why to collect the information that the authorities and insurance companies might need and to eliminate the further spread. You will also want to know what you need to do to make your environment safer in future.
Steven
00:28:28
Containment and Documentation
Make sure that you document everything from the time the incident occurred all the way through the resolution. This is also a good step for any insurance claims that you need to file. Document how much downtime you’ve had and exactly what happened from the time the incident occurred, who was involved, what employees were involved, who was handling what tasks all the way to the final resolution.
Most of these insurance policies do cover downtime, and you could possibly be compensated for that, but you need to check your individual policy. You also want to make sure that you document what system is affected.
Assume everything is affected. Do not let anything go unscanned or untested.
You also need to find the originating point of the incident and isolate that threat. What we’ve seen as backup providers in the past, is that we have restored customer systems but they had not fully isolated the threat so as soon as we restored the data, it was infected again.
You also need to prioritize the handling of the incident. When you go down and you’re affected by ransomware, the event should be treated as a Sev One incident by your IT team. This is a priority. Your first goal is to get your business up and functioning again. The priority is getting your business back into operation so that you can process your payroll, so you can do your accounting, so you can bill your customers. But you do need to reach out to the people suggested above prior to doing any of this so that you don’t cause any issues during their investigation.
It’s very important that you follow the processes, contact the authorities, contact your insurance company, and then prioritize the handling of the incident by factors. The main goal of this is to get you up as quickly as possible and basically back to business as usual.
Kelly
00:31:06
Steven talked about containment a little bit, which is obviously the next step in a process and it includes looking at potential damage, availability of services, et cetera, but there are other things to consider like containment within the media, containment with your investors, containment with your in employee resources, making sure that there are plans on what employees should or should not do, should or should not say while you continue to collect the evidence.
We’ve seen some companies immediately switch anti-virus providers after ransomware hit, uninstall what they had and re-install things. And, there are times when it is good to try to find something that will detect and remediate the issue, but sometimes you could be shooting yourself in the foot because you might be taking away some evidence or taking away something that can provide you insight. Just be really careful when you start making changes like that.
Obviously, with containment, you want to make sure that you can prevent the virus or malware from spreading any further, but just be aware that you could be introducing other issues into the environment.
So let’s talk about eradication It’s a big word for basically meaning getting rid of it, right? So Steven, what would or could we do to eradicate a virus or malware ransomware from an environment?
Steven
00:32:38
What can be done to eradicate a virus or malware ransomware from an environment?
To eradicate ransomware, you first have to identify all those that were affected within the organization. At this point, if you’ve been exposed, assume everything is affected, then you can go ahead and try to mitigate any vulnerabilities.
I would suggest once you work out with your cyber insurance company, then you can go ahead and work on mitigating any vulnerabilities that exist at this point. If your entire infrastructure is affected, I suggest that you consider rebuilding everything. Most malware infections are time-delay. They could have been on your system for months, then certain conditions came into play that triggered them into action. Something as simple as an internet connection can trigger a hidden virus to encrypt everything in your environment.
Imagine Christy in accounting turned off an old laptop in July. Just before she switched to her new device, she clicked on a FedEx tracking number in her email, went to a strange website, and downloaded a job application. Her old laptop is now infected. Months after switching to her new laptop, Christy realized she needed a file that was stored in the old laptop. As soon as she turned it on and connected to the internet, your whole infrastructure became infected.
So that’s an example of what you need to look for and why you need to mitigate all vulnerabilities. I would suggest rebuilding, but if you can’t rebuild, bring your backups up in a dev environment, or one that does not have outside access, and run anti-virus and malware scans. There are several out there that are recommended but work with your security operations center or your security team to find a good anti-virus program that can mitigate any vulnerabilities for any servers that were exploited or exposed. As I said, assume everything is affected.
I would do this in a phased approach. You need to determine what your critical servers are, what your non-critical servers are. Create groups, tier one, tier two, tier three, and get those done in a phased approach so you can get your most critical infrastructure online as quickly as possible.
Kelly
00:35:38
Part of that recovery process is getting things back to normal, and it’s not a quick process. You have to think about remediating issues and all of the steps that we’ve just talked about as a part of the return to normal operations, and it can take time. I mean, simply copying the files back down from a cloud provider can take time because of mechanical limitations. But the important thing that Steven mentioned is that quite often you have to rebuild everything from scratch. Even if the ransom is paid–we do see organizations pay the ransom because it’s the only option they have-if you don’t rebuild that system, there’s a chance that they’re going to come back.
We like to refer to this as the ATM, that you become an ATM for those hackers. When their funds run low, they’ll simply go back down their list of people who they’ve hit and who paid the ransom and hit you again. At that point, if you haven’t remediated, if you haven’t gone to a cloud-based solution with Enhanced Data Protection like Global Data Vault, you may not be able to recover again. You’ll just end up paying the ransom again, so you always want to make sure that you’re remediating issues and recovering properly, as well as implementing any type of additional monitoring recovery tools that are necessary.
Steven
00:37:20
What you need to do is make sure that you have user training. You need to have a good security program in place, and you need to have monitoring.
Kelly
00:37:34
We’ve got a bunch of free resources available that may be helpful. The How to Create a Disaster Plan Webinar and checklist is a good place to start:
Another good webinar is Insider Threat and Enhanced Data Protection. If you’re not familiar with insider threats, then I would recommend that you go watch that one, because it’s not just the people and threats from outside that you need to protect against. They could be coming from inside. It could be time bombs. It could be a whole lot of different things that are impacting our environment. The key here is that Global Data Vault provides a unique solution to protect data and companies from against just that activity.
We’ve seen it happen. It happens all the time. And like Steven said, we were able to recover because we were protecting the data in ways that prevented the ransomware from accessing it. And customers were able to get back up and running more quickly.
So with ransomware, It’s not a matter if but when. Preparation is a key for faster recovery, don’t set it and forget it. Keep reviewing those policies. Go back and make sure everything’s up to date, maintain your good housekeeping, and be adaptable.
As far as being adaptable, technology’s always changing. So what things might we think about in the future when it comes to ransomware?
Steven
00:38:59
Keeping up with advancing technology
You always want to make sure that you’re up to date on the latest technologies, especially your security team, to make sure that they are prepared for any attacks because the ransomware and/or threats change as technology advances. Also, you want to see maybe if your security team can do penetration testing on your infrastructure, find out where the holes and gaps are before an attacker does.
Kelly
00:39:40
Yes, that is a really good point, and maybe you want to have a third party do that testing because your internal tools may not be sophisticated enough. They may be used to your environment, right? So sometimes having a third party do that for your organization has benefits as well.
Kelly
00:40:02
What does Global Data Vault do?
Global Data Vault is a Dataprise company. We’re a fully-managed backup as a service, disaster recovery as a service, and Office 365 backup provider. We support VMware, Hyper-V, and physical servers. We do that based on the understanding that the quality of data is what’s most important, so not only will we tell you that yes, your backups are successful, but all of your data is within SLA. We monitor when your last local and remote backup was, when your last replication was, and judge that against the SLAs we have in place and report that back to you. So in the event of ransomware, you know that as of this morning, my last backup was four hours ago. That’s when I can recover to, and everything’s good.
Enhanced Data Protection is included by default. There’s no upcharge for it as a new customer. If you’re an existing customer and you don’t have it, contact us we’ll be happy to get you some info on that, but it is enabled and available by default to every new customer.
0 Comments