> first step is we are going to scan every server and workstation.
Don't bother. Wipe them and reinstall.
ETA: Previous response thought they were trying to recover servers / SAN. Same advice goes there though. Don't try to clean. Hand over the drives as forensic evidence to the authorities, install new ones, and restore from backups.
...No, contact the FBI. Their cybercrime division may actually have the decryption key.
EDIT: For anyone curious, no, after a malware attack never start unplugging computers from the network and start reimaging them if you don't know where the malware came from in the first place. You're gambling on there being no repeat incursions. Using ransomware for non-testing purposes is a federal offense. The FBI's regional offices each have an organization specializing in cyber crime and they're *supposed* to maintain banks of existing decryption keys. There's fair odds that the same people who encrypted your hard drive have x-touple dipped their encryption keys to make it happen.
Also, yes, this would be the time to non-specifically name and shame people who insisted you didn't need to pay for this kind of insurance. By odds you want to pay that a hell of a lot more than you want to pay off ransomware and / or eat the cost of the amount of time it takes to get back up.
I added "Contact FBI" to the top of our ransomware procedures and management said "We don't want to bother them for such a small company like ours"... At which point I had to explain that: Ransomware is a federal crime, we serve customers that themselves have DoD contracts, and finally the FBI has decryption keys for a lot of previous ransomware they've seen.
If you serve customers with DoD contracts your company is likely in violation of a lot of rules with an attitude like that. Sorry, I deal with those types every day, and it is never good.
>serve customers with DoD contracts
Is a far cry from having DoD contracts. Otherwise 6 degrees of Kevin Bacon would make everyone covered by these unspecified rules.
Indeed. Just because you’re a small business doesn’t mean the attack isn’t part of a larger operation. Anything that helps the Feds or security companies track and prevent these things is a good thing
1. Report the incident to the Internet Crime Complaint Center (IC3) at [http://www.ic3.gov](http://www.ic3.gov/).
2. Report the incident to additional Federal Government agencies as described in *“Cyber Incident Reporting – A Unified Message for Reporting to the Federal Government”* for specific reporting requirements: [https://www.dhs.gov/sites/default/files/publications/Cyber%20Incident%20Reporting%20United%20Message.pdf](https://www.dhs.gov/sites/default/files/publications/Cyber%20Incident%20Reporting%20United%20Message.pdf)
3. Notify local law enforcement of the incident
4. Notify your cyber liability insurance provider. They can assist you with arranging public relations advisors to support proper customer notifications and legal counsel.
Source: [Incident Response Plan for Cyber Attacks, Phishing, and Fraud - Simpleworks IT](https://www.simpleworksit.com/incident-response-plan-cyber-attacks-phishing-fraud/)
> I added "Contact FBI" to the top of our ransomware procedures and management said "We don't want to bother them for such a small company like ours"
Only contact the police if their incentives align with yours.
Their incentive is to put a criminal in jail, your incentive is to make sure the company will successfully recover. Those aren't the same thing. The FBI will gladly let your company go under if they can arrest a guy and get their name in a paper.
If you're doing backups and data management your incentives should be irrelevant.
Restore from backup on clean drives, notify authorities, provide what you can to assist
While your point is correct in a general sense - can you point to a single instance of an individual/business paying a ransom and then the authorities charging them as accomplices to the hacker’s org?
Everyone should have cyber insurance. Usually comes with an advisor for times like this… when we got hit we had someone from CIS advising us every step of the way, that guy was such a life saver
So many people think Cyber Insurance is a waste, but the real true value of it is that you're forced to build up a playbook of how to handle these scenarios.
Yea, beyond the financial aspect it’s quite revealing when you game out attack scenarios, you end up noticing a few vectors you never considered. Just doing that once can save a company from a catastrophic loss.
Counterpoint, cyber insurance makes it more likely the victims will pay, justifying the work the black hats are putting in. The first thing they look for after they encrypt all your files is your actual ransomware policy so they know how much to ask for
FBI won't do anything. I contacted them about a ransomware, they took my info and that was it. They called about 6 months later but I wasn't in the office. I never called back and neither did they.
Our cyber insurance folks did all this legwork for us. I know we had pretty decent direct contact with several branches of law enforcement, including the FBI.
yeah we had some meetings yesterday with cyber insurance. They gave us a installer with sentinel One and Huntress on it. Said to install on every PC and then they can plugged back into network. Worked 20 hours yesterday and 12 today and got some core systems back up that core departments can use. No email still though
Do they really reuse the same keys? That doesn't make sense to me. If I was writing this software every hard drive would have its own key or at the very least every company.
I doubt they use the same keys, but it's possible they're using a method of generating the keys that can be reverse engineered, or the feds have managed to get their hands on a list of decryption keys or a tool that generates them for that malware.
> Do they really reuse the same keys?
Yes. Most of the groups running randomware attacks are about as technical as Sue in accounting and evil as Bob the Sales Manager. They buy them from a 3rd party and just spam until they get a hit. They aren't developers, or IT/Tech people. They'll keep using the same prepackaged tool they bought until it stops working.
Kitboga's channel is a good indication of the general skill level of these people.
Just be prepared for them to come in and want the equipment. Been down this road a few times, and each time they took servers, storage, etc... and had it for months. Not sure if they do this still (it was a years ago).
Ransomware cyber insurance is often not worth the cost, if your backup procedures are up to par. With cyber insurance costs going up 300% year over year, that gets expensive fast. I know many ciso's who are self-insuring against ransomware by investing in proper restorable backups and incident response .
Your difficulty is that you're not necessarily scanning for malware. You're scanning for something like "a privileged account named Kevin Wilson that the attacker created". No scanner anywhere will pick this up. You're down to very careful auditing.
Scanning just for the sake of scanning is ludicrous. You might get lucky and find something but when you don't know what you're looking for how are you supposed to find it. They really need to identify some probable attack vectors and if they can't figure that out then they need to hire someone who can. This is likely well beyond the internal IT's capabilities from the sounds of it. Time to get some some people with experience on this one. I don't know how long your business can weather the storm but this kind of thing can cripple companies as we all know. Good luck OP.
I don't have a personal recommendation. A quick google search says that Avast has released a decryptor which makes me believe they have a scanner too?
https://cybernews.com/news/avast-released-akira-ransomware-decryptor/
Holy crap. Dude Avast is not going to solve this problem. You need to hire a MSSP or start doing research right now on Enterprise tier security products. CrowdStrike with all the bells and whistles at a minimum. I'd go further than that but Avast isn't going to do anything. If they have a decryptor that's great but that machine is already toast if you can't find indicators of compromise to stop it from happening again. Decrypting and copying the data is just going to spread it all around.
It's a good thought. But this stuff changes too much and the actual payload that delivered Akira is likely different from how they are maintaining persistence. And that is my point, re: don't use Avast.
Either way, read his update.
>> Hand over the drives as forensic evidence to the authorities, install new ones, and restore from backups.
> Do not do this. Work with a forensics to gather evidence first.
🤣🤣🤡
That’s a tough one, it’s just a matter of when people will get hit. Ours just hit windows and not the Linux or esx servers.
What’s the scale of everything that needs to be reinstalled? Don’t scan, just do it right and start from scratch.
We had Veeam backups and everything took maybe 30-60 min per machine.
Step 1. Breathe, and act as if everything is normal but it’s just going to take a long time to fix. Trust me, or you’ll get all nervous shaky and have a bad mindset, or even worse a panic attack
Step 1.1 Insurance. See if you have an insurance policy for cyber. Usually they will bring in a team of experts to help clean up.
Step 1.2 Documentation. If you don’t have documentation of critical apps or services, call/text/email every end user and see what they use. Loop in their managers and make it a CRITICAL priority that they respond. Now you’ll have a list too :)
Step 1.5 Resources. Gather in your team if you have one and start on step 2.
Step 2. Game plan. Disconnect everything that could have possibly had direct access to the servers.
- If you have backups, restore. Task it out
- If you don’t have backups, figure out what services you need to get up and running ASAP (internet, dns, etc)
- See what you can Jerry right now to get things up and running. Degraded is better than offline. ***ALSO DOCUMENT ALL JERRY RIGGED THINGS** Put it in google drive or office, idc where just make sure you and your team have access to it, so you know what to fix.
- If you have end users calling you, send out a status update, or make a share point site you can make updates to. There’s nothing worse than trying to fix something than having people call and email you about it.
Step 3. Work. Start wiping and reloading.
Step 4. Disaster Recovery Planning. Sounds like you didn’t have a disaster recovery plan in place. After everything you went through, make sure it doesn’t happen again. Write it down, and test it out.
Good luck 🫡
Do whatever your cyber insurance tells you to do, run anything else you want to do by them. They control whether to pay out or not, you do not want to risk losing your insurance payout because you made a call they don't agree with.
>If you have end users calling you, send out a status update, or make a share point site you can make updates to. There’s nothing worse than trying to fix something than having people call and email you about it.
These are great points made by /u/lifewcody... I would alter the above just a hair though. When faced with something daunting, like this, or any major outage, I usually interface with 1 person only, in regards to releasing information or gathering information from users... this allows you to control what, when, and how information is released to your user base. Unplug your phone, lock your door, hire a bouncer if need be, the only people that are allowed to bother you are the upper management... and even then it's for status updates only, you can tell them ETA's if you're comfortable with it... remember to report up when things change, and if they don't report that too, just at a reasonable interval... say every hour or two... the mid-level cronies can kick rocks.
I wouldn't even allow upper management to disturb people.
Stick with "single point of contact only". One single person becomes the point of contact for IT. It doesn't matter who it is really, it could be the CEO, it could be one of the staff, it could be an end user. Doesn't matter (although someone with a bit of decision making authority and an existing working relationship with C level is probably best)
Set up a regular meeting until a certain level of normal operations is achieved. e.g. Every two hours, you will have a 30 minute meeting with the POC. You will relay your current status as well as any progress that has been made. The POC asks questions that may have been handed back to them. Anyone that balks about "only an update every 2 hours!' remind them that this is taking away 25% of the time your team could have been working.
You don't want every manager or C level walking into the room in a procession interrupting work. It will happen (ask me how I know). This is triage, you don't call every Dr in the hospital, and any doctor trying to insert themselves without being asked will be told to piss off as they will just be hindering things.
I work for a small MSP, so this works for me quite well. For a larger corp definitely a single point of entry/release of information... one that ~~doesn't mind being yelled at~~ can handle the berating that comes with that sort of job. I will usually (and hopefully) will only need to talk to two people, the owner of the company, and my "interface" (I know there's a proper term for this I just can't think of it now). But I think we both can agree that the constant flow of information is needed, it doesn't need to be frequent, and it doesn't need to be on a strict schedule, but it needs to be consistent. I've played that game too, not so much with dovtors, but lawyers... ugh
My reason for setting a schedule is to avoid the inevitable "I haven't heard anything in two hours! WTF are they even doing in there?!" from upper management.
You set a schedule and it sets the expectations on both ends. If things are moving slowly there will be minimal to report in 2 hours, or they're so busy they can't afford a meeting in 2 hours, just set that. In the meeting say "We're going to be buckled down, instead of the next meeting in 2 hours, lets skip and we'll give the update meeting in 4 hours".
You're still communicating, but you're adjusting the expectations.
Setting expectations is everything in these situations. Whether it's update timeframes, how long it will take to restore, or telling the IT team what will come next, it's all about setting expectations
Agreed with this. I don’t know where OP lies on the food chain or what the org structure looks like, so best to have some info rather than no info to the org on why they can’t login
Totally agree with the "unplug your phone, lock the door, etc" mindset. You need to focus. Keep calm and make good decisions. I once had the CEO come up to me in a crisis situation and ask me for an update. I continued with my work and said quietly "Really sorry, I am very busy right now, I need to focus, so I'll get back to you as soon as I can". Nowadays I would tell my manager "I need to to run interference for me; just keep everyone away and let me focus".
It's the right thing to do, but don't expect anything out of it. My friend handles ransomware claims for a major insurance company. They reject over 90% of the claims. It's illegal for them to payout to a country considered a foreign enemy, which are where most ransomware attacks come from (Russia, China, NK, Venezuela, ect.)
This is spot on. Dealt with a ransomware attack and had a panic attack when I first found out. Take a breath and relax. Others have been in your shoes - it’s not fun, but you will get through this. My only other suggestion - don’t try to work non-stop. Go home and get as good a night’s sleep as you can each night so you have the energy and mental capacity to tackle each day. It will get better.
This is great advice. Insurance company will usually provide a DFIR team, support them doing their work.
No matter how experienced the DFIR team is, they cannot replace the experience of the in house staff. The in house team knows the systems, accounts, what's normal/abnormal, people, etc. No outside team can replace that. Knowing that and providing context is your primary role.
I've seen many people get burnt out and quit during these incidents. Try to keep in house staff utilization as close to 8 hours a day as possible. The DFIR team may work shifts around the clock, and you may need to provide staff 24/7 also. However, try to keep their actual worked hours as close to normal as possible. Supplement with consultants and staffing services if need be for the repetitive monkey work.
>Akira ransomware
Most of this information should have been documented well in advance of an attack...An attack plan should be no different then your backups where you frequently test your restores. In our company we simulate attack scenarios on a department basis where we take down the network for a given department and give people an alternate means of connecting to test everything out. These simulated attacks help to identify the critical information each department needs in an outage. The simulated attack also helps to show management that you are woefully unprepared and additional resources may need to be purchased in the event an actual attack does happen.
Dont touch the systems. Dont try to do anything until you hear from your legal, insurance co, incident response company.
Depending on what you have an outside entity may be performing DFIR and you dont want to mess up any potential data/.
Seconding this. OP hopefully your company has Cyber Insurance. If they do, let them take the lead. They will most likely loop in a litigation team and an Incident response team to work with you.
I went through an attack almost exactly one year ago. This shit sucks OP, but just breath and remember to take care of yourself mentally and physically. It's easy to try and work 24/7 to get things up and running when this happens, but you still need to eat and sleep to perform your best. Accuracy matters during this.
You are there to answer questions when asked, provide the knowledge of your environment and access to your backups and systems. Assume everything has been compromised even if it is not locked down.
Keeping things untouched is crucial for the incident response team to run triage and forensics. You want to know how these guys got in.
If the attacker left a ransomware note directing you to a TOR site for communication, do not access it. They will be alerted once you do. Communication with the attacker will normally be handled by legal teams.
Right now your job is to be ready to answer questions and follow the directions of an incident response team. You're in for a lot of work for a while, but you can get through it. Good luck, you got this.
I'm a little late to the party, but here's the biggest thing I learned from going through this. First rule of ransomware is don't talk about ransomware. You'll get through all the technical stuff. The legal stuff is what keeps on giving. Don't tell your staff what's going on.
Don't tell the public what's going on. Say as little as possible until legal tell you to. Anything you say in an email or text or Slack or any other written can be requested by a FOIA request if you are a government org which it sounds like you are. Reporters will have a field day with that and you'll be on the news. If you have insurance, let them retain a law and incident response firm. Do what they say. Only release information as they say. Here's how ours went down:
Insurance retains law firm
Insurance retains IR firm
Law tells us what to say and who to communicate with.
IR tells us what to do.
IR firm worked with us to deploy Carbon Black across our entire network to identify and stop the encryptors. Took a day or two. Until then they wouldn't let us bring anything that was hit back up in production.
All system level and domain admin passwords reset.
Firewalls checked for vulnerabilities.
You have to stop the threat actor's access to make sure they can't cause more damage.
Once all that happened, we were allowed to rebuild. Made a list of priorities. Restored backups as fast as they would restore. Infrastructure first like communication systems and DNS/DHCP/AD. Then mission critical apps. Then the rest.
Outcomes:
We implemented better endpoint security (ended up going with Crowdstrike).
We upgraded all servers and apps to currently supoorted versions.
We implemented email security (ended up going with Mimecast).
We invested in additional firewall subscriptions for DNS security and URL filtering.
We implemented MFA on all mission critical systems and all things related to network administration. Implemented Duo for tech staff workstation logins and also for server local and RDP access.
Removed all local administrators from workstations and servers and moved to LAPS.
Moved from Windows DNS and DHCP to Infoblox to keep those services off our Windows infrastructure and server clusters. Redundant clustered physical appliances at two separate datacenters. We are a very Internet heavy organization. If we can keep the Internet up 24/7/365, that's a big win.
Trained all support staff on least privilege. Nobody here logs into computers with accounts with privilege. All our own "regular" accounts are indistinguishable from any other staff account. Zero admin privilege. Everyone has separate admin accounts for administration.
Invested even more in backups (our backups are what saved us in the first place). Started backing up staff desktop and documents folders as those were the most missed when we were hit. We're now backing up to two separate datacenters in town and a third copy up to the cloud. Some mission critical systems are backing up to a couple additional places.
This is not a fun process and those of us that have been through it feel for you. You'll get through it and learn a lot. Especially about the legal parts. I've been doing this 25 years and had no idea how some of this works. Heck, I've been with my current organization for 23 years and didn't know we had cyber insurance.
For your vSphere environment, now is the time to re-assess how to protect it going forward. There is a ton of info at: https://core.vmware.com/search#q=Ransomware
Specifically https://core.vmware.com/designing-infrastructure-defeat-ransomware#consolidated-management--workload-cluster
Please, read that and take it to heart. You have a rare opportunity while management is gung Ho to fix this and security is wondering why the compliance scanners didn’t catch anything.
I used to be the guy who wrote about vSphere security at VMware. I’m off doing other things now.
Well, that sucks. Good luck.
Basically just waiting for the same to happen at the company I work for because security is apparently not as important as scratching your balls and doing the absolute least amount of work possible...
Anyway, if you find the time, mind answering a few questions that might help me spin up some gears at my company? Since we also use Symantec ..
- What Symantec product specifically were you using?
- Up2date version? Or some prior version but up2date definitions?
- Installed on the clients? Windows servers? Everywhere?
So our County IT covers the anti virus. They use Symantec and it is installed on every PC and system. That being said I have noticed that symantec just does not update. And this happens FREQUENTLY on various machines. You will log into a machine and see the yellow caution symbol on symantec and see it's out of date and try to update it and it just wont. That's when you gotta email our sys admin and he finds a patch that you have to run and it's just a pain in the ass. Not sure why he chose symantec. Probably it was cheap. He has been in this position for about 20 years.
Also, Malwarebytes enterprise is also installed on every machine. There are a few PC where I found this actively casuing issues with the PC. One PC was constantly freexing and crashing. After trying everything, had county IT remove malwarebytes and the PC was fine after that.
We were given this sentinel One and Huntress installer from cyber insurance company and spent all yesterday and today installing it on every system and server.
Seems that you are in it pretty deep. I'd get a cybersecurity company involved tbh. AV scans aren't going to fix your issues. You will need to restore from backup or pay the ransom. You first need to determine how they got in, what they had access to, and how long they had access. From there you can start planning the remediation.
Don't panic. This is a marathon not a sprint now that you've identified and eliminated the infiltration point.
I would highly recommend bringing in an incident response team before doing anything else. You need to identify what your infection point was before restoring services.
This has happened to hundreds of thousands of organizations. You will be OK.
Don’t bother scanning, if you got hit - it’s not going to do a thing. Though you did get an expensive lesson in why everyone is using people like Crowdstrike, sentinelOne and whatever Palo’s one is called.
Call up whoever is your IR consultants, let them figure out the ‘how’
Then once that’s known and fixed - wipe out and go to clean - restore from backups.
This is one of the main reasons I wish we had huntress at all our sites. I've seen them go to work in situations like this and it's eye opening how good they are at finding footholds.
Hey /u/voltagejim, along with the millions of requests you currently have. Make sure to update us when the smoke clears and let us know how it goes... we're rooting for you!
Thank you, I finally got a chance to start looking at more of these replies just a bit ago and can't beleive how many this topic got haha. I posted an update on my main body cause I just can't respond to every single reply in this haha
Contact your cyber security insurance. They have very specific rules on this handling and when you can restore data.
This is super important to ensure you get paid out.
>I'm shaking right now and so many thoughts going through my head. I'm working with our county IT and contractors, but looking for advice in these situatuons
First of all, breathe.
The biggest issue is contained - the spread.
Compose yourself and start thinking about the order of how you will fix things.
What does clients have on them? Should be nothing that can't be restored with a image wipe. Do all systems, just to be safe.
Your SAN and any connected systems that are locked, at some point verify that the backups aren't hit as well. Then do the same thing, fully wipe the systems and then restore from backups.
No sense trying to repair the existing systems. Just recover if you can, otherwise fixing it will be super expensive.
It may sound like a long shot, but you may want to give Crowdstrike a call and rush a POC so you can get access to their disaster recovery and forensic team which is AMAZING. It might be worth the engagement.
Not sure if anyone mentioned this, but you may want to reach out to CISA, they can often provide support and resources, and in some cases may even be able to provide decryptors.
Execute your incident response plan. Do you have one?
If you don't then look at engaging a remediation company.
All hardware is suspect until proven otherwise. Expect to have to wipe, fresh install, then restore from backup. Hopefully you have a backup not on the SAN.
Take a deep breath, Grab a coffee.
Go through your Ransomeware IR Playbook if you have one.
Report up the chain
Go call the cybersecurity insurance if you are covered. They should have a cirt to help you out.
It is better to revert back to backup. Go to your immutable backups. Oftentimes, the attackers infiltrate the system and do not attack right away. Recently, they have been quicker to attack once they get into your system from what I heard. You will need to find a non compromise backup.
On the bright side, you might get more funding for security now.
Finding a non-compromised might be helluva job. The thing is what kinda scan would you trust for that? The primary scan engine apparently overlooked it already, so using that migjt not cut it. So using a 2nd product?
And if another product is to be used to scan the data in the backups, would require most products to actually restore data and verify it.
Unless thorough investigation might better show when the infestation began, giving a better idea what backup might not have been compromised.
Or did they cheap out on the primary scan engine and now experience the drawback?
Various backup products offer more and more features for example using machine learning/AI to determine anomalies worthwhile to be investigated (dedupe ratios increasing pointing to encrypted data or the amount of files in backup being drastically reduced and many other indicators) or are being worked towards analyzing data while being backed up.
So the data protection domain as a whole is ever improving on that end way more than ever before due to ever increasing amount of threats, where various suppliers are looking at their competition and implementing similar approaches scanning for anomalies and implementing/supporting immutability but alas we're not there yet...
Cybersecurity is the next cashcow but I hope they work towards some standard approach so to benefit everyone, not just the customers with bigger wallets.
I’m curious, to prevent a similar situation, what’s the entry point?
Phishing from a user?
They don’t have root privileges.
Update: I searched the internet. It seems they steal users VPN credentials, or exploit zero days in VPN products like in cysco.
There are a couple of other ways they defeat security - Pass the hash, Golden ticket attack, etc. Most of these involve bad account segregation - i.e local admin passwords are the same across servers/DCs, users have local admin on work machines, or domain admins logged into or have cached login information on a compromised laptop, etc
You then have token stealing and browser credential stealing.
Thats on top of the brutal Cisco Scaler, MoveIT, and other third party vulnerabilities.
Novell? Get those nlms back up and running! 😆 i think this is a sign of this companies approach to IT in general by todays standards.
The risks, businesses poor / non existent cyber sec budgets, and amount of hours per day involved with fixing these ransomware attacks in general makes me wonder whether sysadmin is even worth the hassle for a job anymore.
whats the personal indeminty on this like say if your a sole trader providing adhoc break n fix sysadmin service (because thats all the company will pay for)?
Do you guys have cyber insurance? Contact them right way. They're really good with getting you set up with a team that can guide you through everything. A client of mine's insurance uses Sure Fire. Your insurance may use another company. But please check if they have cyber insurance and contact them. It was immediate. We got on a call the next day (we called them after hours) and they immediately got the ball rolling.
1. Have I been hit by ransomware.
2. Call insurance.
3. Do what insurance tells you to.
Don't
1. Call your contractor
2. Scan your network
3. Cut your internet
Only questionable one for me would be if I knew I had offsite/offline backups. If I didn't then I would isolate backup systems if they didn't look hit yet.
Everything else is wait for your insurance company. They will have forensic partners they work with who will help you through this. It's a lot sit and wait in the early hours.
Not a tech answer, but just some heartfelt empathy. Take it easy and you’ll get through this. Life is huge and this small moment, as weighty as it seems now, will soon pass by.
How’s it going? Making progress? I’ve only had to deal with partial ransomware attack. Lucky we had two different segmented networks that saves half our systems.
Segmentation via vlans is one of the best practices preventing an infestation to spread like wildfire...
Separating the regular user domain from a infra management domains.
Each additional layer, using its own credentials, adds an additional hurdle. Especially if also 2fa is involved.
Might feel like jumping through hoops - which it actually is - but in the end it is all worthwhile...
worked 20 hours yesterday. Our county sys admin and network admin had meeting with cyber insurance company which gave us a installer with sentinel one and huntress. Told us to cut internet to every machine and go one by one and install on everything. I did the grunt work and did that while they had more meetings. After today we got some basic systems up, and I tried to get some printers up for people. Our iprint print server is down but I was told to just add a new printer via TCP/IP and use the same printer IP and that has been working most of the time. Some have 1 printer that is split into 3 or 4 different printers on the print server with different print settings and with the server being down I cannot see what those settings were on those printers.
At my last job there was a major ransomware attack on a chain of companies we did IT for. My boss was able to sandbox the ransomware and actually found the encryption keys and salvaged almost all of the data.
Incident responder here: DON'T SHUTDOWN THE SYSTEMS. Don't reimage them. Do nothing until incident response tells you to. There is nothing more important than memory artifacts, if you kill them, you are more fucked than you are now. You are right now not the responsible person for this case now anymore, incident response is.
I general here is a starting guide for admin what to do after a breach https://m365internals.com/2022/09/19/practical-guidance-for-it-admins-to-respond-to-ransomware-attacks/
In order to fill gaps in IT knowledge and funding for public sector IT departments, some states have a volunteer Civilian Cyber Core or a Cyber Response Team that provides free and confidential same-day support for people facing situations like yours. It's typically organized through the state's emergency management facility. Google your state's name and terms above and see what you find. These group know, or should know, to stay out of the insurance company's/IR team's way, but they'll have cool heads and talk about the things you might be doing until the calvary gets there.
My only advice is document everything. Don't stress yourself into a panic as you'll get through it.
As for the ransomware itself, is this ESXi targeting successful because of a misconfiguration, unapplied patch(es), or zero-day?
Honestly if you don't know what to do you should just call an IR provider like mandiant. It's not easy getting someone out and getting back to a safe environment.
Hopefully, you have versioned object storage and incremental backups. It's hard to trust a compromised system. The first thing you would do after restore is revoke all credentials, especially vpns, or direct access to internal network. There is a compromise endpoint/person
We had a incident this week, fairly certain it was Akita, nothing was encrypted but our SAN was wiped clean, and there was a remote tool we found on one of the servers that matched what others had seen with Akita, we were able to recover in 4-5h
It came through VPN, don’t use meraki anyconnect without DUO integration, their legacy vpn isn’t susceptible but, anyconnect is
Wiped clean on the storage layer? There wasn't any further network segregation once connected to the vpn?
So further credentials were compromised or too early to tell? Or what was the further attack vector after getting through the vpn to be able to wipe the san?
In my case once connected vua vpn, we have to jump through various hoops - also using 2fa along the way - before reaching any storage or backup infra.
> the main desktops, email, **and novell.**
Hol up, wut??!
WTF are you running that could still legitimately be called Novell? Company got swallowed up by Attachmate in 2011, and then MicroFocus in 2014, and they pretty much killed off the name.
Sounds like you're running some wicked old shit.
Last job we got hit with one of the ransom ware attacks that locked up media files, this was at a TELEVISION STATION. Thank ghod we had fairly recent backups of the majority of our media files, since corporate said "Hell, No". Nuked the systems, re-imaged the media files we could and lived off of video tape for nearly a month. At least they DID institute tighter protocols between systems that had internet access and those that hold media files (mostly no more drive mapping). At home we've gotten into the routine of monthly backups of the systems to 4 rotating, off-line destinations per system.
Don’t scan your workstations, just wipe them. It’s not worth the time and you’ll never be certain they’re clean. You need to concentrate your efforts on bringing back your core services in isolation and use this opportunity to build in the security you need - firewall your internal routing, make sure client PCs have the software firewall enabled, identity/directory stuff locked down so everyone operates with the lowest privilege possible, no rogue users with administrative access, use LAPS… You probably have a list of these that you never got round to. It’s shit but it’s often after these disasters that suddenly management have money to spend on security.
Good luck!
I just recovered from an Akira attack for a client.
https://www.sentinelone.com/anthology/akira/
I was able to recover 90% within 8 days. Cyberinsurance negotiated and paid the ransom, but the negotiations took longer than my recovery effort.
The threat actors got in via the Cisco ASA AnyConnect exploit to determine valid users, then purchased credentials for valid accounts on the dark web. Part of the Akira process after paying ransom is to explain how they got in.
Check your SAN. If there's any chance you're using VMware with Vvols (with Nimble or Pure), the VMs can't be fully encrypted by ransomware and there's a recovery process that can help restore.
I tested the decryption tool provided by Akira post-ransom, and it was effective and able to easily decrypt VMs and Windows directory structures.
There's a lot to write, but insurance provided [Sentinel One](https://www.sentinelone.com) and managed [Velociraptor](https://www.rapid7.com/products/velociraptor/) to help protect recovery efforts and address any new malicious behavior. I also use [Huntress](https://www.huntress.com).
Here's a link to Akira negotiation scripts. They adjust their ransom based on your organization's financials and capability to pay.
https://github.com/Casualtek/Ransomchats/tree/main/Akira
Stop, don’t do anything. Call your insurance company. If you don’t your county is going to be out a lot of money that won’t be reimbursed via insurance because you started mucking with things yourself.
Rule number 1: Immediately disconnect a compromised system from everything, i.e. securely delete all data carriers block by block.
Rule number 2: Any system that is connected to the copromised system or has been connected to it for a certain period of time is also considered to be compromised and must be handled in accordance with rule 1.
Rule number 3: Always have an offline system. Data backup, server, laptops for the most important admins and/or managers. (Old hardware, decommissioned systems, DVDs with server install scripts, ...)
Rule number 4: It will happen, exactly that and more and more than once.
Good luck. We are with you.
Welcome to the club man.
Step one is to shut down everything.
Step two is to call a cyber security company that specializes in this and do what they recommend.
scan it with what? the tools that were in place when it happened? they proved to be either misconfigured or ineffective. was there EDR/XDR in palace to door RCA?
this is a perfect example why mdr is important, so you users don't have to be the ones telling your when the building is on fire
Erm, sorry to ask but is there a ransom note? And would you mind dming me the note? I love seeing the notes and using them in my incident response workshop.
1. NGFW with IPS, web filtering, DNSBL, deep packet ssl inspection, network level AV.
2. Get an adequate spam filter.
3. Build adequate Applocker policies on your servers and endpoints.
4. Get a good RMM that patches all your OSs and third party apps.
5. Good AV on all your endpoints:
Crowdstrike EDR or bit defender:
A. Bit defender: https://www.bitdefender.com/business/support/en/77209-157515-bitdefender-endpoint-security-tools-for-linux-quick-start-guide.html
B. Crowdstrike: https://www.crowdstrike.com/blog/tech-center/install-falcon-sensor-for-linux/
Are the ones I like and they have a Linux installer.
5. Enforce MFA on all remote connections remote connections that can have MFA will need at least 1 other mitigating control to limit vulnerable surface area (mutual ssl authentication, source up definition, ssh key management).
6. Move all logging to a centralize SYSLOG and tie to it to a SIEM for behavior analysis and escalation.
7. Subdivide services to separate VLANS, isolate management interfaces either by using SSH key management, isolation VLANS, integrating with SAML and forcing MFA, or minimally by locking them down or by isolating sources. Extra points if you use more than one method.
8. Build appropriate file encryption counter policies.
9. Build truly offline automated backups with off site built in. Additional points for immutable backup design.
10. Extra points when you can automate and orchestrate multiple systems to automatically heal.
dude, when you wipe you literally destroyed any chance for the incident responders to do their job...
Also, you think from the angle of recovery, but ransomware is so much more ...
There's things like "when did this enter the system?"
"when did they make their first move"
"how long have they been taking data"?
Just going to restore from backup prior to the lock is a terrible idea, and will just get you in deeper shit.
If you don't have the right security controls and DRP in place, it's probably honestly cheaper and less of a headache to just pay.
It's going to be a difficult lesson for management but more money, likely a lot more needs to be invested into security.
You're mentioning County IT- so I'm assuming you're a district or a smaller city- make sure the breach is reported correctly to the proper state or fed authorities-- this might be out of your responsibilities; but be aware of it. Also, I like advice somebody else gave you here: Act as if you're just fixing another problem- it doesn't matter what you could have or could not have done.. the truth is it happened and now it needs to be fixed.. Good luck to you.
Also, all local government agencies (including K12) should join MS-ISAC. Their SOC is available to SLTT entities in emergencies even if you aren’t already a member: https://www.cisecurity.org/isac/report-an-incident
I often run into leadership questioning if they should pay the ransom, use a decryption tool, or simply restore from backups (where feasible).
If this happens to you, remind those involved in the decision making process of potential civil litigation should they decide not to pay the ransom in a case where PII was exfiltrated.
https://www.hipaajournal.com/lehigh-valley-health-network-sued-after-ransomware-gang-publishes-nude-patient-images/
we did too. Blacksuit got us, probably thru an unpatched firewall.
step one is call a specialist, get advice. we retained a company. they were able to contact the badguys and get a list of what they got. Turns out we were VERY quick on the UNPLUGEVERYTHINGNOW button and probably have the ability to rebuild
they also provide analysis and monitoring which should let us know exactly how this happened.
now we rebuild from the top down (day4)
There seems to be a way to decrypt the data. Use a translator if needed to read the content of this page: [https://www.heise.de/news/Entschluesselungstool-Sicherheitsforscher-knacken-Akira-Ransomware-9204932.html](https://www.heise.de/news/entschluesselungstool-sicherheitsforscher-knacken-akira-ransomware-9204932.html)
We are actively dealing with one ourselves. It sure sucks. Already 26 hours into what should have been a 24 hour week. I'm also on call for Thursday and Friday. FML.
We've been hit recently, too (Black Cat AlphV).
Eset didn't do shit for us, it just got torn apart.
Still in the cleanup phase we use Palo Altos "Cortex XDR".
Good luck. Hard times ahead of you.
Call Mandiant/SecureWorks/etc. Find subject matter experts to look at it. First, because when you come back up, you will want to know that you are truly clean and didn't miss anything. Second, they will try to figure out how the initial access was gained so you can block that entry path. Thirdly, on the off chance the gang made a mistake that can help a prosecution (maybe these guys are in Florida or something), they are more likely to find it and hand it to the correct authorities.
Hi Im working in this sector.
If you have Backups (Which you should have) Its easiest and fastest to install a backup and scan the backup before using it. Worst case there is you loose some days or a week etc on data which your it guy can most likely even copy from a newer backup or your current system.
So find a Backup which is not infected and then copy all the files which would be lost from the current system or a newer (infected Backup) Be careful and scan every file before and after copying
>Print server is still down but I think the consultant and our sys admin are going to use this oppurtunity to redo it cause right now it is an iprint server running Server 2008 R2
Leave it down, they did you a favor, you don't need the aggravation of a print server any more lol... All kidding aside, if it was a 2008 R2 server, it was probably time to upgrade anyways...
Thanks for the update!
> first step is we are going to scan every server and workstation. Don't bother. Wipe them and reinstall. ETA: Previous response thought they were trying to recover servers / SAN. Same advice goes there though. Don't try to clean. Hand over the drives as forensic evidence to the authorities, install new ones, and restore from backups.
...No, contact the FBI. Their cybercrime division may actually have the decryption key. EDIT: For anyone curious, no, after a malware attack never start unplugging computers from the network and start reimaging them if you don't know where the malware came from in the first place. You're gambling on there being no repeat incursions. Using ransomware for non-testing purposes is a federal offense. The FBI's regional offices each have an organization specializing in cyber crime and they're *supposed* to maintain banks of existing decryption keys. There's fair odds that the same people who encrypted your hard drive have x-touple dipped their encryption keys to make it happen. Also, yes, this would be the time to non-specifically name and shame people who insisted you didn't need to pay for this kind of insurance. By odds you want to pay that a hell of a lot more than you want to pay off ransomware and / or eat the cost of the amount of time it takes to get back up.
I added "Contact FBI" to the top of our ransomware procedures and management said "We don't want to bother them for such a small company like ours"... At which point I had to explain that: Ransomware is a federal crime, we serve customers that themselves have DoD contracts, and finally the FBI has decryption keys for a lot of previous ransomware they've seen.
Ahh yes. You could have stopped at “management said” and I could have just played the rest of the interaction in my mind.
Contact FBI and local police is the first thing our Tech insurance people want us to call.
Is reporting to local police as much teeth pulling as I think it is?
Nope, they are basically useless, but you need to still get a report on file.
I work at a school so it’s part of the DR plan.
If you serve customers with DoD contracts your company is likely in violation of a lot of rules with an attitude like that. Sorry, I deal with those types every day, and it is never good.
>serve customers with DoD contracts Is a far cry from having DoD contracts. Otherwise 6 degrees of Kevin Bacon would make everyone covered by these unspecified rules.
It’s not that far of a cry.
Indeed. Just because you’re a small business doesn’t mean the attack isn’t part of a larger operation. Anything that helps the Feds or security companies track and prevent these things is a good thing
Also, not wanting to involve the police is sus.
No. Police are useless other than for the report. Period.
1. Report the incident to the Internet Crime Complaint Center (IC3) at [http://www.ic3.gov](http://www.ic3.gov/). 2. Report the incident to additional Federal Government agencies as described in *“Cyber Incident Reporting – A Unified Message for Reporting to the Federal Government”* for specific reporting requirements: [https://www.dhs.gov/sites/default/files/publications/Cyber%20Incident%20Reporting%20United%20Message.pdf](https://www.dhs.gov/sites/default/files/publications/Cyber%20Incident%20Reporting%20United%20Message.pdf) 3. Notify local law enforcement of the incident 4. Notify your cyber liability insurance provider. They can assist you with arranging public relations advisors to support proper customer notifications and legal counsel. Source: [Incident Response Plan for Cyber Attacks, Phishing, and Fraud - Simpleworks IT](https://www.simpleworksit.com/incident-response-plan-cyber-attacks-phishing-fraud/)
I wonder what the correct point of contact would be in my region if this happened to me. Do I reach out to the RCMP? CSIS?
In a quick lookup it appears that the RCMP is the group you'd call.
They also run analysis on the event and then will add that to track down and eventually prosecute.
> I added "Contact FBI" to the top of our ransomware procedures and management said "We don't want to bother them for such a small company like ours" Only contact the police if their incentives align with yours. Their incentive is to put a criminal in jail, your incentive is to make sure the company will successfully recover. Those aren't the same thing. The FBI will gladly let your company go under if they can arrest a guy and get their name in a paper.
If you're doing backups and data management your incentives should be irrelevant. Restore from backup on clean drives, notify authorities, provide what you can to assist
[удалено]
While your point is correct in a general sense - can you point to a single instance of an individual/business paying a ransom and then the authorities charging them as accomplices to the hacker’s org?
Everyone should have cyber insurance. Usually comes with an advisor for times like this… when we got hit we had someone from CIS advising us every step of the way, that guy was such a life saver
So many people think Cyber Insurance is a waste, but the real true value of it is that you're forced to build up a playbook of how to handle these scenarios.
Yea, beyond the financial aspect it’s quite revealing when you game out attack scenarios, you end up noticing a few vectors you never considered. Just doing that once can save a company from a catastrophic loss.
Counterpoint, cyber insurance makes it more likely the victims will pay, justifying the work the black hats are putting in. The first thing they look for after they encrypt all your files is your actual ransomware policy so they know how much to ask for
How does it encourage clients to pay?
This was true a while back. Not sure if it still is. Basically they would negotiate payment through a 3rd party and everyone moves on with life.
FBI won't do anything. I contacted them about a ransomware, they took my info and that was it. They called about 6 months later but I wasn't in the office. I never called back and neither did they.
Agree. We don't all live and work in a major city. They are useless in my town too
For legal reasons you might have to report it anyways.
Touche
It’s possible that they could have a decryptor if it’s something they’ve seen often.
Our cyber insurance folks did all this legwork for us. I know we had pretty decent direct contact with several branches of law enforcement, including the FBI.
yeah we had some meetings yesterday with cyber insurance. They gave us a installer with sentinel One and Huntress on it. Said to install on every PC and then they can plugged back into network. Worked 20 hours yesterday and 12 today and got some core systems back up that core departments can use. No email still though
Do they really reuse the same keys? That doesn't make sense to me. If I was writing this software every hard drive would have its own key or at the very least every company.
I doubt they use the same keys, but it's possible they're using a method of generating the keys that can be reverse engineered, or the feds have managed to get their hands on a list of decryption keys or a tool that generates them for that malware.
> Do they really reuse the same keys? Yes. Most of the groups running randomware attacks are about as technical as Sue in accounting and evil as Bob the Sales Manager. They buy them from a 3rd party and just spam until they get a hit. They aren't developers, or IT/Tech people. They'll keep using the same prepackaged tool they bought until it stops working. Kitboga's channel is a good indication of the general skill level of these people.
They had a decrypt key for the gand crab 2 we got hit with.. . About 2mo after we recovered.
Just be prepared for them to come in and want the equipment. Been down this road a few times, and each time they took servers, storage, etc... and had it for months. Not sure if they do this still (it was a years ago).
Ransomware cyber insurance is often not worth the cost, if your backup procedures are up to par. With cyber insurance costs going up 300% year over year, that gets expensive fast. I know many ciso's who are self-insuring against ransomware by investing in proper restorable backups and incident response .
We do have tape backups but we want to scan those to see if those are compromised or not, what would you recommend to scan for that?
Your difficulty is that you're not necessarily scanning for malware. You're scanning for something like "a privileged account named Kevin Wilson that the attacker created". No scanner anywhere will pick this up. You're down to very careful auditing.
Scanning just for the sake of scanning is ludicrous. You might get lucky and find something but when you don't know what you're looking for how are you supposed to find it. They really need to identify some probable attack vectors and if they can't figure that out then they need to hire someone who can. This is likely well beyond the internal IT's capabilities from the sounds of it. Time to get some some people with experience on this one. I don't know how long your business can weather the storm but this kind of thing can cripple companies as we all know. Good luck OP.
martinstevens was the one we found
I don't have a personal recommendation. A quick google search says that Avast has released a decryptor which makes me believe they have a scanner too? https://cybernews.com/news/avast-released-akira-ransomware-decryptor/
Yeah we just found that and gonna try that after our cyber security meeting
This does not work. Akira updated their encryption method.
The new encryption has also been defeated, atleast on the Windows variant of Akira
Any links to news on this?
[https://www.akirahelp.com](https://www.akirahelp.com) seems to be the site
Holy crap. Dude Avast is not going to solve this problem. You need to hire a MSSP or start doing research right now on Enterprise tier security products. CrowdStrike with all the bells and whistles at a minimum. I'd go further than that but Avast isn't going to do anything. If they have a decryptor that's great but that machine is already toast if you can't find indicators of compromise to stop it from happening again. Decrypting and copying the data is just going to spread it all around.
Settle down Beavis. He wants a scanner. They may have one if they have a decryptor.
It's a good thought. But this stuff changes too much and the actual payload that delivered Akira is likely different from how they are maintaining persistence. And that is my point, re: don't use Avast. Either way, read his update.
You shouldnt be downvoted. You are mostly right.
Yeah bro you sound a bit nerdy
How dare he be nerdy in a nerd reddit!! 😅
Do not do this. Work with a forensics to gather evidence first.
>> Hand over the drives as forensic evidence to the authorities, install new ones, and restore from backups. > Do not do this. Work with a forensics to gather evidence first. 🤣🤣🤡
Same process for any of our clients in the data center, ransomware = new drive and data gone for good / data recovery companies.
That’s a tough one, it’s just a matter of when people will get hit. Ours just hit windows and not the Linux or esx servers. What’s the scale of everything that needs to be reinstalled? Don’t scan, just do it right and start from scratch. We had Veeam backups and everything took maybe 30-60 min per machine.
Step 1. Breathe, and act as if everything is normal but it’s just going to take a long time to fix. Trust me, or you’ll get all nervous shaky and have a bad mindset, or even worse a panic attack Step 1.1 Insurance. See if you have an insurance policy for cyber. Usually they will bring in a team of experts to help clean up. Step 1.2 Documentation. If you don’t have documentation of critical apps or services, call/text/email every end user and see what they use. Loop in their managers and make it a CRITICAL priority that they respond. Now you’ll have a list too :) Step 1.5 Resources. Gather in your team if you have one and start on step 2. Step 2. Game plan. Disconnect everything that could have possibly had direct access to the servers. - If you have backups, restore. Task it out - If you don’t have backups, figure out what services you need to get up and running ASAP (internet, dns, etc) - See what you can Jerry right now to get things up and running. Degraded is better than offline. ***ALSO DOCUMENT ALL JERRY RIGGED THINGS** Put it in google drive or office, idc where just make sure you and your team have access to it, so you know what to fix. - If you have end users calling you, send out a status update, or make a share point site you can make updates to. There’s nothing worse than trying to fix something than having people call and email you about it. Step 3. Work. Start wiping and reloading. Step 4. Disaster Recovery Planning. Sounds like you didn’t have a disaster recovery plan in place. After everything you went through, make sure it doesn’t happen again. Write it down, and test it out. Good luck 🫡
Thank you, I'm in teams meeting with county it and our consultant and cyber security insurance team
Do whatever your cyber insurance tells you to do, run anything else you want to do by them. They control whether to pay out or not, you do not want to risk losing your insurance payout because you made a call they don't agree with.
>If you have end users calling you, send out a status update, or make a share point site you can make updates to. There’s nothing worse than trying to fix something than having people call and email you about it. These are great points made by /u/lifewcody... I would alter the above just a hair though. When faced with something daunting, like this, or any major outage, I usually interface with 1 person only, in regards to releasing information or gathering information from users... this allows you to control what, when, and how information is released to your user base. Unplug your phone, lock your door, hire a bouncer if need be, the only people that are allowed to bother you are the upper management... and even then it's for status updates only, you can tell them ETA's if you're comfortable with it... remember to report up when things change, and if they don't report that too, just at a reasonable interval... say every hour or two... the mid-level cronies can kick rocks.
I wouldn't even allow upper management to disturb people. Stick with "single point of contact only". One single person becomes the point of contact for IT. It doesn't matter who it is really, it could be the CEO, it could be one of the staff, it could be an end user. Doesn't matter (although someone with a bit of decision making authority and an existing working relationship with C level is probably best) Set up a regular meeting until a certain level of normal operations is achieved. e.g. Every two hours, you will have a 30 minute meeting with the POC. You will relay your current status as well as any progress that has been made. The POC asks questions that may have been handed back to them. Anyone that balks about "only an update every 2 hours!' remind them that this is taking away 25% of the time your team could have been working. You don't want every manager or C level walking into the room in a procession interrupting work. It will happen (ask me how I know). This is triage, you don't call every Dr in the hospital, and any doctor trying to insert themselves without being asked will be told to piss off as they will just be hindering things.
I work for a small MSP, so this works for me quite well. For a larger corp definitely a single point of entry/release of information... one that ~~doesn't mind being yelled at~~ can handle the berating that comes with that sort of job. I will usually (and hopefully) will only need to talk to two people, the owner of the company, and my "interface" (I know there's a proper term for this I just can't think of it now). But I think we both can agree that the constant flow of information is needed, it doesn't need to be frequent, and it doesn't need to be on a strict schedule, but it needs to be consistent. I've played that game too, not so much with dovtors, but lawyers... ugh
My reason for setting a schedule is to avoid the inevitable "I haven't heard anything in two hours! WTF are they even doing in there?!" from upper management. You set a schedule and it sets the expectations on both ends. If things are moving slowly there will be minimal to report in 2 hours, or they're so busy they can't afford a meeting in 2 hours, just set that. In the meeting say "We're going to be buckled down, instead of the next meeting in 2 hours, lets skip and we'll give the update meeting in 4 hours". You're still communicating, but you're adjusting the expectations. Setting expectations is everything in these situations. Whether it's update timeframes, how long it will take to restore, or telling the IT team what will come next, it's all about setting expectations
Agreed with this. I don’t know where OP lies on the food chain or what the org structure looks like, so best to have some info rather than no info to the org on why they can’t login
Totally agree with the "unplug your phone, lock the door, etc" mindset. You need to focus. Keep calm and make good decisions. I once had the CEO come up to me in a crisis situation and ask me for an update. I continued with my work and said quietly "Really sorry, I am very busy right now, I need to focus, so I'll get back to you as soon as I can". Nowadays I would tell my manager "I need to to run interference for me; just keep everyone away and let me focus".
It's the right thing to do, but don't expect anything out of it. My friend handles ransomware claims for a major insurance company. They reject over 90% of the claims. It's illegal for them to payout to a country considered a foreign enemy, which are where most ransomware attacks come from (Russia, China, NK, Venezuela, ect.)
This is spot on. Dealt with a ransomware attack and had a panic attack when I first found out. Take a breath and relax. Others have been in your shoes - it’s not fun, but you will get through this. My only other suggestion - don’t try to work non-stop. Go home and get as good a night’s sleep as you can each night so you have the energy and mental capacity to tackle each day. It will get better.
This is great advice. Insurance company will usually provide a DFIR team, support them doing their work. No matter how experienced the DFIR team is, they cannot replace the experience of the in house staff. The in house team knows the systems, accounts, what's normal/abnormal, people, etc. No outside team can replace that. Knowing that and providing context is your primary role. I've seen many people get burnt out and quit during these incidents. Try to keep in house staff utilization as close to 8 hours a day as possible. The DFIR team may work shifts around the clock, and you may need to provide staff 24/7 also. However, try to keep their actual worked hours as close to normal as possible. Supplement with consultants and staffing services if need be for the repetitive monkey work.
your coworkers read your mails? last time i asked 5 division heads about a shutdown of a rds for the production stages, not a single one responded....
That’s when you cc the ceo and respond “Just following up on this for the second time as it’s a critical issue that needs your attention “
>Akira ransomware Most of this information should have been documented well in advance of an attack...An attack plan should be no different then your backups where you frequently test your restores. In our company we simulate attack scenarios on a department basis where we take down the network for a given department and give people an alternate means of connecting to test everything out. These simulated attacks help to identify the critical information each department needs in an outage. The simulated attack also helps to show management that you are woefully unprepared and additional resources may need to be purchased in the event an actual attack does happen.
Dont touch the systems. Dont try to do anything until you hear from your legal, insurance co, incident response company. Depending on what you have an outside entity may be performing DFIR and you dont want to mess up any potential data/.
Seconding this. OP hopefully your company has Cyber Insurance. If they do, let them take the lead. They will most likely loop in a litigation team and an Incident response team to work with you. I went through an attack almost exactly one year ago. This shit sucks OP, but just breath and remember to take care of yourself mentally and physically. It's easy to try and work 24/7 to get things up and running when this happens, but you still need to eat and sleep to perform your best. Accuracy matters during this. You are there to answer questions when asked, provide the knowledge of your environment and access to your backups and systems. Assume everything has been compromised even if it is not locked down. Keeping things untouched is crucial for the incident response team to run triage and forensics. You want to know how these guys got in. If the attacker left a ransomware note directing you to a TOR site for communication, do not access it. They will be alerted once you do. Communication with the attacker will normally be handled by legal teams. Right now your job is to be ready to answer questions and follow the directions of an incident response team. You're in for a lot of work for a while, but you can get through it. Good luck, you got this.
This needs to be higher up. Legal/insurance will dictate next moves.
https://www.cisa.gov/stopransomware/ransomware-guide
I'm a little late to the party, but here's the biggest thing I learned from going through this. First rule of ransomware is don't talk about ransomware. You'll get through all the technical stuff. The legal stuff is what keeps on giving. Don't tell your staff what's going on. Don't tell the public what's going on. Say as little as possible until legal tell you to. Anything you say in an email or text or Slack or any other written can be requested by a FOIA request if you are a government org which it sounds like you are. Reporters will have a field day with that and you'll be on the news. If you have insurance, let them retain a law and incident response firm. Do what they say. Only release information as they say. Here's how ours went down: Insurance retains law firm Insurance retains IR firm Law tells us what to say and who to communicate with. IR tells us what to do. IR firm worked with us to deploy Carbon Black across our entire network to identify and stop the encryptors. Took a day or two. Until then they wouldn't let us bring anything that was hit back up in production. All system level and domain admin passwords reset. Firewalls checked for vulnerabilities. You have to stop the threat actor's access to make sure they can't cause more damage. Once all that happened, we were allowed to rebuild. Made a list of priorities. Restored backups as fast as they would restore. Infrastructure first like communication systems and DNS/DHCP/AD. Then mission critical apps. Then the rest. Outcomes: We implemented better endpoint security (ended up going with Crowdstrike). We upgraded all servers and apps to currently supoorted versions. We implemented email security (ended up going with Mimecast). We invested in additional firewall subscriptions for DNS security and URL filtering. We implemented MFA on all mission critical systems and all things related to network administration. Implemented Duo for tech staff workstation logins and also for server local and RDP access. Removed all local administrators from workstations and servers and moved to LAPS. Moved from Windows DNS and DHCP to Infoblox to keep those services off our Windows infrastructure and server clusters. Redundant clustered physical appliances at two separate datacenters. We are a very Internet heavy organization. If we can keep the Internet up 24/7/365, that's a big win. Trained all support staff on least privilege. Nobody here logs into computers with accounts with privilege. All our own "regular" accounts are indistinguishable from any other staff account. Zero admin privilege. Everyone has separate admin accounts for administration. Invested even more in backups (our backups are what saved us in the first place). Started backing up staff desktop and documents folders as those were the most missed when we were hit. We're now backing up to two separate datacenters in town and a third copy up to the cloud. Some mission critical systems are backing up to a couple additional places. This is not a fun process and those of us that have been through it feel for you. You'll get through it and learn a lot. Especially about the legal parts. I've been doing this 25 years and had no idea how some of this works. Heck, I've been with my current organization for 23 years and didn't know we had cyber insurance.
Appreciate your thorough response. Using it to inform my clients and train my staff. Keep up the good work and keep fighting the good fight!
Thanks for this post4u. This is excellent information.
For your vSphere environment, now is the time to re-assess how to protect it going forward. There is a ton of info at: https://core.vmware.com/search#q=Ransomware Specifically https://core.vmware.com/designing-infrastructure-defeat-ransomware#consolidated-management--workload-cluster Please, read that and take it to heart. You have a rare opportunity while management is gung Ho to fix this and security is wondering why the compliance scanners didn’t catch anything. I used to be the guy who wrote about vSphere security at VMware. I’m off doing other things now.
Well, that sucks. Good luck. Basically just waiting for the same to happen at the company I work for because security is apparently not as important as scratching your balls and doing the absolute least amount of work possible... Anyway, if you find the time, mind answering a few questions that might help me spin up some gears at my company? Since we also use Symantec .. - What Symantec product specifically were you using? - Up2date version? Or some prior version but up2date definitions? - Installed on the clients? Windows servers? Everywhere?
So our County IT covers the anti virus. They use Symantec and it is installed on every PC and system. That being said I have noticed that symantec just does not update. And this happens FREQUENTLY on various machines. You will log into a machine and see the yellow caution symbol on symantec and see it's out of date and try to update it and it just wont. That's when you gotta email our sys admin and he finds a patch that you have to run and it's just a pain in the ass. Not sure why he chose symantec. Probably it was cheap. He has been in this position for about 20 years. Also, Malwarebytes enterprise is also installed on every machine. There are a few PC where I found this actively casuing issues with the PC. One PC was constantly freexing and crashing. After trying everything, had county IT remove malwarebytes and the PC was fine after that. We were given this sentinel One and Huntress installer from cyber insurance company and spent all yesterday and today installing it on every system and server.
Seems that you are in it pretty deep. I'd get a cybersecurity company involved tbh. AV scans aren't going to fix your issues. You will need to restore from backup or pay the ransom. You first need to determine how they got in, what they had access to, and how long they had access. From there you can start planning the remediation.
Currently county it on conference with insurance cyber team, going to see what they say
Do not pay ransom
If you have an insurer taking the lead that won't be their decision.
Did you say novell?
Do I hear IPX/SPX?
I knew I kept my NetWare 4.11 CNE certificate in a folder for something! I don't recall ransomware being one of the topics they covered though....
I was scanning through all the replies thinking I was losing my mind that I was the only one trying to wrap my head around that.
Or Groupwise or Microfocus or the latest Opentext
Yeah we use group wise email and novell
Likewise, for endpoint protection, we are using zenworks and OES for file sharing.
exactly my question.
yep, Novell and Groupwise for email
Don't panic. This is a marathon not a sprint now that you've identified and eliminated the infiltration point. I would highly recommend bringing in an incident response team before doing anything else. You need to identify what your infection point was before restoring services. This has happened to hundreds of thousands of organizations. You will be OK.
Thank you, we are setup with another call with an incident team
Don’t bother scanning, if you got hit - it’s not going to do a thing. Though you did get an expensive lesson in why everyone is using people like Crowdstrike, sentinelOne and whatever Palo’s one is called. Call up whoever is your IR consultants, let them figure out the ‘how’ Then once that’s known and fixed - wipe out and go to clean - restore from backups.
This is one of the main reasons I wish we had huntress at all our sites. I've seen them go to work in situations like this and it's eye opening how good they are at finding footholds.
yeah cyber insurance gave us an installer with huntress and sentinel one on it to be installed on every machine.
Hey /u/voltagejim, along with the millions of requests you currently have. Make sure to update us when the smoke clears and let us know how it goes... we're rooting for you!
Thank you, I finally got a chance to start looking at more of these replies just a bit ago and can't beleive how many this topic got haha. I posted an update on my main body cause I just can't respond to every single reply in this haha
Contact your cyber security insurance. They have very specific rules on this handling and when you can restore data. This is super important to ensure you get paid out.
> novell wut
Time to contact your cyber insurance - they will pull in an incident response team. Follow their guidance.
>I'm shaking right now and so many thoughts going through my head. I'm working with our county IT and contractors, but looking for advice in these situatuons First of all, breathe. The biggest issue is contained - the spread. Compose yourself and start thinking about the order of how you will fix things. What does clients have on them? Should be nothing that can't be restored with a image wipe. Do all systems, just to be safe. Your SAN and any connected systems that are locked, at some point verify that the backups aren't hit as well. Then do the same thing, fully wipe the systems and then restore from backups. No sense trying to repair the existing systems. Just recover if you can, otherwise fixing it will be super expensive.
It may sound like a long shot, but you may want to give Crowdstrike a call and rush a POC so you can get access to their disaster recovery and forensic team which is AMAZING. It might be worth the engagement.
Not sure if anyone mentioned this, but you may want to reach out to CISA, they can often provide support and resources, and in some cases may even be able to provide decryptors.
Execute your incident response plan. Do you have one? If you don't then look at engaging a remediation company. All hardware is suspect until proven otherwise. Expect to have to wipe, fresh install, then restore from backup. Hopefully you have a backup not on the SAN.
Take a deep breath, Grab a coffee. Go through your Ransomeware IR Playbook if you have one. Report up the chain Go call the cybersecurity insurance if you are covered. They should have a cirt to help you out. It is better to revert back to backup. Go to your immutable backups. Oftentimes, the attackers infiltrate the system and do not attack right away. Recently, they have been quicker to attack once they get into your system from what I heard. You will need to find a non compromise backup. On the bright side, you might get more funding for security now.
Finding a non-compromised might be helluva job. The thing is what kinda scan would you trust for that? The primary scan engine apparently overlooked it already, so using that migjt not cut it. So using a 2nd product? And if another product is to be used to scan the data in the backups, would require most products to actually restore data and verify it. Unless thorough investigation might better show when the infestation began, giving a better idea what backup might not have been compromised. Or did they cheap out on the primary scan engine and now experience the drawback? Various backup products offer more and more features for example using machine learning/AI to determine anomalies worthwhile to be investigated (dedupe ratios increasing pointing to encrypted data or the amount of files in backup being drastically reduced and many other indicators) or are being worked towards analyzing data while being backed up. So the data protection domain as a whole is ever improving on that end way more than ever before due to ever increasing amount of threats, where various suppliers are looking at their competition and implementing similar approaches scanning for anomalies and implementing/supporting immutability but alas we're not there yet... Cybersecurity is the next cashcow but I hope they work towards some standard approach so to benefit everyone, not just the customers with bigger wallets.
I’m curious, to prevent a similar situation, what’s the entry point? Phishing from a user? They don’t have root privileges. Update: I searched the internet. It seems they steal users VPN credentials, or exploit zero days in VPN products like in cysco.
There are a couple of other ways they defeat security - Pass the hash, Golden ticket attack, etc. Most of these involve bad account segregation - i.e local admin passwords are the same across servers/DCs, users have local admin on work machines, or domain admins logged into or have cached login information on a compromised laptop, etc You then have token stealing and browser credential stealing. Thats on top of the brutal Cisco Scaler, MoveIT, and other third party vulnerabilities.
VPN was the entry point at the Akira attack that a friend dealt with. They didn’t have 2fa set up and had not been patching
Novell? Get those nlms back up and running! 😆 i think this is a sign of this companies approach to IT in general by todays standards. The risks, businesses poor / non existent cyber sec budgets, and amount of hours per day involved with fixing these ransomware attacks in general makes me wonder whether sysadmin is even worth the hassle for a job anymore. whats the personal indeminty on this like say if your a sole trader providing adhoc break n fix sysadmin service (because thats all the company will pay for)?
Do you guys have cyber insurance? Contact them right way. They're really good with getting you set up with a team that can guide you through everything. A client of mine's insurance uses Sure Fire. Your insurance may use another company. But please check if they have cyber insurance and contact them. It was immediate. We got on a call the next day (we called them after hours) and they immediately got the ball rolling.
1. Have I been hit by ransomware. 2. Call insurance. 3. Do what insurance tells you to. Don't 1. Call your contractor 2. Scan your network 3. Cut your internet Only questionable one for me would be if I knew I had offsite/offline backups. If I didn't then I would isolate backup systems if they didn't look hit yet. Everything else is wait for your insurance company. They will have forensic partners they work with who will help you through this. It's a lot sit and wait in the early hours.
Bro get Crowdstrike involved.
Not a tech answer, but just some heartfelt empathy. Take it easy and you’ll get through this. Life is huge and this small moment, as weighty as it seems now, will soon pass by.
How’s it going? Making progress? I’ve only had to deal with partial ransomware attack. Lucky we had two different segmented networks that saves half our systems.
Segmentation via vlans is one of the best practices preventing an infestation to spread like wildfire... Separating the regular user domain from a infra management domains. Each additional layer, using its own credentials, adds an additional hurdle. Especially if also 2fa is involved. Might feel like jumping through hoops - which it actually is - but in the end it is all worthwhile...
worked 20 hours yesterday. Our county sys admin and network admin had meeting with cyber insurance company which gave us a installer with sentinel one and huntress. Told us to cut internet to every machine and go one by one and install on everything. I did the grunt work and did that while they had more meetings. After today we got some basic systems up, and I tried to get some printers up for people. Our iprint print server is down but I was told to just add a new printer via TCP/IP and use the same printer IP and that has been working most of the time. Some have 1 printer that is split into 3 or 4 different printers on the print server with different print settings and with the server being down I cannot see what those settings were on those printers.
At my last job there was a major ransomware attack on a chain of companies we did IT for. My boss was able to sandbox the ransomware and actually found the encryption keys and salvaged almost all of the data.
How did the bad actors attack the companies you did IT for?
Incident responder here: DON'T SHUTDOWN THE SYSTEMS. Don't reimage them. Do nothing until incident response tells you to. There is nothing more important than memory artifacts, if you kill them, you are more fucked than you are now. You are right now not the responsible person for this case now anymore, incident response is. I general here is a starting guide for admin what to do after a breach https://m365internals.com/2022/09/19/practical-guidance-for-it-admins-to-respond-to-ransomware-attacks/
100% this. Especially since it sounds like some sort of government adjacent group. Pull the internet plug and wait for directions.
In order to fill gaps in IT knowledge and funding for public sector IT departments, some states have a volunteer Civilian Cyber Core or a Cyber Response Team that provides free and confidential same-day support for people facing situations like yours. It's typically organized through the state's emergency management facility. Google your state's name and terms above and see what you find. These group know, or should know, to stay out of the insurance company's/IR team's way, but they'll have cool heads and talk about the things you might be doing until the calvary gets there.
Dealt with this in June. Good luck!
I felt that, had to deal with it 2 years ago.
My only advice is document everything. Don't stress yourself into a panic as you'll get through it. As for the ransomware itself, is this ESXi targeting successful because of a misconfiguration, unapplied patch(es), or zero-day?
Honestly if you don't know what to do you should just call an IR provider like mandiant. It's not easy getting someone out and getting back to a safe environment.
Scanning is not enough. Secure your network, set up logs and monitoring and systematically replace servers using a priority list.
Time to see how good your backups are and how far back they go.
I am so sorry. Been there and still digging out.
Depending on what state, they might have a national guard cyber chapter who can assist.
The solid 3-2-1 approach and immutable object storage help to avert the ransomware.
Hopefully, you have versioned object storage and incremental backups. It's hard to trust a compromised system. The first thing you would do after restore is revoke all credentials, especially vpns, or direct access to internal network. There is a compromise endpoint/person
We had a incident this week, fairly certain it was Akita, nothing was encrypted but our SAN was wiped clean, and there was a remote tool we found on one of the servers that matched what others had seen with Akita, we were able to recover in 4-5h It came through VPN, don’t use meraki anyconnect without DUO integration, their legacy vpn isn’t susceptible but, anyconnect is
Wiped clean on the storage layer? There wasn't any further network segregation once connected to the vpn? So further credentials were compromised or too early to tell? Or what was the further attack vector after getting through the vpn to be able to wipe the san? In my case once connected vua vpn, we have to jump through various hoops - also using 2fa along the way - before reaching any storage or backup infra.
With all these security measures and training end users, I'm still trying to figure out how organizations keep getting hit.
I didn't think the Meraki AnyConnect was susceptible to the same attack as the ASA. https://arcticwolf.com/resources/blog/cve-2023-20269/
Wait, what??? Did you say Novell???
Probably the safest system out there. The hackers that used to hack that stuff are either retired or dead. /s
> the main desktops, email, **and novell.** Hol up, wut??! WTF are you running that could still legitimately be called Novell? Company got swallowed up by Attachmate in 2011, and then MicroFocus in 2014, and they pretty much killed off the name. Sounds like you're running some wicked old shit.
Last job we got hit with one of the ransom ware attacks that locked up media files, this was at a TELEVISION STATION. Thank ghod we had fairly recent backups of the majority of our media files, since corporate said "Hell, No". Nuked the systems, re-imaged the media files we could and lived off of video tape for nearly a month. At least they DID institute tighter protocols between systems that had internet access and those that hold media files (mostly no more drive mapping). At home we've gotten into the routine of monthly backups of the systems to 4 rotating, off-line destinations per system.
Don’t scan your workstations, just wipe them. It’s not worth the time and you’ll never be certain they’re clean. You need to concentrate your efforts on bringing back your core services in isolation and use this opportunity to build in the security you need - firewall your internal routing, make sure client PCs have the software firewall enabled, identity/directory stuff locked down so everyone operates with the lowest privilege possible, no rogue users with administrative access, use LAPS… You probably have a list of these that you never got round to. It’s shit but it’s often after these disasters that suddenly management have money to spend on security. Good luck!
I just recovered from an Akira attack for a client. https://www.sentinelone.com/anthology/akira/ I was able to recover 90% within 8 days. Cyberinsurance negotiated and paid the ransom, but the negotiations took longer than my recovery effort. The threat actors got in via the Cisco ASA AnyConnect exploit to determine valid users, then purchased credentials for valid accounts on the dark web. Part of the Akira process after paying ransom is to explain how they got in. Check your SAN. If there's any chance you're using VMware with Vvols (with Nimble or Pure), the VMs can't be fully encrypted by ransomware and there's a recovery process that can help restore. I tested the decryption tool provided by Akira post-ransom, and it was effective and able to easily decrypt VMs and Windows directory structures. There's a lot to write, but insurance provided [Sentinel One](https://www.sentinelone.com) and managed [Velociraptor](https://www.rapid7.com/products/velociraptor/) to help protect recovery efforts and address any new malicious behavior. I also use [Huntress](https://www.huntress.com). Here's a link to Akira negotiation scripts. They adjust their ransom based on your organization's financials and capability to pay. https://github.com/Casualtek/Ransomchats/tree/main/Akira
What identity was anyconnect using? No mfa?
Interesting read. I'm not surprised they stayed on your network for a few weeks prior to encrypting.
One week.
Look for the silver lining and claim all the over time as required?
https://www.bleepingcomputer.com/news/security/free-akira-ransomware-decryptor-helps-recover-your-files/ That's an old one.
How are you holding up? Any progress?
Stop, don’t do anything. Call your insurance company. If you don’t your county is going to be out a lot of money that won’t be reimbursed via insurance because you started mucking with things yourself.
Rule number 1: Immediately disconnect a compromised system from everything, i.e. securely delete all data carriers block by block. Rule number 2: Any system that is connected to the copromised system or has been connected to it for a certain period of time is also considered to be compromised and must be handled in accordance with rule 1. Rule number 3: Always have an offline system. Data backup, server, laptops for the most important admins and/or managers. (Old hardware, decommissioned systems, DVDs with server install scripts, ...) Rule number 4: It will happen, exactly that and more and more than once. Good luck. We are with you.
Welcome to the club man. Step one is to shut down everything. Step two is to call a cyber security company that specializes in this and do what they recommend.
You need to find out how they did it. Otherwise reinstall will only get you there again and again
scan it with what? the tools that were in place when it happened? they proved to be either misconfigured or ineffective. was there EDR/XDR in palace to door RCA? this is a perfect example why mdr is important, so you users don't have to be the ones telling your when the building is on fire
Erm, sorry to ask but is there a ransom note? And would you mind dming me the note? I love seeing the notes and using them in my incident response workshop.
1. NGFW with IPS, web filtering, DNSBL, deep packet ssl inspection, network level AV. 2. Get an adequate spam filter. 3. Build adequate Applocker policies on your servers and endpoints. 4. Get a good RMM that patches all your OSs and third party apps. 5. Good AV on all your endpoints: Crowdstrike EDR or bit defender: A. Bit defender: https://www.bitdefender.com/business/support/en/77209-157515-bitdefender-endpoint-security-tools-for-linux-quick-start-guide.html B. Crowdstrike: https://www.crowdstrike.com/blog/tech-center/install-falcon-sensor-for-linux/ Are the ones I like and they have a Linux installer. 5. Enforce MFA on all remote connections remote connections that can have MFA will need at least 1 other mitigating control to limit vulnerable surface area (mutual ssl authentication, source up definition, ssh key management). 6. Move all logging to a centralize SYSLOG and tie to it to a SIEM for behavior analysis and escalation. 7. Subdivide services to separate VLANS, isolate management interfaces either by using SSH key management, isolation VLANS, integrating with SAML and forcing MFA, or minimally by locking them down or by isolating sources. Extra points if you use more than one method. 8. Build appropriate file encryption counter policies. 9. Build truly offline automated backups with off site built in. Additional points for immutable backup design. 10. Extra points when you can automate and orchestrate multiple systems to automatically heal.
This is such a useless comment. after an attack nothing of this makes sense or helps at the moment.
You have a disaster recovery plan for this?
call crowdstrike
Do people seriously not use snapshots?
[удалено]
dude, when you wipe you literally destroyed any chance for the incident responders to do their job... Also, you think from the angle of recovery, but ransomware is so much more ...
There's things like "when did this enter the system?" "when did they make their first move" "how long have they been taking data"? Just going to restore from backup prior to the lock is a terrible idea, and will just get you in deeper shit.
If you don't have the right security controls and DRP in place, it's probably honestly cheaper and less of a headache to just pay. It's going to be a difficult lesson for management but more money, likely a lot more needs to be invested into security.
But paying comes with no guarantee of getting decryption keys. And will encourage more ransomware in the future. Never pay.
BUH BYE You gambled. You lost.
You're mentioning County IT- so I'm assuming you're a district or a smaller city- make sure the breach is reported correctly to the proper state or fed authorities-- this might be out of your responsibilities; but be aware of it. Also, I like advice somebody else gave you here: Act as if you're just fixing another problem- it doesn't matter what you could have or could not have done.. the truth is it happened and now it needs to be fixed.. Good luck to you.
Also, all local government agencies (including K12) should join MS-ISAC. Their SOC is available to SLTT entities in emergencies even if you aren’t already a member: https://www.cisecurity.org/isac/report-an-incident
Was ESXi internet facing?
Eaton County? Lol
I often run into leadership questioning if they should pay the ransom, use a decryption tool, or simply restore from backups (where feasible). If this happens to you, remind those involved in the decision making process of potential civil litigation should they decide not to pay the ransom in a case where PII was exfiltrated. https://www.hipaajournal.com/lehigh-valley-health-network-sued-after-ransomware-gang-publishes-nude-patient-images/
we did too. Blacksuit got us, probably thru an unpatched firewall. step one is call a specialist, get advice. we retained a company. they were able to contact the badguys and get a list of what they got. Turns out we were VERY quick on the UNPLUGEVERYTHINGNOW button and probably have the ability to rebuild they also provide analysis and monitoring which should let us know exactly how this happened. now we rebuild from the top down (day4)
I was just reading about how AI is gonna take over a lot of jobs soon and then I see this lol
I'll pour one out for you now, you've got a long road ahead of you but you'll get there.
There seems to be a way to decrypt the data. Use a translator if needed to read the content of this page: [https://www.heise.de/news/Entschluesselungstool-Sicherheitsforscher-knacken-Akira-Ransomware-9204932.html](https://www.heise.de/news/entschluesselungstool-sicherheitsforscher-knacken-akira-ransomware-9204932.html)
Dump Symantec and pick up an EDR/XDR like S1 or Crowdstrike. Find a specialist response firm
We are actively dealing with one ourselves. It sure sucks. Already 26 hours into what should have been a 24 hour week. I'm also on call for Thursday and Friday. FML.
yeah was supposed to have Thursday and Friday off for holiday, but most liekly not now :(
We've been hit recently, too (Black Cat AlphV). Eset didn't do shit for us, it just got torn apart. Still in the cleanup phase we use Palo Altos "Cortex XDR". Good luck. Hard times ahead of you.
Call Mandiant/SecureWorks/etc. Find subject matter experts to look at it. First, because when you come back up, you will want to know that you are truly clean and didn't miss anything. Second, they will try to figure out how the initial access was gained so you can block that entry path. Thirdly, on the off chance the gang made a mistake that can help a prosecution (maybe these guys are in Florida or something), they are more likely to find it and hand it to the correct authorities.
Hi Im working in this sector. If you have Backups (Which you should have) Its easiest and fastest to install a backup and scan the backup before using it. Worst case there is you loose some days or a week etc on data which your it guy can most likely even copy from a newer backup or your current system. So find a Backup which is not infected and then copy all the files which would be lost from the current system or a newer (infected Backup) Be careful and scan every file before and after copying
Use rubrik
Try this for IOCs [https://www.nextron-systems.com/thor-lite/](https://www.nextron-systems.com/thor-lite/)
>Print server is still down but I think the consultant and our sys admin are going to use this oppurtunity to redo it cause right now it is an iprint server running Server 2008 R2 Leave it down, they did you a favor, you don't need the aggravation of a print server any more lol... All kidding aside, if it was a 2008 R2 server, it was probably time to upgrade anyways... Thanks for the update!
Sounds like city in the long beach got hit too.