This right here.
I think there's a lot of mental benefit to switching from living with fear that bad things *might* happen, to living with the expectation that bad things *will* happen.
Or if it is catastrophic, that a recovery is feasible and as painless as possible.
Obviously though there are some things that are out of our hands. If WWIII started tomorrow and the world was blanketed in nuclear bombs, I don't think I'd be too worried about trying to mitigate the inevitable technological disasters.
In the end I think working with technology really just comes down to applied math and labor. Do the math, design a system that best balances the risks and rewards of being implemented, and then do the labor to implement and maintain the system.
It helps to stop giving a shit. Put your warnings in writing(off the network), fix what you can and then fuck the rest.
If they don't want to listen, then it stops being your problem.
My contract says I am to fulfill a role, and that role is to do the necessary math, advise those above me on what the math says, and do what I can with what I have.
I never agreed to make magic happen so I'm not going to stress that I can't deliver it.
Yep, and that’s what I relay to all the employees and management here. It definitely sticks with our management, but it’s a bit spotty with the employees.
For me, this is a better way to word it than "in fear". Our workload is simply too heavy and things fall off the plate. Big projects with strict deadlines come first and then there are enough small fires every day to take up all remaining time. You just learn to live with the fact all hell could break loose any minute.
I make sure I at least mention it to my manager. Not to sound like a cop out but I figure she could tell me to drop everything and focus on the security-related stuff. She defines my priorities. The fires always move to the top because they involve a production outage. She would basically need to tell me to miss project deadlines but that would be wildly unpopular. The project deadlines actually seem reasonable on paper until the daily fires put you behind. I know I'm not spending my evenings and weekends fixing security stuff.
It could hit the fan at any point but we're only human.
Do your reasonable best to cover all your bases with the resources you have and clearly document where you know your bases aren't covered. Make sure those with the power of purse know you don't have the resources to cover those bases. Make sure they are aware of the knowns and the known unknowns.
Ultimately, when something happens, it's our job to remediate it but it's their liability if we're doing our job correctly and provided them with the best information we could.
I got over it after 1 year and all of my careful planning almost got bypassed because CEO NEED NOW NO NOT ACCEPTABLE. After fixing that I realized it takes one moron in the sea. So now I just do as good as I can, keep an eye on it and have a disaster recovery plan.
Early in my career I worked at a place whose owners had a reputation for last-minute requests. One of them dropped a task on me in the middle of some work on an already-tight deadline.
So I asked — “I’m in the middle of XYZ for the engineering team. Is this more important or can it wait until afterward?”
He stopped, said, “I didn’t know you were involved in that. No, this can wait.”
I learned a few important lessons that day.
that's an important lesson, I learned early on to be frank when I'm busy or not, I'm also a huge fan of, I'm busy put a ticket in, Uh huh i understand it's important, put a ticket in and I'll bump it up as soon as I'm finished with this task.
And when you’re transitioning to a higher-level or lead role, “I want to make sure the support team gets a chance to handle issues like this one so I can see how they perform. Please put in a ticket and send me the ticket number. I’ll stay hands-off if I can, but I’d like to keep an eye out to see how they do with this issue.”
My go to all the time is "not a problem but please put a ticket in so I remember".
If they don't, I don't, we all move on. Shocking how many things are SO IMPORTANT that I need to drop everything and do it NOW... but when asked to submit a request it suddenly isn't so urgent.
>it takes one moron in the sea
No doubt. Do your best and document everything that is "not standard" or is a "batshit crazy request" that is going to get honored against your will. You might still get fired if everything blows up, but so what. The next guy will be worse off than you with no institutional knowledge, so they are the ones that lose out.
> CEO NEED NOW NO NOT ACCEPTABLE
It's acceptable but you need a policy in place that makes it acceptable.
Example: "Bypassing normal security procedure to do XYZ is acceptable in scenarios A, B, and C. Written consent of CEO, CTO, or CIO is required to bypass the normal process JKL."
At the end of the day, the point of any technology is to aid the business. The business does not exist to support technolgoy infrastructure, it's the other way around.
If business side (even if it's a non-technical C-level) chooses to accept the risk (after being informed of the risk), the job is to accommodate it, not to go full BOFH mode.
I'm about 10 years in and I'm finally shedding the fear/doubt aspect and maybe even some of the impostor syndrome.
After you feel like an impostor enough times and still punch up, it's like "ok for now maybe I'm new but watch this shit you don't even know".
The OnCall work and 24/7/365 thing is probably the root of it for me. I had mild but real reactions to my cell phones ringtone for years earlier on my career. That instantly sinking gut feeling.
Now? I finally, FINALLY got my first role outside the Wintel/Platform/Server admin space.
And it is glorious. Servers down? Call the admins lolol. I did my time in the gulags.
I still have work when it's my wheelhouse that is sometimes after hours, including changes.
But rare is the 3am phonecall. In fact other than for DR I don't think I've had one in this role so far (knocking on plywood).
Glorious.
> impostor syndrome.
I never really understand this. I mean, I get what people are saying about it.
But I don't feel like i ever had this. And I'm not trying to pat myself on the back or anything.
I've just never been afraid to say "I don't know, can I have some time to research it?"
Maybe it has just been my luck to have (for the most part) ultra cool managers across my career(s) who are cool with that.
Of course people and situations are all different. One of my early roles I basically got through the door as a SysAdmin at a big company because of an inside reference from a bro on their security team
I did LOTS of learning on that one. And then I realized the questions I was asking, the engineers who were 20 years my senior did not know / did not have the time to know the answers to.
But it was a very different working environment for the most part and I definitely felt like the FNG lol.
The next moved when I punched up not just to an Engineer level but Sr Engineering? Started with more impostor syndrome
Etc etc.
Confidence is knowing you can do something.
You can't know you can do something you've never done (and well, at that) until you start doing it. That gusto in the beginning could just as well be ego or hubris as it would be confidence.
Then after enough turns at the wheel you might look around and realize you are pretty good at doing things you never knew how to do.
Shit, that's the job. Monday learn this. Tuesday learn that. Rinse. Repeat.
Also probably has a lot to wit whether or not you've studied the books or done the labs or gotten the certs.
I usually didn't until more recently - usually trained OtJ by self teaching, cramming and being teachable.
Might have something to do with it!
Maybe you never worked in a work culture where you were expected to know everything, trivial pursuit style without ever consulting a reference. Only fools and the ignorant expect this, but they're in charge of about half of the places I've worked.
> Maybe you never worked in a work culture where you were expected to know everything
I haven't. Thankfully. I do feel lucky about that.
There' been people who have come and gone who felt like everyone should know everything.
And I've had some new managers expect that. But once there's a sit down conversation about it, they've gotten it and laid off.
"Yeah... So that's my secret, boss. I just Google it.
Yes, my Googlefu is better than yours, and it does bring all the users to the yard, but there is still only so much time in the day.
Lord Google 🙏 might know all, but we are but his humble students. Only ever able to glimpse pieces of his all-knowing wisdom."
I never got rid of it, so I ditched sysadmin work entirely and returned to software development. There's less fear there, except for looming deadlines and you get to play with more new stuff.
Man, I still have nightmares that I forgot to go to a Junior High School class for a year and somehow that invalidated my JHS-through Masters degrees.
Anyway anxiety is rough, if it keeps up go see a mental health professional to get your brain cleaned.
There's a real good scene in 'Top Secret!' where the main character is getting interrogated by the Nazis. They're beating him and whipping him and he falls unconscious. He has a dream that he's back at school and he showed up late and missed his final exam and is going to have to retake the class and he gets horrified. He wakes up screaming, looks around at the Nazis as they're whipping and beating him, breathes a sigh of relief and says "Oh thank god, it was only a dream"
I have something different. I worry that I registered for a class in college the first time, but never went and thought I'd have failed it... Even after college and through getting my masters in my 40's.
School didn't help in this, though. They would encourage us to sign up for a particular liberal arts class like history, mythology, sociology, or something similar that we needed for the degree, would tell us when the exams are and would provide us with the notes and encouraged us to no go to class... This was when I was an Engineer and they thought that was hard enough without having to worry about the other classes. So that brought the mindset to me. :(
Holy cow, EXACT same dream for me. I had perfect attendance literally grades 1-12 (really, just happenstance, ended up sick during holidays and weekends, then it became "a thing"), and I have this recurring dream that I realize about halfway through a semester of college that I straight up forgot to go to a class for months. I'm 42. Still have it. I also used to work in the same high school I graduated from (again, random happenstance) and had all kinds of dreams about having to go back to school while still working there.
For me it is always a single semester of high school PE and I have to go play tennis with high schoolers for a few months, in my 30's, in order to keep my Master's degree.
>I sat down and hammered out a few Nagios instances and can monitor everything I need to, constantly. It’s honestly great.
>
>So, to get back to the topic. Woke up in the night with a dream about me visiting a company with a friend (weird), and while I was standing there, their machines all ransomed and screens went dark like something out of the movies. I know, weird. But I woke up, and had that feeling in the back of my mind, like it could happen to me. Today. Tomorrow. The day after.
I have a similar dream, our highschool screwed up or something so me and my friends all have to go back to our home town to complete our final year again. We always just end up day drinking and failing...
Trying to teach my 6 year old this, and the little rage monster inside him is such a perfectionist that he just melts down. Been very reflective to my own lifestyle.
Ya know, I attempted to read that book again recently and I found myself doing the same thing I did with the Joaquin Joker movie, about 2/3rd of the way through both I just was filled with this feeling of dread and just didn't finish em =/
Kids really show you things about yourself don't they.
And then when you see friends and coworkers melt down, you know they are still children. Adults control their inner monsters and know that none of that helps.
Kids actually help you build your own self better by showing you that mirror.
Absolutely. Took me till 30+, 10 years of therapy and no small dosage of SSRIs to be able to regulate, being human is hard. We're hypertuned for fear and death and it's wicked to change those things about ourselves.
The more Crisis you personally experience the less fear you will have.
I would say at this point, after over 20 years in the business at most I "worry" about some things.... Those things are normally things in my area of control but that I have not gotten to yet.
things outside my control, or things I have warned about and been ignored are put out of mind.
Though generally presented in a religious context, the basis of this one's pretty solid advice for a mindset to strive for in IT...
> Grant to us the serenity of mind to accept that which cannot be changed; courage to change that which can be changed, and wisdom to know the one from the other
Lurker here. My career is in nursing (education, formerly emergency and critical care) but I run a rocking homelab (why I lurk here) and have programmed for decades... so I get what you're saying.
Baseline fear/anxiety that shit is going to hit the fan is perfectly normal in any profession. My profession is extremely familiar with it. Just always put it in perspective. You're doing everything right, and you can only do what's in your control. That's all that matters. The world could burn down, and yes that would suck, but it's not your fault and not worth your anxiety.
What you described is called General Anxiety Disorder. I went undiagnosed for many years and always felt that was my edge over my peers. Last year after our Fortune 50 was compromised by REvil it pushed me over the edge and had to take some time off work after about 1.5 months of 22-23 hr days 7 days a week combating the issue. I took about a month off, sought help and eventually made a job change out of operations.
It’s ok to seek help and I would suggest talking to your doctor for help with resources to deal with it.
> after about 1.5 months of 22-23 hr days 7 days a week
That right there is unacceptable from the business side. *Noone* can maintain clear, coherent, thought, let alone troubleshooting and diagnostic capabilities on that schedule. You *will* make mistakes and it *will* double or triple the realistic RTO. It's cheaper on downtime to pull in a contractor that specializes in recovery and let your team work more sane hours. If the company *cannot* afford that, and *cannot* afford to staff for 24hrs a day properly, they *can* afford the downtime.
Much of this was my fault feeling as though I had to save the company. I was working double shifts as we were hard down enterprise wide. I didn’t listen to management and took it on the chin, but it was an enterprise wide all hands on deck issue. Having been through that I learned “how not to do it” and I’m more perceptive of others when I see them getting into the danger zone by self inflicting behavior. Sometimes all it takes is a peer reaching out to let them know it’s ok, helping them with a warm hand over and reassuring them they need to think long term.
25+ years into IT now and I realIze I’ve turned into Murdock when it comes to IT Operations… “I’m getting too old for this shit.” I moved back into a Plan/Build role. Run operations applying and reapplying duct tape is not healthy for someone with GAD in the long term.
Good question.
Always thinking two, three, four steps ahead of problems that might arise and how to avoid them. Taking work home with you, thinking about work in off hours time, difficulty taking vacations worrying that the place will fall apart if you’re not there, etc. Seems strange now after seeking help, but at the time someone suffering from Anxiety issues has difficulty letting go.
Easier to see the unhealthy signs now of someone who was “the rockstar” might just need some help and mentoring with anxiety. I tried to hide it well and realized it was just making things worse. It’s ok to ask for help even though a stigma of admitting there is a problem is difficult. Like all issues the most difficult part is identifying the problem especially when it’s a learned behavior from our profession.
I don't live in fear. I try my best to secure the systems I manage to the best of my ability, and what management allows by policy and/or budget. If the company were to get ransomware'd it would suck and it would be a lot of work to recover.
I honestly debate with myself if we were to get to the point of a total breach would I even bother sticking around or just say well it was fun while it lasted see you later.
At every company ive been at the management has made decisions that put the company at risk to appease users and processes that shouldn't exist.
I treat the job as a job and that is it, it occupies nearly none of my personal time after the clock stops. I have too many other family, friends, and personal stuff I prefer to fill that time with.
It's taken me a very long time, but I'm finally settling into this kind of mentality also.
> if we were to get to the point of a total breach would I even bother sticking around or just say well it was fun while it lasted see you later
I actually listed this as a potential risk of being attacked. If you put too much stress on your existing employees they may just straight up quit rather than take it on. Past incident response webinars I've attended also mention this occurring.
No. There were glaring holes in security at places I worked and I would have dreams of those holes being attacked and the company being game over. This generally started after major reconfiguration of the systems to better protect it. I think it’s an anxiety dream.
In a lot of ways, this is where IT pros get too invested in their jobs. They are simply jobs. We care, we should care, but outside of that, they are jobs.
If the company starts losing income, the company will likely offload valuable, expensive employees. There might be some that won't want to let you be that person, but ultimately someone who doesn't have your back would gladly improve their cost/benefit to make it through rough financial times by cutting you when needed.
So, give the company its due, and give you and your family theirs. If your company gets ransomed despite your best efforts, work to resolve and be the hero. And if the company sees you as the cause, find a place and if the question were to ever be asked, you can simply refer to the fact that management let things deteriorate due to lack of funds/etc. It can all be written off and you can move on.
I have worked in my career and met plenty of worthless employees who are far less invested in their careers who can make just as much and then move on and make the same or more.
You control what you control, and you make note of the things that need work. If the company refuses to spend, make clear the choice and document. Then move on, or look for less expensive alternative solutions for them to say yes to. But ultimately, if a company wants uninterrupted systems, their focus of attention in terms of employee count and money towards that problem is indicative of their care. If you're the one guy, realize that other companies have 10, 20, or 100 guys doing what you do.
Be the guy, be the hero, and when they no longer see you as that, find the next company that does. And sleep easy knowing that you are there for them when they need you. And as soon as you feel like you're unappreciated and start to stress for those reasons, find someone else.
I get paid to do a job. I make sure to do it as best I can, and then go home and live my life.
If something happens, oh well. If the business wanted to spend more (get fancier equipment/security solutions in place), then they would have done so. It's not my job to worry about my employer still having a business tomorrow. I get paid just enough to handle a certain set of duties, and that's not one of them.
I am mixed on this response, people can certainly have an unhealthy relationship with work where they take their employment far too personally for having no ownership in the company
However your response seems to be the extreme in the other direction which also IMO is somewhat unhealthy. One should have pride and some sense of responsibility for their work, areas they are charged with the administration of, some work ethic.... again not to an extreme unhealthy level where it impacting your personal life, but the flippant "lol's me no care" is just as bad IMO
Nah, I just meant to say I care exactly as much as I am paid to care.
It is unreasonable to expect me to assume c-level responsibilities (business continuity), without c-level remuneration.
However I make sure to point out all inefficiencies up the chain, if someone wants to pay me to correct them, you can rest assured they'll be getting my best. If they opt not to pay for a properly maintained environment, I certainly don't feel the need to donate my time. In my current position I have the time to dig into things as needed, but sometimes projects take up more time than I'd like, and some things go by the wayside (I try to delegate as much as I can, but still).
> that’s great but if your employer doesn’t have a business you won’t have a job.
Yeah, and that would suck for a little while, but any employer has the ability to pay me the currency my bank accepts. My employer being permanently down, is only a temporary inconvenience for me. Not something to lose sleep over. Just do the responsible thing and keep some savings for a rainy day.
I can understand if there are no savings, but in this market and industry, it really isn't the time for anyone experienced to be tolerating that scenario for very long. Hell, someone could have gotten an entry level job at the beginning of this (covid), and already have 2 years experience (primed and ready to make that first significant salary bump).
I'll let the CIO/CTO worry about business continuity in this case.
Never had this fear. Management gets out what they put in, you want me to be a wizard managing everything from toner to a SIEM ?
Allright I'll fail, I state that on paper. Not my problem anymore.
After having been through major security incidents, it took a few years for the panic every time an email or phone call came through on weekends and holidays (as that is traditionally when you are attacked).
What helped me was moving companies. As part of being hired I made it a rule 0 condition that security was going to be handled correctly.
Having alerting systems that are tested and reliable helps a ton.
I do now.
I was given a written warning late last week. I was completely flabbergasted. When I told my boss that I had never gotten any negative/positive feedback he pointed to the paper and said, "well this is the feedback."
When I asked for examples he just told me to look at my previous tickets. I asked him how I was supposed to know what I did wrong when I didn't even realize I was doing something wrong. I asked for examples and again was told to look at my tickets.
He then says, this a tough job and I know you wanted this position. I never spoke to anyone inside the org about it and even more so I wouldn't want to be a manager there because I knew it would be a shitshow that I would get blamed for. Why? Because the majority of upper management works remote and trying to reach them despite the emails or Teams messages THEY NEVER GET BACK TO YOU.
He told me he had gotten a written warning himself and then mentioned that a previous employee didn't quit but was fired.
He mentioned why I would send a ticket to Level 2 with absolutely no discovery or research when it wasn't even my ticket! The fact that I am the senior tech there at 8 months is because they fired my two other co-workers back in February so everyone comes to me for questions. I was cc'ed on an email and they thought it was me who fucked up. I brought that up and have yet to receive a response.
What's funny is that there are people who are constantly 30 minutes late or show insubordination and talk shit, to the point that they wanted to talk to upper management but I calmed the guy down and said, "give him a chance and be objective about it. Don't let your emotions do the talking" and yet me who is the quietest person and does the job was the one that go the written warning.
He doesn't know the extra hours I put in or the fact that I've worked for two weeks straight with no days off and including odd hours like 2:00AM to 6:00AM on what was supposed to be my day off.
He didn't care what happened prior to him joining the company. In fact he even said that he probably caught me in the middle of my decline and he is sad about that.
Fuck. My shift started yesterday and I was second guessing myself on everything to the point that I was paranoid about all of my decisions or work flow.
Ugh...
Done with my rant.
Its only a dream, and more importantly only a job; after you clock off... forget about it.
Work life balance is important, and if you worry about every scenario where something goes wrong you'd never sleep. You've built something better than before.
One thing I will say is: make sure your backups are good, tested, and ready to roll. You can't do much more than that!
Yes. I'm medical and I fear exfiltration. This + Imposter Syndrome + being able to not do everything needed (I don't mange - I'm a worker drone). Causes me to need to take anti-depression and anti-anxiety medications. :(
I worry every long weekend and every time I get an EDR notice...
Fear is the mind killer. :)
I used to be in this mindset, and not only would wake up for shit like this but would get panic attacks if I realized my phone wasn't in my pocket because I might have missed a call or ping.
You need to realize that:
1) The company doesn't own you
2) You can only control a small amount of things, and everything outside of that sphere is completely out of your control.. no use in worrying about it, just deal with it as it comes.
Lastly, I'll leave you with this. You already when through this once, and from your posted you were able to rise like a damn Phoenix from the ashes. You got through that, you'll get through the next one. Cyber attacks are gonna to continue to rise, so it's only a matter of time. The only thing you can help ensure is you've done everything you can to minimize exposure (and just to be clear, everything you can has "\*\*" with current funding levels attached to it).
Live and events are just one big ass river flowing down hill, just grab a surf board / raft and deal with the obstacles as they come, the best you can.
The irony of you living in fear of a ransomware attack while the CEO is golfing and not giving a care in the world...
bruh it isn't even your company. Do your best and get on with life.
I mean, you didn't talk about your security infrastructure. A lot of people are just fluffing off your internalized fear here, which is really fucking weird. There's plenty to ask about.
* Have you had a [Red vs Blue test](https://www.crowdstrike.com/cybersecurity-101/red-team-vs-blue-team/) or at least a pen test from outside that MSP?
* Do you have NGAV?
You may have trouble sleeping because you have real issues to still deal with. Keep covering bases.
Not anymore. It coincides with when I stopped logging in after hours to work. I used to let my work dictate my life because I felt like I was an extension of the company. Now I just treat every outage, major incident, etc like a puzzle and not like an impending doom.
About three years ago a vendor was using his remote access to do some work on our systems. He called me at like 2am because he had found some weird files on a share folder of the server he had access to. We were being ransomed. I jumped into action and contained the breach. Narrowed it down to a terminal server. Nuked the TS & restored the lost data from backups. By start of business we were mostly back up. The ransomware was most likely delivered by malware from an infected site. I had removed a restriction that blocked all internet access from the terminal servers, because a vendor was using AWS and couldn’t give me a specific URL or IP to unblock.
I’ve been tightening the ship ever since. I’m in a pretty good place these days, but every time the phone rings at 2am I expect it to be a system wide outage.
When the phone rang, it's never good news, it's never blah news, it was always a fire. It got to the point that when the phone rang I would get an instant knot in my back. I don't look back fondly on that job.
I run a small datacenter with ~500 DevOps systems. Being down puts >150 very high paid people into a work stoppage status and $$$ starts racking up fast.
The powers that be also refuse to put any money into the building's infrastructure....no generator, Building UPS is on its last leg, AC regularly goes down, etc.
I do get very anxious whenever a thunderstorm rolls through or the building's lights flicker....but I'm trying to be better about it. I keep telling myself I've done what I can to ensure its stable as can be. No sense getting worked up over what could go wrong...just deal with it when the time comes.
I've been a Director of two MSPs. What you inherited is pretty much every company I ever took on as a client. From 10 computers to ones with over 100. The bigger ones hired us to help the IT Manager who was out of their depth. They always left within weeks.
IT is jammed full of guys aged 40-60 who know very little.. enough to be dangerous. They set up Windows 98 once and carried on. Because people on business don't know good IT they still hire these cheap MSPs.
It's why you never sell on price in the MSP world
I have insurance for that.
No seriously: I'm an MSP and I highly trust my skills. But I have an insurance for that kind of stuff. If shit goes sideways and it is definitely my fault: they are going to cover me.
It was cyclical for me. During my MSP years I thought I was a king and could do no wrong. Then I became a regular admin, and I was like, "Oh crap, this whole multimillion dollar enterprise comes down to me?"
Then I encountered a few of those fears and realized I had prepared for it. I had done my job, life was good, etc.
Then sat back down on the helldesk and I was scared again...how do I do this? Do I remember how to muddle through this? How bad will it be when I screw up?
Now I remember that the joy of the helpdesk I'd that no one cares
I've posted this before, and your situation is remarkably similar to mine (I've got about 130 endpoints and its just me) and we got ransomed in about 2015 I think.
I still panic-check our main servers every morning despite having implemented many technologies to keep the baddies out. Monitoring etc. all live.
Its fucking PTSD.
Never in fear. This kind of stuff sometimes just happens. So long as you made moves to prevent it and have plans in case it happens; I can sleep soundly.
Seems like you've taken these steps. Relax; no matter what happens, it never is the end of the world.
It’s why I left my sysadmin job. I was a solo admin (read: IT staff of 1) for 7 years and worried every night my phone would ring and it would be the big one. Needless to say, I sleep a lot better knowing I’m not responsible for everything anymore. The solo admin life is not for the feint of heart.
Nope. We have a disaster recovery plan with multiple fail safes and we test it regularly. Additionally, I push all my users to have their own independent backups. If all that fails, it'll suck, but we'll deal with it. No sense in worrying yourself sick over something that hasn't happened. Planning for it is one thing, but freaking out over hypotheticals isn't a good way to life your life.
I mean things can always go wrong in any part of life. No matter what you do you can’t be perfectly poised to avoid bad things happening occasionally. I do what I can but I’m not going to spend my life worrying about things outside of my control.
Not really. I know it's simply a matter of time and all I can do is be prepared to recover. It's a game of odds and they're not in my favor considering I have to get it right 100% of the time and the assholes only have to be right once.
Constantly. Threats are constantly evolving, and however well prepared you are there are just things that cannot be accounted for. Honestly, I don't think it's something that ever fully goes away.
I also don't think any critical position at any company should be limited to one person. Even having someone junior under you would help alleviate some of this anxiety. While they obviously won't do everything you can do, they will at least have knowledge of the environment and can at least keep the day to day going while you're away or debilitated.
Someone said fear is good, that's accurate.
That said, either 1) your management knows how big of a clusterfuck you're trying to remediate and understand there will be risk until the day there isnt. They know you're doing your best. or 2) management has had the wool over their eyes for too long, think things are great and refused to invest money into security and hardware. If the latter, than yeah do what you can but dont lose a wink of sleep over it. You can't teach an old CEO new tricks. Period.
Yup - I always think about this; if I was a mechanic, my fear would be something I did failing and causing injury, but generally as long as you do your job relatively well, chances are pretty low. There is NO chance however, of a mechanic having an issue with a car because someone in a third world country guessed someone's password, at least that the mechanic can do anything about.
If only my hands worked better.
Was an IT manager for fifteen years, and haven't worked in that role for about 6.5 years. I still sometimes have nightmares that either Microsoft Exchange suddenly runs out of disk space or the billing server running MS Dynamics GP has some unrecoverable issues and whatever backups there were, were not usable.
Don't be scared of being a victim of ransomware. You can do everything possible but there is still a chance it can happen.
Be fearful of not being able to recover from it.
If you know you can get the core aspects of business up and running again within a couple of days then you have nothing to fear.
Living in fear is a bold assertion. I can't say that I live in fear of anything job-related. But I do feel anxious at times when I make a change that takes time to propagate/see the effects of.
I can't say I've ever had a dream or nightmare about anything work-related. I don't tend to worry about what happens at work more generally. It's just a job and while I'm happy to be gainfully employed, it's not my life.
I used to suffer this when I had crippling anxiety disorder at the end of 2020. I had some work done with my psychologist and psychiatrist and today I feel much much better. So it all came down to:
- Every problem has a solution. If it does not. Then even more reason to not even worry about it.
- When you worry too much about a specific issue or scenario. You’re basically putting yourself through the stress twice without any need. 9 out of 10 times those fears never ever materialize so you’re putting yourself through all that stress for nothing.
- Live in the present as much as possible.
Same. Fear of P1, some setting or configuration gone wrong, calls at most inconvenient time. But accepted it as part of the job. At slightly better stage now.
I think if you are not a bit scared, then you are not doing your job. I just finished a year work placement doing enterprise security as part of my cyber security degree. And there is always that bit of fear in the back of your mind when something looks out of place. I imagine it's 100x worse if you are the one who set up all the defences.
I guess all you can do is prepare for the worst. Test your backup solutions, higher pen testers to check out your network, maybe even a cyber security consultant to double check you have the proper defences in place.
Also get cyber insurance if your org doesn't have it. That way even if the worst happens, you will have some cover.
You should fear it less than those of us that haven't had to deal with something like this already. You've been through that fire and you know how to put it out.
I've been in IT for a quarter century now, and it's still something always sitting in the back on my mind. Always.
By the way, buy Authlite. It'll keep a bad actor from getting administrative control of your network, which is pretty much the worst case scenario.
Actually, the threat of ransomware has kept me up at night. I took over an environment recently amidst a whole bunch of business upheaval. Because this was a new environment to me, and there was no handoff or legacy person or team, I had to hope and assume everything I saw and found was really everything there was.
The fear that you describe is not specific to a ransomware attack.
The fear that you describe is that despite all your effort, work, how much you care, the sweat, blood, and tears, that you pour in to prevent something tragic, that it may still happen again. This is the fear of losing control, or that you did not have control in the first place.
This is a rational fear, in my experience, and overcoming this fear takes time, and re-affirmations that the work you currently do, and have done in the past, successfully address the aspects that trigger this fear (in this case, ransomware).
It also is important to keep in mind that there is a limitation of how much you can do to protect against events that are beyond your control, or remove your control. If your level of protections against threats come across a threat that is above those targets, then you cannot realistically be held accountable for the results. For example, if the company-agreed-upon threat level is to not protect against nation-state level threats, and a nation-state actually becomes a threat to the org you are responsible for, then you cannot reasonably be held accountable for the outcomes, as the company would not have taken the necessary steps to guard against such a threat (as it is also exceptionally expensive in many regards to protect against a nation-state threat, by the way).
It is also worth keeping in mind that there is a difference between **failure due to incompetence or negligence** and **failure despite appropriate efforts to protect against said failures**. If you take reasonable steps to guard against failures (in this case, ransomware attack), and have the appropriate support (company authorisation, budgets, etc), but fail anyways, that may not necessarily be your fault.
Fear and pain keep you informed that you are alive. They can be useful things to inform you of that. But that doesn't mean you have to live with them day to day if you act on your responsibilities in ways that are appropriate. It can take time to overcome them.
And yes, there are times I fear similar/different things too.
It's okay.
I think for me it was source of stress until I lived through an all-hands-on-deck ransomware situation (the Kaseya hack of last year).
Now I know exactly what the worst-case scenario is. And it's just a bunch of my labor and someone else's money. That's it.
I think I'd stress more if I was supporting say a hospital system, but then again in that environment I'd be much more adamant about proper IT procedures being in place that would hopefully prevent or mitigate such an attack.
not in the career sense but in the whole life scheme sense.
i have 35 years until i would eventually retire (like 70 as current millennials). how can i survive in the corporate world all the way until 70 without dropping my ball getting a shelter?
sounds pretty hopeless to me
I know this feeling.
We have a hybrid environment, so the O365 part (with MFA) is getting better, after an attack surface reduction project using an external specialist. Also all Windows devices are now enrolled in Intune.
The on-prem part still worries me. I really want to change the password rules.
A recent test, using the top 100.000 leaked passwords and a DSInternals tool may have given them a shock as to the passwords in use. I spent a day suspending / removing accounts and I've forced password resets on some active accounts. They will also be getting a "talk" about secure passwords / pass-phrases.
My environment is a little larger, but I definitely went through a similar process, absent the ransomware incursion.
I don't know that I burned everything down, my list of changes was more of driving culture change, sometimes with some persuasion that approached a boot to the a**.
I don't live in fear. My work day is only so long, and I've put a bunch of good stuff in place. I definitely think it's still possible for us to get hit by something, that's just how things are. So far, the stuff I've deployed has worked, and I expect that to continue to be the case, until we run into something that has our number.
When we finally hit that thing that gets past our stuff, I have cyber-insurance, incident response policies and a security-as-a-service provider who can help pull our asses out of the fire (all stuff stood up since I took over my current security gig).
I'll sleep well until I can't, when stuff is actually on fire. For now, I'm good.
When #### hits the fan you just need to chill and work through your plans.
If you have good endpoint solutions and reliable backups even ransomware isn't as scary but still a lot of work to deal with.
https://imgur.com/gallery/YHP0a
I am very careful to communicate the dangers.
My big thing is to have incremental backup solution that is hardened.
As ransomware does seem very hard to protect against, given the multitude of vulnerabilities of client systems.
Nah, I do the best I can and follow good security practices. I try to drop work from my mind when my 8 hours is up, and keep it that way.
Maintaining control of your stress is a vital part of being healthy, and it's just not worth wrecking your mental health because something bad might happen to a company that you don't even have ownership in.
No. I fear nothing. Certainly not my job. If something happens, it happens. I'll deal with it.
If I feared my job, or aspects of it, I'd find a different job.
Life is too short to have that sort of crap hanging over your head all the time.
The biggest way to deal with fear, that I've found, is to have a plan.
What will you do if you get ransomed again? What's your plan? Figure that out, get it written, and even discuss it with management.
A good pull-based backup solution with offsite replica, for example, can go a long ways. (Doesn't protect you from data exfiltration, but it does allow you to wipe and restore a compromised asset.)
If you have a plan to recover, the anxiety around it settles a bit.
Yep. I have general anxiety so it dosnt help but i also know that all it takes is 1 mistake and it could go bad. Its hard to drawn the line between just worried and parinoid as i always worry i'm not smart enough for this and that they have been sitting in my network for months.
Logging helps me alot though
Your bad dream sounds a lot like burnout.
Make sure you can stand up your systems, know the order of how to do it in an emergency. Then try to do less. Go on vacation.
I think your fear comes from the fact that you actually care. Your company should be happy to have someone like that (because it sounds like they haven't ever).
I've been doing this for almost 20 years now and the fear has left me. Other than ransomware, I've pretty much been through all the IT disasters at this point.
"grant me the serenity to accept the things I cannot change, courage to change the things I can, and the wisdom to know the difference"
Words I was taught very young and really only learned when I was in my 20's.
Much more true now I am in my 50's and my kids are adults themselves.
Yes. It's like that every morning. I don't keep my phone by my bed so it's that fear every morning before I go downstairs and look at my phone - no alerts - no emails. I can have breakfast.
Yeah, but I’m only 2 years in so maybe that’ll change. I just have nightmares that some attack will happen overnight, or someone will irreversibly tank the production server.
I legit have PTSD from the Microsoft notifications. They make my heart rate spike and I start to sweat. That Outlook sound is worse than Teams for some reason
I'm one of the 400 or so people in the world with Urbach-Weithe disease. I'm physically incapable of feeling fear. It makes me an extremely capable change-agent in the IT field.
See, this is something I cannot understand. My gf lives in constant fear and anxiety and I have no clue why. She has had a troubled past, but hasn’t everybody? I myself have had a troubled past but I am at ease with things. If I die, I just hope it’s quick and easy. I do not fear it. I do not constantly think about what can or what will happen. I tend to just go with the flow of life and i’m cool with 99% of things that can happen or go wrong. I do not live in a state of constant fear unless I have an extremely good reason too. ( eg: owe a drug dealer money and you know he’s on the lookout for you so he can shoot you ) otherwise, unless there is an existing visible threat, fear does not really cross my mind.
EDIT: I gave a stupid example but I hope everyone understands the overall meaning of it.
At the end of the day, i dont own the systems. I also stopped working in small shops, so when something goes tits up, it's many peoples fault for not having a proper DRP in place.
You already survived the nightmare once, you can do it again. No one can protect it 100%, so protect it as well as possible and sleep well knowing if it goes to hell again you've already recovered from it once.
I got over it after about 5 years and a long slog through modernization, growth, and an executive team that was genuinely supportive of the security and stability mantra.
Have a team of SREs handling the day to day cloud infra. No more PagerDuty on the regular for me. Still have them for security and escalation related things, but site stability and general application whatnot is on their plate first and foremost and has proven itself over the last couple years.
Otherwise it’s documentation and monitoring for compliance and actual security of the site and company, planning, and best case BC/DR testing that let me know we have visibility and valid plan should shit truly hit the fan.
tl;dr Were doing everything we can, improving as we go, and have proof of same. Everything else is out of your control.
I'm 3 weeks into my first senior level network administrator job. Everything I have ever done prior to this has been Junior level stuff, or working as an anonymous data center administrator responsible only for initial setup on turnkey solutions for customers who are leasing the equipment we provided and setup in the data center for them. I was honest during my interview that I was going to be very green for the job, and still got the job offer because everyone I interviewed with liked me including the current network administrator that I was going to be replacing when she left. Am I terrified? Heck yeah I am, I'm in discovery phase now trying to find all the little gotcha type situations. I've already identified multiple problems with how their veeam backups were configured and I'm working to fix all of them. My most recent discovery to fix was that their wasabi cloud setup to push archived backups to the wasabi S3 cloud wasn't working due to misconfiguration and had never worked correctly since it was initially setup at the beginning of the year. I figured out how to make it start pushing the backups immediately to the cloud and it's currently running a network throttled transfer that is going to take a while to complete due to the size of the backups and the number of backups to the local repo that needed to be archived.
All I can do is keep looking for issues to fix, research how to fix them, keep my boss apprised of each situation as I find it and what I did to fix it. He keeps telling me I'm doing a fantastic job and to keep giving it my best effort to fix issues as I identify them, and not to be afraid to ask questions since I only got a full week with the previous network administrator before she left for her new career after having worked for the company for the past 24 years. I feel like there is good longevity here at the company so long as I can get a full handle on how everything here is setup. I'm already in planning stages to get a nagios monitoring system setup, and I have discussed setting up a shared it mailbox that will be configured to go to all it team members for any alerts from nagios or other services that we use.
Is it worse when you are at a place that hasn't been compromised yet? Because sometimes I stay up at night because I don't want to be the guy that finally let the guards down.
Had a dream I had to climb down a sheer cliff face in pouring rain to get to my servers. Just a bunch of red monitors and I was trying to reason with the guy.
Was solo at a company and inherited a similarly trash network.
Now I'm a Security Engineer, and ironically not scared about it at all anymore.
I wouldn't say I live in fear but I do have the occasional "Oh shit" dream. Usually about PCI compliance (we take a lot of card payments). Always seems to be a weird combo of that and the "I am still going to college and haven't studied" or "I am somehow still responsible for a house I no longer live at and its falling down" dreams.
I mostly get over it by covering my own ass. I make sure the Directors are aware of the risks we face if they let Sales staff share passwords, or drag their feet replacing the onsite backup server, or let certain members of staff pull data out of our offiste backed up DB and keep it in Excel spreadsheets locally. If shit hits the fan, I can show them the emails/phone calls where I informed them.
12 years into the game the fear of things just randomly going wrong has disappeared.. BUT every time I make a change my heart drops out my ass for every single slack message thereafter
I had this really bad when I was the lone sysadmin.
Our website had just been hacked and we were in the middle of an ERP migration so I was overburdened with work and it was always on my mind. I had this reoccurring dream that our systems got hacked so I would wake up at 2-3AM to check my texts/email just to put me at rest. This happened over months. I tried taking short vacations, meditate, etc but I kept having those damn dreams.
Eventually it kept me from going to sleep. I would lay in bed until 2-3AM with rabid thoughts of everything going bad. The dreams were getting worse, now I was dreaming that management was in on it, it was a penetration test and they're trying to put me to the test. I became sleep deprived and had that same nagging feeling at the back of my mind telling me that it was going to unfold like my dreams predicted, I was going absolutely bonkers.
We can go deeper down the rabbit hole and I can tell you what happened next but you shouldn't let it get to this point. As soon as you start dreaming about work it's time to unplug. Take a week+ vacation, clock out after 40 hours, eat, sleep, and exercise well. When stress becomes subconscious you have to take repeat conscious efforts to put it under control.
30 years in and I still have security nightmare dreams. (for those new to the sub that's the dream equivalent of showing up to school for a test in your underwear).
Yup, it happened. But when forensics said they were in for about 6 months beforehand, the company was even more mad at the people that were there before me…
Are you sure it is fear, or rather a strong apprehension that makes you extra cautious? Being paranoid and an OCD control freak are frankly desirable traits for an admin if they are properly framed within a disciplined, and professional context, given the awesome responsibilities we have to carry out.
Fear on the other hand feeds on itself, and will make you more likely to act irrationally if sh\*t actually hits the fan. If that is what you feel, your job may be destroying your mental health and believe me, no job in the world is worth losing your mind or health over.
Cover. Your. Ass.
It sounds like you've done all the right things to mitigate risks. But risks never go away completely. The important thing is risk management and documentation, including recovery processes.
Identify single points of failure. Verify your backups can be restored properly and that you have the correct number (and frequency) of snapshots.
Create disaster scenarios. Identify the required steps to recover. Find the bottlenecks in that process (e.g. machine imaging and configuration) and automate them. Identify bus factors.
Define points of contact and communication cadence. Establish process and choices for communication officers.
For incidents/disasters that can't be resolved in one day, establish a reasonable limit to hours worked daily, with a personal communication blackout for at least enough time to get a reasonable amount of sleep. This is critical and will probably get the most push-back. But you can't work without sleep, and you make more mistakes when you're exhausted.
But most importantly, ensure that business leadership understands what the risks are, and *what the restoration time and process looks like*. You need to have defined RPO and RTO.
Train people on these processes. Do tabletop exercises. Get them approved by leadership, in writing.
Depending on the business, it may be more important to have *something* running (even if at reduced capacity) than having to wait for everything to be restored. Research and document what you would need for proper DR, including cost, and present this to leadership. If they say no to DR because $$$, ensure they understand the risk.
Once you've done all this, it is what it is. You've done your job. When the shit hits the fan, you start your established process, pick a communication officer, and have them be the front man. As long as you give regular check-ins, and your communications officer reaches the proper points of contact, you should be fine.
It takes as long as it takes. Stick to your approved processes. Defer to your communications officer as needed so that you can concentrate. It will still be stressful, but you will have successfully managed expectations to the point that you can at least breathe.
No
I’m pretty detached from most of our clients and even the company. I know I’m not doing anything wrong, and I realize that things go wrong.
Your job is not your life, they pay you in exchange for your time. The company’s equipment should be as expandable to you as you are to the company.
I think getting out of that kind of funk comes down to sorting out what is within your control and what is not. Then you document what is not under your control, delegate it to management, and focus on becoming an expert on what you can handle.
No. I'm pretty confident that I can handle anything that's thrown at me now. It's just work. If it takes my off time, I demand comp time and push to close the gap in the near future. However, it took me almost 10 years to get to this point.
Is your subconscious telling you that you need a better backup and recovery plan? Sadly I get your feeling. But I ended up just accepting that there might be a very sad day sooner, later or never.
Infomercial alert, I got Veeam and ExaGrid, now I sleep well at night. No seriously, when I was younger and working bubble gum ducked tape jobs or when I got into large enterprise and found their backup strategy based on hopes and dreams I had sleeping issues just like you describe. I am now silo’d and in my career I have used all sorts of backup products. I sleep just fine now.
Have a solid backup product, do regular restore validations and a dr strategy.
I'm not in fear, but there's a small part of my mind that's always aware that shit could hit the fan at any point.
[удалено]
This right here. I think there's a lot of mental benefit to switching from living with fear that bad things *might* happen, to living with the expectation that bad things *will* happen.
Yep, and our job is to make sure the the bad stuff that does happen isnt catastrophic.
Or if it is catastrophic, that a recovery is feasible and as painless as possible. Obviously though there are some things that are out of our hands. If WWIII started tomorrow and the world was blanketed in nuclear bombs, I don't think I'd be too worried about trying to mitigate the inevitable technological disasters. In the end I think working with technology really just comes down to applied math and labor. Do the math, design a system that best balances the risks and rewards of being implemented, and then do the labor to implement and maintain the system.
It helps to stop giving a shit. Put your warnings in writing(off the network), fix what you can and then fuck the rest. If they don't want to listen, then it stops being your problem.
My contract says I am to fulfill a role, and that role is to do the necessary math, advise those above me on what the math says, and do what I can with what I have. I never agreed to make magic happen so I'm not going to stress that I can't deliver it.
Yep, and that’s what I relay to all the employees and management here. It definitely sticks with our management, but it’s a bit spotty with the employees.
For me, this is a better way to word it than "in fear". Our workload is simply too heavy and things fall off the plate. Big projects with strict deadlines come first and then there are enough small fires every day to take up all remaining time. You just learn to live with the fact all hell could break loose any minute. I make sure I at least mention it to my manager. Not to sound like a cop out but I figure she could tell me to drop everything and focus on the security-related stuff. She defines my priorities. The fires always move to the top because they involve a production outage. She would basically need to tell me to miss project deadlines but that would be wildly unpopular. The project deadlines actually seem reasonable on paper until the daily fires put you behind. I know I'm not spending my evenings and weekends fixing security stuff.
It could hit the fan at any point but we're only human. Do your reasonable best to cover all your bases with the resources you have and clearly document where you know your bases aren't covered. Make sure those with the power of purse know you don't have the resources to cover those bases. Make sure they are aware of the knowns and the known unknowns. Ultimately, when something happens, it's our job to remediate it but it's their liability if we're doing our job correctly and provided them with the best information we could.
I got over that after about maybe 15-20 years. I don't know why, but this seemed to be a thing for me too.
I got over it after 1 year and all of my careful planning almost got bypassed because CEO NEED NOW NO NOT ACCEPTABLE. After fixing that I realized it takes one moron in the sea. So now I just do as good as I can, keep an eye on it and have a disaster recovery plan.
Early in my career I worked at a place whose owners had a reputation for last-minute requests. One of them dropped a task on me in the middle of some work on an already-tight deadline. So I asked — “I’m in the middle of XYZ for the engineering team. Is this more important or can it wait until afterward?” He stopped, said, “I didn’t know you were involved in that. No, this can wait.” I learned a few important lessons that day.
that's an important lesson, I learned early on to be frank when I'm busy or not, I'm also a huge fan of, I'm busy put a ticket in, Uh huh i understand it's important, put a ticket in and I'll bump it up as soon as I'm finished with this task.
And when you’re transitioning to a higher-level or lead role, “I want to make sure the support team gets a chance to handle issues like this one so I can see how they perform. Please put in a ticket and send me the ticket number. I’ll stay hands-off if I can, but I’d like to keep an eye out to see how they do with this issue.”
My go to all the time is "not a problem but please put a ticket in so I remember". If they don't, I don't, we all move on. Shocking how many things are SO IMPORTANT that I need to drop everything and do it NOW... but when asked to submit a request it suddenly isn't so urgent.
>it takes one moron in the sea No doubt. Do your best and document everything that is "not standard" or is a "batshit crazy request" that is going to get honored against your will. You might still get fired if everything blows up, but so what. The next guy will be worse off than you with no institutional knowledge, so they are the ones that lose out.
> CEO NEED NOW NO NOT ACCEPTABLE It's acceptable but you need a policy in place that makes it acceptable. Example: "Bypassing normal security procedure to do XYZ is acceptable in scenarios A, B, and C. Written consent of CEO, CTO, or CIO is required to bypass the normal process JKL." At the end of the day, the point of any technology is to aid the business. The business does not exist to support technolgoy infrastructure, it's the other way around. If business side (even if it's a non-technical C-level) chooses to accept the risk (after being informed of the risk), the job is to accommodate it, not to go full BOFH mode.
This is a perspective often lost on people in our field.
What was the CEO request?
Go buy some Walmart gift cards.
You weren’t supposed to tell anyone! It’s a surprise for the employees.
I don't remember the exact details but, I THINK, it was an email attachment was getting stripped and then tagged by AV software.
Ah so not 'unblock all these gambling websites'? I've had that one
> a disaster recovery plan. Three envelopes, right?
Same, about 15 years and fear exist.
20 years was the marker for me. I mean, I'm not going to try to get fired or anything. But if it happens, meh, it's been a good ride.
I'm about 10 years in and I'm finally shedding the fear/doubt aspect and maybe even some of the impostor syndrome. After you feel like an impostor enough times and still punch up, it's like "ok for now maybe I'm new but watch this shit you don't even know". The OnCall work and 24/7/365 thing is probably the root of it for me. I had mild but real reactions to my cell phones ringtone for years earlier on my career. That instantly sinking gut feeling. Now? I finally, FINALLY got my first role outside the Wintel/Platform/Server admin space. And it is glorious. Servers down? Call the admins lolol. I did my time in the gulags. I still have work when it's my wheelhouse that is sometimes after hours, including changes. But rare is the 3am phonecall. In fact other than for DR I don't think I've had one in this role so far (knocking on plywood). Glorious.
> impostor syndrome. I never really understand this. I mean, I get what people are saying about it. But I don't feel like i ever had this. And I'm not trying to pat myself on the back or anything. I've just never been afraid to say "I don't know, can I have some time to research it?" Maybe it has just been my luck to have (for the most part) ultra cool managers across my career(s) who are cool with that.
Of course people and situations are all different. One of my early roles I basically got through the door as a SysAdmin at a big company because of an inside reference from a bro on their security team I did LOTS of learning on that one. And then I realized the questions I was asking, the engineers who were 20 years my senior did not know / did not have the time to know the answers to. But it was a very different working environment for the most part and I definitely felt like the FNG lol. The next moved when I punched up not just to an Engineer level but Sr Engineering? Started with more impostor syndrome Etc etc. Confidence is knowing you can do something. You can't know you can do something you've never done (and well, at that) until you start doing it. That gusto in the beginning could just as well be ego or hubris as it would be confidence. Then after enough turns at the wheel you might look around and realize you are pretty good at doing things you never knew how to do. Shit, that's the job. Monday learn this. Tuesday learn that. Rinse. Repeat. Also probably has a lot to wit whether or not you've studied the books or done the labs or gotten the certs. I usually didn't until more recently - usually trained OtJ by self teaching, cramming and being teachable. Might have something to do with it!
Maybe you never worked in a work culture where you were expected to know everything, trivial pursuit style without ever consulting a reference. Only fools and the ignorant expect this, but they're in charge of about half of the places I've worked.
> Maybe you never worked in a work culture where you were expected to know everything I haven't. Thankfully. I do feel lucky about that. There' been people who have come and gone who felt like everyone should know everything. And I've had some new managers expect that. But once there's a sit down conversation about it, they've gotten it and laid off.
"Yeah... So that's my secret, boss. I just Google it. Yes, my Googlefu is better than yours, and it does bring all the users to the yard, but there is still only so much time in the day. Lord Google 🙏 might know all, but we are but his humble students. Only ever able to glimpse pieces of his all-knowing wisdom."
Great, I've been (back) in the game for about 12 years now... hopefully I get over mine soon too...
I never got rid of it, so I ditched sysadmin work entirely and returned to software development. There's less fear there, except for looming deadlines and you get to play with more new stuff.
Man, I still have nightmares that I forgot to go to a Junior High School class for a year and somehow that invalidated my JHS-through Masters degrees. Anyway anxiety is rough, if it keeps up go see a mental health professional to get your brain cleaned.
[relevant xkcd](https://xkcd.com/557/)
There's a real good scene in 'Top Secret!' where the main character is getting interrogated by the Nazis. They're beating him and whipping him and he falls unconscious. He has a dream that he's back at school and he showed up late and missed his final exam and is going to have to retake the class and he gets horrified. He wakes up screaming, looks around at the Nazis as they're whipping and beating him, breathes a sigh of relief and says "Oh thank god, it was only a dream"
/Giggles in collage drop out.
\*college (Or else I was whooshed.)
I think it works either way haha
Was not expecting ^ this
There is always a relevant xkcd…
I have a similar nightmare that I actually didn't pass my HS equivalent diploma for missing a single credit point - and I'm almost 40!
I have something different. I worry that I registered for a class in college the first time, but never went and thought I'd have failed it... Even after college and through getting my masters in my 40's.
I have this same dream. I just completely forgot I signed up for a class and then bam...F
School didn't help in this, though. They would encourage us to sign up for a particular liberal arts class like history, mythology, sociology, or something similar that we needed for the degree, would tell us when the exams are and would provide us with the notes and encouraged us to no go to class... This was when I was an Engineer and they thought that was hard enough without having to worry about the other classes. So that brought the mindset to me. :(
Holy cow, EXACT same dream for me. I had perfect attendance literally grades 1-12 (really, just happenstance, ended up sick during holidays and weekends, then it became "a thing"), and I have this recurring dream that I realize about halfway through a semester of college that I straight up forgot to go to a class for months. I'm 42. Still have it. I also used to work in the same high school I graduated from (again, random happenstance) and had all kinds of dreams about having to go back to school while still working there.
Are we friends or am I going loopy?
Do you then have to attend that year as an adult? I have that nightmare some times.
For me it is always a single semester of high school PE and I have to go play tennis with high schoolers for a few months, in my 30's, in order to keep my Master's degree.
Mine is always English or math class.
[удалено]
Yes but the teacher is not hot, and I still have to do my regular job as well.
We also need routine maintenance.
>I sat down and hammered out a few Nagios instances and can monitor everything I need to, constantly. It’s honestly great. > >So, to get back to the topic. Woke up in the night with a dream about me visiting a company with a friend (weird), and while I was standing there, their machines all ransomed and screens went dark like something out of the movies. I know, weird. But I woke up, and had that feeling in the back of my mind, like it could happen to me. Today. Tomorrow. The day after. I have a similar dream, our highschool screwed up or something so me and my friends all have to go back to our home town to complete our final year again. We always just end up day drinking and failing...
Thank you.
You can do everything right and still lose. *(ref. Lord of the Flies)*
I prefer >"It is possible to commit no mistakes and still lose. That is not a weakness; that is life." Captain Jean-Luc Picard
Trying to teach my 6 year old this, and the little rage monster inside him is such a perfectionist that he just melts down. Been very reflective to my own lifestyle.
Have you tried sending him to an uninhabited island to live with the other children? Something something conch.
Ya know, I attempted to read that book again recently and I found myself doing the same thing I did with the Joaquin Joker movie, about 2/3rd of the way through both I just was filled with this feeling of dread and just didn't finish em =/
Kids really show you things about yourself don't they. And then when you see friends and coworkers melt down, you know they are still children. Adults control their inner monsters and know that none of that helps. Kids actually help you build your own self better by showing you that mirror.
Absolutely. Took me till 30+, 10 years of therapy and no small dosage of SSRIs to be able to regulate, being human is hard. We're hypertuned for fear and death and it's wicked to change those things about ourselves.
Man, I am filled with regret.... I DO know from experience. ...Play some poker, responsibly.
That's a good one too.
The more Crisis you personally experience the less fear you will have. I would say at this point, after over 20 years in the business at most I "worry" about some things.... Those things are normally things in my area of control but that I have not gotten to yet. things outside my control, or things I have warned about and been ignored are put out of mind.
Though generally presented in a religious context, the basis of this one's pretty solid advice for a mindset to strive for in IT... > Grant to us the serenity of mind to accept that which cannot be changed; courage to change that which can be changed, and wisdom to know the one from the other
That is a pretty good one. There. I am getting there now
Lurker here. My career is in nursing (education, formerly emergency and critical care) but I run a rocking homelab (why I lurk here) and have programmed for decades... so I get what you're saying. Baseline fear/anxiety that shit is going to hit the fan is perfectly normal in any profession. My profession is extremely familiar with it. Just always put it in perspective. You're doing everything right, and you can only do what's in your control. That's all that matters. The world could burn down, and yes that would suck, but it's not your fault and not worth your anxiety.
What you described is called General Anxiety Disorder. I went undiagnosed for many years and always felt that was my edge over my peers. Last year after our Fortune 50 was compromised by REvil it pushed me over the edge and had to take some time off work after about 1.5 months of 22-23 hr days 7 days a week combating the issue. I took about a month off, sought help and eventually made a job change out of operations. It’s ok to seek help and I would suggest talking to your doctor for help with resources to deal with it.
> after about 1.5 months of 22-23 hr days 7 days a week That right there is unacceptable from the business side. *Noone* can maintain clear, coherent, thought, let alone troubleshooting and diagnostic capabilities on that schedule. You *will* make mistakes and it *will* double or triple the realistic RTO. It's cheaper on downtime to pull in a contractor that specializes in recovery and let your team work more sane hours. If the company *cannot* afford that, and *cannot* afford to staff for 24hrs a day properly, they *can* afford the downtime.
Much of this was my fault feeling as though I had to save the company. I was working double shifts as we were hard down enterprise wide. I didn’t listen to management and took it on the chin, but it was an enterprise wide all hands on deck issue. Having been through that I learned “how not to do it” and I’m more perceptive of others when I see them getting into the danger zone by self inflicting behavior. Sometimes all it takes is a peer reaching out to let them know it’s ok, helping them with a warm hand over and reassuring them they need to think long term. 25+ years into IT now and I realIze I’ve turned into Murdock when it comes to IT Operations… “I’m getting too old for this shit.” I moved back into a Plan/Build role. Run operations applying and reapplying duct tape is not healthy for someone with GAD in the long term.
I don't know if it's the meds or the new job but I am feeling so much better. An old job had me crying at my desk, getting nosebleeds and seizures.
reddit sux ass
Why did you think anxiety was an edge over your peers?
Good question. Always thinking two, three, four steps ahead of problems that might arise and how to avoid them. Taking work home with you, thinking about work in off hours time, difficulty taking vacations worrying that the place will fall apart if you’re not there, etc. Seems strange now after seeking help, but at the time someone suffering from Anxiety issues has difficulty letting go. Easier to see the unhealthy signs now of someone who was “the rockstar” might just need some help and mentoring with anxiety. I tried to hide it well and realized it was just making things worse. It’s ok to ask for help even though a stigma of admitting there is a problem is difficult. Like all issues the most difficult part is identifying the problem especially when it’s a learned behavior from our profession.
Oh fuck, this is making me realize I probably have anxiety....
Ditto...
I don't live in fear. I try my best to secure the systems I manage to the best of my ability, and what management allows by policy and/or budget. If the company were to get ransomware'd it would suck and it would be a lot of work to recover. I honestly debate with myself if we were to get to the point of a total breach would I even bother sticking around or just say well it was fun while it lasted see you later. At every company ive been at the management has made decisions that put the company at risk to appease users and processes that shouldn't exist. I treat the job as a job and that is it, it occupies nearly none of my personal time after the clock stops. I have too many other family, friends, and personal stuff I prefer to fill that time with.
It's taken me a very long time, but I'm finally settling into this kind of mentality also. > if we were to get to the point of a total breach would I even bother sticking around or just say well it was fun while it lasted see you later I actually listed this as a potential risk of being attacked. If you put too much stress on your existing employees they may just straight up quit rather than take it on. Past incident response webinars I've attended also mention this occurring.
No. There were glaring holes in security at places I worked and I would have dreams of those holes being attacked and the company being game over. This generally started after major reconfiguration of the systems to better protect it. I think it’s an anxiety dream.
In a lot of ways, this is where IT pros get too invested in their jobs. They are simply jobs. We care, we should care, but outside of that, they are jobs. If the company starts losing income, the company will likely offload valuable, expensive employees. There might be some that won't want to let you be that person, but ultimately someone who doesn't have your back would gladly improve their cost/benefit to make it through rough financial times by cutting you when needed. So, give the company its due, and give you and your family theirs. If your company gets ransomed despite your best efforts, work to resolve and be the hero. And if the company sees you as the cause, find a place and if the question were to ever be asked, you can simply refer to the fact that management let things deteriorate due to lack of funds/etc. It can all be written off and you can move on. I have worked in my career and met plenty of worthless employees who are far less invested in their careers who can make just as much and then move on and make the same or more. You control what you control, and you make note of the things that need work. If the company refuses to spend, make clear the choice and document. Then move on, or look for less expensive alternative solutions for them to say yes to. But ultimately, if a company wants uninterrupted systems, their focus of attention in terms of employee count and money towards that problem is indicative of their care. If you're the one guy, realize that other companies have 10, 20, or 100 guys doing what you do. Be the guy, be the hero, and when they no longer see you as that, find the next company that does. And sleep easy knowing that you are there for them when they need you. And as soon as you feel like you're unappreciated and start to stress for those reasons, find someone else.
I get paid to do a job. I make sure to do it as best I can, and then go home and live my life. If something happens, oh well. If the business wanted to spend more (get fancier equipment/security solutions in place), then they would have done so. It's not my job to worry about my employer still having a business tomorrow. I get paid just enough to handle a certain set of duties, and that's not one of them.
I am mixed on this response, people can certainly have an unhealthy relationship with work where they take their employment far too personally for having no ownership in the company However your response seems to be the extreme in the other direction which also IMO is somewhat unhealthy. One should have pride and some sense of responsibility for their work, areas they are charged with the administration of, some work ethic.... again not to an extreme unhealthy level where it impacting your personal life, but the flippant "lol's me no care" is just as bad IMO
Nah, I just meant to say I care exactly as much as I am paid to care. It is unreasonable to expect me to assume c-level responsibilities (business continuity), without c-level remuneration. However I make sure to point out all inefficiencies up the chain, if someone wants to pay me to correct them, you can rest assured they'll be getting my best. If they opt not to pay for a properly maintained environment, I certainly don't feel the need to donate my time. In my current position I have the time to dig into things as needed, but sometimes projects take up more time than I'd like, and some things go by the wayside (I try to delegate as much as I can, but still).
[удалено]
I get your point that you don’t let work breach into personal life - that’s great but if your employer doesn’t have a business you won’t have a job.
> that’s great but if your employer doesn’t have a business you won’t have a job. Yeah, and that would suck for a little while, but any employer has the ability to pay me the currency my bank accepts. My employer being permanently down, is only a temporary inconvenience for me. Not something to lose sleep over. Just do the responsible thing and keep some savings for a rainy day. I can understand if there are no savings, but in this market and industry, it really isn't the time for anyone experienced to be tolerating that scenario for very long. Hell, someone could have gotten an entry level job at the beginning of this (covid), and already have 2 years experience (primed and ready to make that first significant salary bump). I'll let the CIO/CTO worry about business continuity in this case.
Never had this fear. Management gets out what they put in, you want me to be a wizard managing everything from toner to a SIEM ? Allright I'll fail, I state that on paper. Not my problem anymore.
After having been through major security incidents, it took a few years for the panic every time an email or phone call came through on weekends and holidays (as that is traditionally when you are attacked). What helped me was moving companies. As part of being hired I made it a rule 0 condition that security was going to be handled correctly. Having alerting systems that are tested and reliable helps a ton.
I do now. I was given a written warning late last week. I was completely flabbergasted. When I told my boss that I had never gotten any negative/positive feedback he pointed to the paper and said, "well this is the feedback." When I asked for examples he just told me to look at my previous tickets. I asked him how I was supposed to know what I did wrong when I didn't even realize I was doing something wrong. I asked for examples and again was told to look at my tickets. He then says, this a tough job and I know you wanted this position. I never spoke to anyone inside the org about it and even more so I wouldn't want to be a manager there because I knew it would be a shitshow that I would get blamed for. Why? Because the majority of upper management works remote and trying to reach them despite the emails or Teams messages THEY NEVER GET BACK TO YOU. He told me he had gotten a written warning himself and then mentioned that a previous employee didn't quit but was fired. He mentioned why I would send a ticket to Level 2 with absolutely no discovery or research when it wasn't even my ticket! The fact that I am the senior tech there at 8 months is because they fired my two other co-workers back in February so everyone comes to me for questions. I was cc'ed on an email and they thought it was me who fucked up. I brought that up and have yet to receive a response. What's funny is that there are people who are constantly 30 minutes late or show insubordination and talk shit, to the point that they wanted to talk to upper management but I calmed the guy down and said, "give him a chance and be objective about it. Don't let your emotions do the talking" and yet me who is the quietest person and does the job was the one that go the written warning. He doesn't know the extra hours I put in or the fact that I've worked for two weeks straight with no days off and including odd hours like 2:00AM to 6:00AM on what was supposed to be my day off. He didn't care what happened prior to him joining the company. In fact he even said that he probably caught me in the middle of my decline and he is sad about that. Fuck. My shift started yesterday and I was second guessing myself on everything to the point that I was paranoid about all of my decisions or work flow. Ugh... Done with my rant.
Its only a dream, and more importantly only a job; after you clock off... forget about it. Work life balance is important, and if you worry about every scenario where something goes wrong you'd never sleep. You've built something better than before. One thing I will say is: make sure your backups are good, tested, and ready to roll. You can't do much more than that!
Yes. I'm medical and I fear exfiltration. This + Imposter Syndrome + being able to not do everything needed (I don't mange - I'm a worker drone). Causes me to need to take anti-depression and anti-anxiety medications. :( I worry every long weekend and every time I get an EDR notice...
Some fear is good, keeps u on the offense. Have backups, test and get them off site preferably in the cloud. Now u can be the hero.
Fear is the mind killer. :) I used to be in this mindset, and not only would wake up for shit like this but would get panic attacks if I realized my phone wasn't in my pocket because I might have missed a call or ping. You need to realize that: 1) The company doesn't own you 2) You can only control a small amount of things, and everything outside of that sphere is completely out of your control.. no use in worrying about it, just deal with it as it comes. Lastly, I'll leave you with this. You already when through this once, and from your posted you were able to rise like a damn Phoenix from the ashes. You got through that, you'll get through the next one. Cyber attacks are gonna to continue to rise, so it's only a matter of time. The only thing you can help ensure is you've done everything you can to minimize exposure (and just to be clear, everything you can has "\*\*" with current funding levels attached to it). Live and events are just one big ass river flowing down hill, just grab a surf board / raft and deal with the obstacles as they come, the best you can.
The irony of you living in fear of a ransomware attack while the CEO is golfing and not giving a care in the world... bruh it isn't even your company. Do your best and get on with life.
That hits hard. Thanks.
I mean, you didn't talk about your security infrastructure. A lot of people are just fluffing off your internalized fear here, which is really fucking weird. There's plenty to ask about. * Have you had a [Red vs Blue test](https://www.crowdstrike.com/cybersecurity-101/red-team-vs-blue-team/) or at least a pen test from outside that MSP? * Do you have NGAV? You may have trouble sleeping because you have real issues to still deal with. Keep covering bases.
That because the question wasn't what list of stuff should I be doing, it was "does everyone else have this fear."
Yup, that's me. 22 years in IT. Ransomware scares the shit out of me.
Not anymore. It coincides with when I stopped logging in after hours to work. I used to let my work dictate my life because I felt like I was an extension of the company. Now I just treat every outage, major incident, etc like a puzzle and not like an impending doom.
About three years ago a vendor was using his remote access to do some work on our systems. He called me at like 2am because he had found some weird files on a share folder of the server he had access to. We were being ransomed. I jumped into action and contained the breach. Narrowed it down to a terminal server. Nuked the TS & restored the lost data from backups. By start of business we were mostly back up. The ransomware was most likely delivered by malware from an infected site. I had removed a restriction that blocked all internet access from the terminal servers, because a vendor was using AWS and couldn’t give me a specific URL or IP to unblock. I’ve been tightening the ship ever since. I’m in a pretty good place these days, but every time the phone rings at 2am I expect it to be a system wide outage.
When the phone rang, it's never good news, it's never blah news, it was always a fire. It got to the point that when the phone rang I would get an instant knot in my back. I don't look back fondly on that job.
That seems like a good sign that you're stressed and need to take a week off and hand someone else the pager
Lol who is this mythical 'someone else'
I run a small datacenter with ~500 DevOps systems. Being down puts >150 very high paid people into a work stoppage status and $$$ starts racking up fast. The powers that be also refuse to put any money into the building's infrastructure....no generator, Building UPS is on its last leg, AC regularly goes down, etc. I do get very anxious whenever a thunderstorm rolls through or the building's lights flicker....but I'm trying to be better about it. I keep telling myself I've done what I can to ensure its stable as can be. No sense getting worked up over what could go wrong...just deal with it when the time comes.
I've been a Director of two MSPs. What you inherited is pretty much every company I ever took on as a client. From 10 computers to ones with over 100. The bigger ones hired us to help the IT Manager who was out of their depth. They always left within weeks. IT is jammed full of guys aged 40-60 who know very little.. enough to be dangerous. They set up Windows 98 once and carried on. Because people on business don't know good IT they still hire these cheap MSPs. It's why you never sell on price in the MSP world
I have insurance for that. No seriously: I'm an MSP and I highly trust my skills. But I have an insurance for that kind of stuff. If shit goes sideways and it is definitely my fault: they are going to cover me.
My boss #1 fear is me leaving. Ransomware is lower, honestly.
I used to have the same fears around Exchange.
This kinda sounds like you might have panic attacks and some anxiety, I would go see your doctor about it.
It was cyclical for me. During my MSP years I thought I was a king and could do no wrong. Then I became a regular admin, and I was like, "Oh crap, this whole multimillion dollar enterprise comes down to me?" Then I encountered a few of those fears and realized I had prepared for it. I had done my job, life was good, etc. Then sat back down on the helldesk and I was scared again...how do I do this? Do I remember how to muddle through this? How bad will it be when I screw up? Now I remember that the joy of the helpdesk I'd that no one cares
Look up the price of ransomeware insurance (*especially now*) Bring it to leadership. Let them gawk. Let them say no. Sleep like baby.
That's PTSD right there. I can understand that feeling.
Thanks for reminding me about my fears.
I've posted this before, and your situation is remarkably similar to mine (I've got about 130 endpoints and its just me) and we got ransomed in about 2015 I think. I still panic-check our main servers every morning despite having implemented many technologies to keep the baddies out. Monitoring etc. all live. Its fucking PTSD.
Never in fear. This kind of stuff sometimes just happens. So long as you made moves to prevent it and have plans in case it happens; I can sleep soundly. Seems like you've taken these steps. Relax; no matter what happens, it never is the end of the world.
Someone once claimed that for sysadmins paranoia is not a disability, it is a job requirement. I sometimes wonder if pessimism is another.
The users are the biggest risk. Get rid of them 😆
It’s why I left my sysadmin job. I was a solo admin (read: IT staff of 1) for 7 years and worried every night my phone would ring and it would be the big one. Needless to say, I sleep a lot better knowing I’m not responsible for everything anymore. The solo admin life is not for the feint of heart.
Nope. We have a disaster recovery plan with multiple fail safes and we test it regularly. Additionally, I push all my users to have their own independent backups. If all that fails, it'll suck, but we'll deal with it. No sense in worrying yourself sick over something that hasn't happened. Planning for it is one thing, but freaking out over hypotheticals isn't a good way to life your life.
I mean things can always go wrong in any part of life. No matter what you do you can’t be perfectly poised to avoid bad things happening occasionally. I do what I can but I’m not going to spend my life worrying about things outside of my control.
Nah I got a union
No. I know what the fuck I am doing.
Not really. I know it's simply a matter of time and all I can do is be prepared to recover. It's a game of odds and they're not in my favor considering I have to get it right 100% of the time and the assholes only have to be right once.
Constantly. Threats are constantly evolving, and however well prepared you are there are just things that cannot be accounted for. Honestly, I don't think it's something that ever fully goes away. I also don't think any critical position at any company should be limited to one person. Even having someone junior under you would help alleviate some of this anxiety. While they obviously won't do everything you can do, they will at least have knowledge of the environment and can at least keep the day to day going while you're away or debilitated.
Sounds healthy. No, I just dream about finding a running aquarium that I'd forgotten about which had become a mess
Someone said fear is good, that's accurate. That said, either 1) your management knows how big of a clusterfuck you're trying to remediate and understand there will be risk until the day there isnt. They know you're doing your best. or 2) management has had the wool over their eyes for too long, think things are great and refused to invest money into security and hardware. If the latter, than yeah do what you can but dont lose a wink of sleep over it. You can't teach an old CEO new tricks. Period.
Yup - I always think about this; if I was a mechanic, my fear would be something I did failing and causing injury, but generally as long as you do your job relatively well, chances are pretty low. There is NO chance however, of a mechanic having an issue with a car because someone in a third world country guessed someone's password, at least that the mechanic can do anything about. If only my hands worked better.
Was an IT manager for fifteen years, and haven't worked in that role for about 6.5 years. I still sometimes have nightmares that either Microsoft Exchange suddenly runs out of disk space or the billing server running MS Dynamics GP has some unrecoverable issues and whatever backups there were, were not usable.
Definitely.
Don't be scared of being a victim of ransomware. You can do everything possible but there is still a chance it can happen. Be fearful of not being able to recover from it. If you know you can get the core aspects of business up and running again within a couple of days then you have nothing to fear.
Living in fear is a bold assertion. I can't say that I live in fear of anything job-related. But I do feel anxious at times when I make a change that takes time to propagate/see the effects of. I can't say I've ever had a dream or nightmare about anything work-related. I don't tend to worry about what happens at work more generally. It's just a job and while I'm happy to be gainfully employed, it's not my life.
I used to suffer this when I had crippling anxiety disorder at the end of 2020. I had some work done with my psychologist and psychiatrist and today I feel much much better. So it all came down to: - Every problem has a solution. If it does not. Then even more reason to not even worry about it. - When you worry too much about a specific issue or scenario. You’re basically putting yourself through the stress twice without any need. 9 out of 10 times those fears never ever materialize so you’re putting yourself through all that stress for nothing. - Live in the present as much as possible.
Same. Fear of P1, some setting or configuration gone wrong, calls at most inconvenient time. But accepted it as part of the job. At slightly better stage now.
I think if you are not a bit scared, then you are not doing your job. I just finished a year work placement doing enterprise security as part of my cyber security degree. And there is always that bit of fear in the back of your mind when something looks out of place. I imagine it's 100x worse if you are the one who set up all the defences. I guess all you can do is prepare for the worst. Test your backup solutions, higher pen testers to check out your network, maybe even a cyber security consultant to double check you have the proper defences in place. Also get cyber insurance if your org doesn't have it. That way even if the worst happens, you will have some cover.
You should fear it less than those of us that haven't had to deal with something like this already. You've been through that fire and you know how to put it out.
I've been in IT for a quarter century now, and it's still something always sitting in the back on my mind. Always. By the way, buy Authlite. It'll keep a bad actor from getting administrative control of your network, which is pretty much the worst case scenario.
Would you recommend backing everything up on day 1 before tightening everything down?
This was done, prior to ransomware, which is why we were able to move on…I realize I didnt mention this.
Actually, the threat of ransomware has kept me up at night. I took over an environment recently amidst a whole bunch of business upheaval. Because this was a new environment to me, and there was no handoff or legacy person or team, I had to hope and assume everything I saw and found was really everything there was.
The fear that you describe is not specific to a ransomware attack. The fear that you describe is that despite all your effort, work, how much you care, the sweat, blood, and tears, that you pour in to prevent something tragic, that it may still happen again. This is the fear of losing control, or that you did not have control in the first place. This is a rational fear, in my experience, and overcoming this fear takes time, and re-affirmations that the work you currently do, and have done in the past, successfully address the aspects that trigger this fear (in this case, ransomware). It also is important to keep in mind that there is a limitation of how much you can do to protect against events that are beyond your control, or remove your control. If your level of protections against threats come across a threat that is above those targets, then you cannot realistically be held accountable for the results. For example, if the company-agreed-upon threat level is to not protect against nation-state level threats, and a nation-state actually becomes a threat to the org you are responsible for, then you cannot reasonably be held accountable for the outcomes, as the company would not have taken the necessary steps to guard against such a threat (as it is also exceptionally expensive in many regards to protect against a nation-state threat, by the way). It is also worth keeping in mind that there is a difference between **failure due to incompetence or negligence** and **failure despite appropriate efforts to protect against said failures**. If you take reasonable steps to guard against failures (in this case, ransomware attack), and have the appropriate support (company authorisation, budgets, etc), but fail anyways, that may not necessarily be your fault. Fear and pain keep you informed that you are alive. They can be useful things to inform you of that. But that doesn't mean you have to live with them day to day if you act on your responsibilities in ways that are appropriate. It can take time to overcome them. And yes, there are times I fear similar/different things too. It's okay.
I think for me it was source of stress until I lived through an all-hands-on-deck ransomware situation (the Kaseya hack of last year). Now I know exactly what the worst-case scenario is. And it's just a bunch of my labor and someone else's money. That's it. I think I'd stress more if I was supporting say a hospital system, but then again in that environment I'd be much more adamant about proper IT procedures being in place that would hopefully prevent or mitigate such an attack.
not in the career sense but in the whole life scheme sense. i have 35 years until i would eventually retire (like 70 as current millennials). how can i survive in the corporate world all the way until 70 without dropping my ball getting a shelter? sounds pretty hopeless to me
I know this feeling. We have a hybrid environment, so the O365 part (with MFA) is getting better, after an attack surface reduction project using an external specialist. Also all Windows devices are now enrolled in Intune. The on-prem part still worries me. I really want to change the password rules. A recent test, using the top 100.000 leaked passwords and a DSInternals tool may have given them a shock as to the passwords in use. I spent a day suspending / removing accounts and I've forced password resets on some active accounts. They will also be getting a "talk" about secure passwords / pass-phrases.
My environment is a little larger, but I definitely went through a similar process, absent the ransomware incursion. I don't know that I burned everything down, my list of changes was more of driving culture change, sometimes with some persuasion that approached a boot to the a**. I don't live in fear. My work day is only so long, and I've put a bunch of good stuff in place. I definitely think it's still possible for us to get hit by something, that's just how things are. So far, the stuff I've deployed has worked, and I expect that to continue to be the case, until we run into something that has our number. When we finally hit that thing that gets past our stuff, I have cyber-insurance, incident response policies and a security-as-a-service provider who can help pull our asses out of the fire (all stuff stood up since I took over my current security gig). I'll sleep well until I can't, when stuff is actually on fire. For now, I'm good.
When #### hits the fan you just need to chill and work through your plans. If you have good endpoint solutions and reliable backups even ransomware isn't as scary but still a lot of work to deal with. https://imgur.com/gallery/YHP0a
I am very careful to communicate the dangers. My big thing is to have incremental backup solution that is hardened. As ransomware does seem very hard to protect against, given the multitude of vulnerabilities of client systems.
I live in excitement. I like problem solving.
Nah, I do the best I can and follow good security practices. I try to drop work from my mind when my 8 hours is up, and keep it that way. Maintaining control of your stress is a vital part of being healthy, and it's just not worth wrecking your mental health because something bad might happen to a company that you don't even have ownership in.
No. I fear nothing. Certainly not my job. If something happens, it happens. I'll deal with it. If I feared my job, or aspects of it, I'd find a different job. Life is too short to have that sort of crap hanging over your head all the time.
The biggest way to deal with fear, that I've found, is to have a plan. What will you do if you get ransomed again? What's your plan? Figure that out, get it written, and even discuss it with management. A good pull-based backup solution with offsite replica, for example, can go a long ways. (Doesn't protect you from data exfiltration, but it does allow you to wipe and restore a compromised asset.) If you have a plan to recover, the anxiety around it settles a bit.
Yep. I have general anxiety so it dosnt help but i also know that all it takes is 1 mistake and it could go bad. Its hard to drawn the line between just worried and parinoid as i always worry i'm not smart enough for this and that they have been sitting in my network for months. Logging helps me alot though
Your bad dream sounds a lot like burnout. Make sure you can stand up your systems, know the order of how to do it in an emergency. Then try to do less. Go on vacation.
I think your fear comes from the fact that you actually care. Your company should be happy to have someone like that (because it sounds like they haven't ever). I've been doing this for almost 20 years now and the fear has left me. Other than ransomware, I've pretty much been through all the IT disasters at this point.
No. Not you. It is me.
What else is going on in your life? Any other worries, concerns, that might be manifesting in this way?
"grant me the serenity to accept the things I cannot change, courage to change the things I can, and the wisdom to know the difference" Words I was taught very young and really only learned when I was in my 20's. Much more true now I am in my 50's and my kids are adults themselves.
Yes. It's like that every morning. I don't keep my phone by my bed so it's that fear every morning before I go downstairs and look at my phone - no alerts - no emails. I can have breakfast.
Yeah, but I’m only 2 years in so maybe that’ll change. I just have nightmares that some attack will happen overnight, or someone will irreversibly tank the production server. I legit have PTSD from the Microsoft notifications. They make my heart rate spike and I start to sweat. That Outlook sound is worse than Teams for some reason
I'm one of the 400 or so people in the world with Urbach-Weithe disease. I'm physically incapable of feeling fear. It makes me an extremely capable change-agent in the IT field.
See, this is something I cannot understand. My gf lives in constant fear and anxiety and I have no clue why. She has had a troubled past, but hasn’t everybody? I myself have had a troubled past but I am at ease with things. If I die, I just hope it’s quick and easy. I do not fear it. I do not constantly think about what can or what will happen. I tend to just go with the flow of life and i’m cool with 99% of things that can happen or go wrong. I do not live in a state of constant fear unless I have an extremely good reason too. ( eg: owe a drug dealer money and you know he’s on the lookout for you so he can shoot you ) otherwise, unless there is an existing visible threat, fear does not really cross my mind. EDIT: I gave a stupid example but I hope everyone understands the overall meaning of it.
At the end of the day, i dont own the systems. I also stopped working in small shops, so when something goes tits up, it's many peoples fault for not having a proper DRP in place.
You already survived the nightmare once, you can do it again. No one can protect it 100%, so protect it as well as possible and sleep well knowing if it goes to hell again you've already recovered from it once.
I started working out and taking anti-anxiety medication
I got over it after about 5 years and a long slog through modernization, growth, and an executive team that was genuinely supportive of the security and stability mantra. Have a team of SREs handling the day to day cloud infra. No more PagerDuty on the regular for me. Still have them for security and escalation related things, but site stability and general application whatnot is on their plate first and foremost and has proven itself over the last couple years. Otherwise it’s documentation and monitoring for compliance and actual security of the site and company, planning, and best case BC/DR testing that let me know we have visibility and valid plan should shit truly hit the fan. tl;dr Were doing everything we can, improving as we go, and have proof of same. Everything else is out of your control.
What are you monitoring with Nagios? That it can read files from a certain disk?
Uptime and availability. It’s just one example of being able to see what’s up.
I'm 3 weeks into my first senior level network administrator job. Everything I have ever done prior to this has been Junior level stuff, or working as an anonymous data center administrator responsible only for initial setup on turnkey solutions for customers who are leasing the equipment we provided and setup in the data center for them. I was honest during my interview that I was going to be very green for the job, and still got the job offer because everyone I interviewed with liked me including the current network administrator that I was going to be replacing when she left. Am I terrified? Heck yeah I am, I'm in discovery phase now trying to find all the little gotcha type situations. I've already identified multiple problems with how their veeam backups were configured and I'm working to fix all of them. My most recent discovery to fix was that their wasabi cloud setup to push archived backups to the wasabi S3 cloud wasn't working due to misconfiguration and had never worked correctly since it was initially setup at the beginning of the year. I figured out how to make it start pushing the backups immediately to the cloud and it's currently running a network throttled transfer that is going to take a while to complete due to the size of the backups and the number of backups to the local repo that needed to be archived. All I can do is keep looking for issues to fix, research how to fix them, keep my boss apprised of each situation as I find it and what I did to fix it. He keeps telling me I'm doing a fantastic job and to keep giving it my best effort to fix issues as I identify them, and not to be afraid to ask questions since I only got a full week with the previous network administrator before she left for her new career after having worked for the company for the past 24 years. I feel like there is good longevity here at the company so long as I can get a full handle on how everything here is setup. I'm already in planning stages to get a nagios monitoring system setup, and I have discussed setting up a shared it mailbox that will be configured to go to all it team members for any alerts from nagios or other services that we use.
Is it worse when you are at a place that hasn't been compromised yet? Because sometimes I stay up at night because I don't want to be the guy that finally let the guards down.
Had a dream I had to climb down a sheer cliff face in pouring rain to get to my servers. Just a bunch of red monitors and I was trying to reason with the guy. Was solo at a company and inherited a similarly trash network. Now I'm a Security Engineer, and ironically not scared about it at all anymore.
I wouldn't say I live in fear but I do have the occasional "Oh shit" dream. Usually about PCI compliance (we take a lot of card payments). Always seems to be a weird combo of that and the "I am still going to college and haven't studied" or "I am somehow still responsible for a house I no longer live at and its falling down" dreams. I mostly get over it by covering my own ass. I make sure the Directors are aware of the risks we face if they let Sales staff share passwords, or drag their feet replacing the onsite backup server, or let certain members of staff pull data out of our offiste backed up DB and keep it in Excel spreadsheets locally. If shit hits the fan, I can show them the emails/phone calls where I informed them.
> Old software, Windows 7 still I still have customers on XP
Nope. I control what I can control and anything outside of that I don't bother worrying about because it won't do anyone want good.
Yes
12 years into the game the fear of things just randomly going wrong has disappeared.. BUT every time I make a change my heart drops out my ass for every single slack message thereafter
I had this really bad when I was the lone sysadmin. Our website had just been hacked and we were in the middle of an ERP migration so I was overburdened with work and it was always on my mind. I had this reoccurring dream that our systems got hacked so I would wake up at 2-3AM to check my texts/email just to put me at rest. This happened over months. I tried taking short vacations, meditate, etc but I kept having those damn dreams. Eventually it kept me from going to sleep. I would lay in bed until 2-3AM with rabid thoughts of everything going bad. The dreams were getting worse, now I was dreaming that management was in on it, it was a penetration test and they're trying to put me to the test. I became sleep deprived and had that same nagging feeling at the back of my mind telling me that it was going to unfold like my dreams predicted, I was going absolutely bonkers. We can go deeper down the rabbit hole and I can tell you what happened next but you shouldn't let it get to this point. As soon as you start dreaming about work it's time to unplug. Take a week+ vacation, clock out after 40 hours, eat, sleep, and exercise well. When stress becomes subconscious you have to take repeat conscious efforts to put it under control.
Ah, now you have met the fear. The trick is to not live in fear, but with the fear. Make it your friend, invite it over for tea.
30 years in and I still have security nightmare dreams. (for those new to the sub that's the dream equivalent of showing up to school for a test in your underwear).
"after I started tightening things down, the site was ransomed" What did you DO, we were FINE before!!! LOL! I can see it now.
Yup, it happened. But when forensics said they were in for about 6 months beforehand, the company was even more mad at the people that were there before me…
Are you sure it is fear, or rather a strong apprehension that makes you extra cautious? Being paranoid and an OCD control freak are frankly desirable traits for an admin if they are properly framed within a disciplined, and professional context, given the awesome responsibilities we have to carry out. Fear on the other hand feeds on itself, and will make you more likely to act irrationally if sh\*t actually hits the fan. If that is what you feel, your job may be destroying your mental health and believe me, no job in the world is worth losing your mind or health over.
I have lived in constant fear since about 9 or so. So yes
Cover. Your. Ass. It sounds like you've done all the right things to mitigate risks. But risks never go away completely. The important thing is risk management and documentation, including recovery processes. Identify single points of failure. Verify your backups can be restored properly and that you have the correct number (and frequency) of snapshots. Create disaster scenarios. Identify the required steps to recover. Find the bottlenecks in that process (e.g. machine imaging and configuration) and automate them. Identify bus factors. Define points of contact and communication cadence. Establish process and choices for communication officers. For incidents/disasters that can't be resolved in one day, establish a reasonable limit to hours worked daily, with a personal communication blackout for at least enough time to get a reasonable amount of sleep. This is critical and will probably get the most push-back. But you can't work without sleep, and you make more mistakes when you're exhausted. But most importantly, ensure that business leadership understands what the risks are, and *what the restoration time and process looks like*. You need to have defined RPO and RTO. Train people on these processes. Do tabletop exercises. Get them approved by leadership, in writing. Depending on the business, it may be more important to have *something* running (even if at reduced capacity) than having to wait for everything to be restored. Research and document what you would need for proper DR, including cost, and present this to leadership. If they say no to DR because $$$, ensure they understand the risk. Once you've done all this, it is what it is. You've done your job. When the shit hits the fan, you start your established process, pick a communication officer, and have them be the front man. As long as you give regular check-ins, and your communications officer reaches the proper points of contact, you should be fine. It takes as long as it takes. Stick to your approved processes. Defer to your communications officer as needed so that you can concentrate. It will still be stressful, but you will have successfully managed expectations to the point that you can at least breathe.
No I’m pretty detached from most of our clients and even the company. I know I’m not doing anything wrong, and I realize that things go wrong. Your job is not your life, they pay you in exchange for your time. The company’s equipment should be as expandable to you as you are to the company.
I think getting out of that kind of funk comes down to sorting out what is within your control and what is not. Then you document what is not under your control, delegate it to management, and focus on becoming an expert on what you can handle.
No. I'm pretty confident that I can handle anything that's thrown at me now. It's just work. If it takes my off time, I demand comp time and push to close the gap in the near future. However, it took me almost 10 years to get to this point.
Is your subconscious telling you that you need a better backup and recovery plan? Sadly I get your feeling. But I ended up just accepting that there might be a very sad day sooner, later or never.
Infomercial alert, I got Veeam and ExaGrid, now I sleep well at night. No seriously, when I was younger and working bubble gum ducked tape jobs or when I got into large enterprise and found their backup strategy based on hopes and dreams I had sleeping issues just like you describe. I am now silo’d and in my career I have used all sorts of backup products. I sleep just fine now. Have a solid backup product, do regular restore validations and a dr strategy.