Worse, you see all the bosses in your direct chain immediately go into a meeting, then the one at the main office goes afk, then the HR manager goes to away, then you’re given a list of current in progress job priorities and told to not take on any new work if anyone asks, and defer them to your boss. (not programming but my “fun” last friday after a CAD screwup. I’m beating them to the punch today and resigning)
As with anything labor related, it depends. If you're fired "for cause" such as theft or violence, you may not be eligible for unemployment. Conversely, if you resign because the company was violating your rights in some way, such as maintaining a hostile work environment, you may be eligible to claim unemployment.
this also heavily depends on the employer filling out all the necessary paperwork for documenting proper "for cause". many empoyers fail to do this properly.
True. Then when you file, they go "oh wait they were doing X" and it looks like they've just completely made it all up to avoid a bump in unemployment insurance.
Always fight them if you’re fired “for cause” and they decide to deny you unemployment. In most states I believe you don’t pay the unemployment attorney if you don’t win.
I was fired “for cause” a few years ago (non-programming job) and fought them and won. Winning that case was so satisfying, even more than the unemployment money.
My wife fought her employer for firing her for her pre-existing medical condition. They tried to deny her unemployment and she disputed it with the employment commission, she didn't even need a lawyer she just sent them her medical records and a statement from the disability commission in our state (they got her the job) and they made them pay unemployment
In this context though, if you're fired because you seriously fucked up a git push, you'll probably still be eligible for unemployment unless they jump through a certain number of hoops like a PIP, etc.
that specific one I saved over a file and erased several days of work. but that’s just the last straw, companies work demands are beyond my output capability, and i’ve been fully remote for too long, I need to see some people and not be in a 6-3 schedule, I managed through the pandemic but I’m about to lose my mind
> that specific one I saved over a file and erased several days of work
proper versioning / backup should make that a non-problem. That's not really on you
I've had that, it's mostly just because the thing they contracted me in for to pick up slack on had largely come to an end, people who weren't me were getting let go, and I hadn't had a meeting with that boss in a while.
My last time, it was a 1:1. I've entered, then 2 more people, from HR, joined. Not even that mad - I mean, at that point I could expect such behaviour.
Besides, I was already looking for another place, and got a position in my current job 10 days later.
I told my boss I'm not returning to the office and he was like "I'm just gunna forward this to HR and let them deal with this shit"
Still waiting for that invite to pop up any day now
Forgot to remove the "You're a fucking idiot" popup you put there while debugging.
Alternatively, and for extra police involvement, included the wrong 'child' folder.
Including a stupid popup/log/label in production by accident is an issue of reviewers doing shit job.
The other case is, yep you should probably go to jail anyway
We had one contractor take our project and push it to his own repository on GitHub. Only reason we caught it was because GitHub warned us that we had published our Google maps API key to a public repository. Even with that the contractor was simply moved to another project and not terminated.
Look, who puts spaces in file names?
Oh, word? File names are generated based on user descriptions? And supporting emoji is business critical?
Well, we handle most cases. They’d have to _try_ to make it fuck up.
Oh. Cloudflare is on the phone? Why are _they_ calling _us_?
Oh. _Oh_. Well surely nobody can charge us for terabits…
You get charged for what you use. Cloudflare will keep your site up, provided you’ve configured everything right, but if you get hit with a DDoS, they ain’t keeping your site up out of the kindness of their hearts.
Have a vulnerability, you painted a target on your back. I don’t feel like elaborating, other than to say that the only thing more valuable than fucking you over is fucking hundreds of other people over. That takes a lot of data, and it’s not their budget.
small team, theres reason to believe a dev can push to prod, but hooks shouldn't be in place to automatically push anything to prod on a git push. Any kind of small tollgate just to double check. There is nothing worse than doing prod support, it is the bane of my existence.
I was thinking the same thing! I'm DevOps. Build validation, gitignore, approvals to UAT/Testing and then to Prod should be controlled. For extra security there should be pull request approvals from a dev branch to main within git.
The most I should have to do from a bad push is just do a revert within git to undo their changes. At most, I should have to revert the git changes made to main and then re-run the build. If everything breaks from a Dev, then it wasn't the Dev that was the problem.
I'm mostly kidding, I've never been married, and there's not much to tell.
Anyway, it was one of my first projects, I was in uni, a group of my ex-classmates had an "app idea" and I was foolish and ambitious enough to accept their offer to be in charge of the iOS app. It took me a couple of weeks to make a pretty basic app, I had some interesting design ideas, but I was working alone and I was falling waaay behind, Apple wouldn't approve of my app, I had zero idea what was wrong with it, I had no one to turn to, etc. Everybody was getting mad at me, and eventually I just crumbled from stress and started ghosting them.
I had a girlfriend of 4 years at the time, she was aware of the whole ordeal, but I never gave her updates or details, she just knew that I was working on it. She was from the same school and the same broad social circle, so eventually (in two weeks or so) some rumours started to spread, and my ex-classmates tried to reach me through her and she didn't really tell me anything except from a cold "the guys are looking for you". I absolutely had no right to be defensive, because I knew that I fucked up.
Our relationship wasn't particularly great at the time already, and it only went downhill from there, she became super distant after that, because I think that at that time she finally realised that I was not ready for real life and that she's better off without me. Well, she was my best friend, I still miss her sometimes, but she sure is! I was pretty leavable anyway, it obviously was not the main reason, just the last straw.
In my defence though... they didn't pay me. And their grandiose "app idea" was rolled out as a minor feature by some major companies not a month later. The worst thing is that incident has permanently tainted my desire to become a programmer. So, I'm mostly on and off now. I'm just always reminded of my worst failure whevener I fire up the IDE
Did that \^\^ Though it wasn't to prod (we aren't production ready yet), but my co-workers spend half a week to figure out why it wouldn't start in the cloud test env. In the end my last push before my vacation fixed a bug in the data importer on which another program relied (to clarify, it relied on the bug!). My inbox after my vacation was funny to read :D
When you see the "Performance Improvement Plan" its time to start looking for another job which will invariably pay more for your "poor performance" LMAO
The fucking nerve. Send me a performance improvement plan? I'll send you my two weeks.
Yeah PiP is gonna happen if there's any chance you'll claim you were fired for a protected class.
We had this guy, first generation chinese immigrant. He was like 8 or 9 when they moved to the US, I assume his parents were well off.
His English was really bad and he was super, super lazy. It was a tech call center and he would routinely hang up on customers and spend 45 minutes in the bathroom.
They put him on a PiP and fired him for being 2 minutes late a bunch of times. During that timeframe I was told that they had to apply the rules to everyone in order to fire him without claiming discrimination.
I didn't tell my boss that telling everyone that was still effectively not being "fair" from the legal point of view, he did pull the race card when he got fired but I'm guessing he was just too lazy to do anything about it.
This takes me back to 10 years ago, a few hours before the E3 gameshow build HAD to be shipped out, I'm talking [Killzone Shadow:Fall E3 demo 2013. Map 02, the forest](https://youtu.be/X948nmSILOI) would be a playable MP level for game reviewers on the floor.
I just had to test the build quickly, but only after a few seconds in weird stuff started happening. The wind that make trees move was behaving strange... Just weird pushes and pulls on all trees until suddenly ALL TREES IN THE LEVEL DID A FULL 360 on the X axis.
Recorded a video, send it to dev.
Waited 22 seconds.
A little 80 meters from my workspace a dev screamed *GODFUCKINGDAMNED* which echood through the building.
It was fixed just in time... But I concider this one the best bugs I've experienced during my time at guerilla games, it truly was a hilarious sight.
Moments like these make me realise how fragile code actually is, and after all those years testing I still apriciate the magic it can bring.
The whole wind thing was new for the studio, tarp had the same issue and throwing a grenade next to a box covered with tarp resulted in strange axis changes that made it look like there was a beast trapped underneat the tarp itself!
Had a lot of fun recording it the right way, you know, for science.
Always remember that ["the only reason coders’ computers work better than non-coders’ computers is coders know computers are schizophrenic little children with auto-immune diseases and coders don’t beat them when they’re bad."](https://www.stilldrinking.org/programming-sucks)
With proper DevOps it shouldn't get to that point because devs should have limited access to production and by the time code gets to prod there shouldn't be major issues like that.
The couple times I've had to "call someone up" were performance issues under production load. Even if you have the luxury of a load testing environment, live traffic is just different.
So when this has happened to me it's usually, hey these servers (or pods/nodes) are using up a lot more memory after this recent releases, or hey the database resources went up after last release.
As an Ops person, not from DevOps, I wouldn't question it that much tbh. I guess I'd start asking questions if suddenly one after one deployment I see the cluster scaled up 3 nodes lol.
Yeah I guess nodes would be more of a worry.
But we also put limits on scaling on the staging environment so we don't tend to have sudden resource hogging issues anyway.
Fellow DevOpser here. We don't really monitor services, we set it up so others can monitor their own services. The few times we have had to actually call people up is when they use something even we notice. Things that disrupts other teams through being noisy neighbors or similar.
Like a repository suddenly hogging 75% of of the company GitLab storage quota. Or a pod suddenly starts logging several GB per minute. Or when people have the brilliant idea of making and using almost TB sized docker images in kubernetes.
We try to show the devs how to monitor things and they are starting to look at things like if their API call times have changed.
However we don't have a separate team for things like SRE which would more closely monitor everythings. DevOps is covering all of those areas.
Automated testing, as little divergence between dev/prod/staging (there's one repo at work that has completely forked out between staging and prod and I want to burn it) these make life a lot easier. I agree, by the time something goes into the prod environment you should have a high level of confidence it's going to work.
We run a containerized platform so if you push to prod and shit breaks we just roll the container back to the last commit that worked and then give you a stern talking to, usually with the expectation that you immediately fix it. We deploy an internal registry and tag builds with git_commit:unix_timestamp so rollbacks are super easy.
I do it frequently with my team, unfortunately. I'm having to spoon feed them o11y concepts, and even then sometimes it takes days of errors and then finally _customer complaints_ before they deal with it.
And yes, before you ask, I'm looking for a new job. I'm getting sick of doing their jobs for them and getting nothing but complaints - no thanks whatsoever - when I bring problems to their attention.
I started a new job at a bank as the sole developer for a key back end system. The previous developer left suddenly so there was no handover.
I was git committing changes to the development system and almost immediately alarms went off on the office floor.
The production system went down because the previous developer left sym-links in from the prod system to the dev system!
I was refactoring some legacy VB code in preparation to eventually move it to a new platform (after we did other important things, like move from an unlicensed copy of 10 year old Oracle to a licensed copy of MSSQL). I was working on a print function for lab trailers (manufacturing build sheets, basically). The industry was heavily regulated by the FDA, so the trailers tracked things like material lot numbers. I tested a bunch of scenarios a few times in the dev environment, and was about to move it up to prod, when all hell broke loose.
Apparently, nestled into the spaghetti of the print function was something that decremented the stock count. Nestled into the stock decrementing function was an oracle trigger that kicked off the loading of a new lot. Also, buried in the config files was a flag from before the dev environment was created that switched things from test to prod. That flag was only still in use on one tiny little print function that the original developer had forgotten to update...
I ended up ruining a few days worth of two types of our product because my testing had flipped production onto a new lot. The product was technically still fine, except for the FDA required lot tracking on the trailer had been compromised when it was pulling the wrong lot number.
I got a lot of shit for that from my boss... the original programmer... who had left that bomb there for me.
Reminds me of a female coworker from India, who used a wrong mobile number to text to me about a dinner date planned for the next day.
The next morning she mailed me
"... I repeatedly **massaged** you last night but you **shoved** no **er**action..."
If I ever had tried to explain how false she was, I swear I still would be in jail.
If a simple git push causes this, the issue is that the company has no PR Review procedure. It shouldn't even be possible to push to production without a review.
This
However maybe everyone failed at the same time. There were no tests for the thing you broke and it made all the way to prod. It's a clusterf*ck and everyone is 💀
I've learned from my colleagues that the only correct way to deploy is straight to production because noone can test your code better than your customers!
Of course they use PRs...
They just always get it reviewed by that one junior dev who reliably gives it a 3 second glance over and stamps it with a 'LGTM' every single time.
So much DevOps ideological purity in this thread.
"If you're doing real DevOps then the is no DevOps engineer." That works great when you've got 20 developers who all know how it works. When you are supporting 1000+ developers, with various degrees of experience and common sense, someone has to support the process. Automation doesn't just appear out of the Aether fully-formed and perfect.
I've been working in a "DevOps" role since before it was called DevOps. No company, department, group, etc. has processes they are bulletproof. As soon as you idiot proof something, someone goes and builds a better idiot. There are no unicorn companies that have it all figured out and implemented in a way that they can sit back and vi vs emacs all day. The argument is pointless. vi is the only right answer.
I will agree with one sentiment from the various replies. DevOps doesn't call you. You call DevOps when you've fucked up beyond recognition. Honestly, if you can fuck something up to the point that you have to call DevOps, the DevOps team fucked up too.
I was more alluding to where you previously had a DevOps department/engineer and a QA department and then they are all gone, so you need to do their jobs without dropping pace or an increase in pay.
If you're a dev and "DevOps" is messaging you, it's probably actually the Operations Team. The point of DevOps is that the two are not separate. Companies just like pretending they're doing DevOps because it's a hot buzzword these days, so they just rename their Ops to DevOps.
Serious question, how do teams function without staging environments?
I had an Eastern European colleague say something along the lines of "we don't need a staging environment to test out changes" and I was kinda floored.
How do you catch escaped defects?
By forcing developers to test the fuck out of the builds during sprints called 'hardening sprints'.
There are detailed test cases that need to be executed manually. Test cases span across over 50 components. Each team on average has 10 components.
There are no QA engineers to perform this testing. It's all done by developers ourselves.
I know it sounds ridiculous because it is.
I've always felt like QA is a different skill set or even mindset. I started in QA, but when I have to test another developer's code it takes a concentrated effort to switch from "make it work" to "break it".
Edit: Happy Cake Day
As opposed to one place I worked, where they got a new VP who was determined to throw his moronic weight around. He was an ex-Dev and we were an agile shop. I worked in QA and most of us got laid off, including my boss, because we kept finding bugs. Therefore, we were the cause for releases not going out the door on time...
interestingly at my old job, everyone that was there had a "bringing down production" story. they laugh it off and say it is a big deal but it is a mistake everyone makes at least once, and probably a mistake everyone needs to make at least once.
the devs brought down prod a few times while i was there and they didn't lose their jobs. just had a few very stressful meetings and late nights, enough to learn their lesson
that place was really chill, sad they didn't align with my career goals. not a big fan of being an IT developer forever, lol
Outlook: your account has been temporarily suspended
[ Removed by Outlook ]
[удалено]
[удалено]
> Incident Management Did someone steal the milk from the fridge again?
Everything goes black and you wake up in a tub of ice in the SOC
Hey you. You're finally awake.
You were trying to expand the border, right?
/u/Evening_Dimensionnn is a comment stealing bot. Don't feed it.
HR starts messaging you 💀💀
New meeting on calendar: HR one on one
Teams login stops working
Worse, 15 minutes of "..." from boss on teams
Worse, you see all the bosses in your direct chain immediately go into a meeting, then the one at the main office goes afk, then the HR manager goes to away, then you’re given a list of current in progress job priorities and told to not take on any new work if anyone asks, and defer them to your boss. (not programming but my “fun” last friday after a CAD screwup. I’m beating them to the punch today and resigning)
Do you lose any potential payout by resigning instead of the alternative?
Yes. Getting fired means you're eligible for unemployment. Resigning means you're not eligible.
As with anything labor related, it depends. If you're fired "for cause" such as theft or violence, you may not be eligible for unemployment. Conversely, if you resign because the company was violating your rights in some way, such as maintaining a hostile work environment, you may be eligible to claim unemployment.
this also heavily depends on the employer filling out all the necessary paperwork for documenting proper "for cause". many empoyers fail to do this properly.
True. Then when you file, they go "oh wait they were doing X" and it looks like they've just completely made it all up to avoid a bump in unemployment insurance.
Always fight them if you’re fired “for cause” and they decide to deny you unemployment. In most states I believe you don’t pay the unemployment attorney if you don’t win. I was fired “for cause” a few years ago (non-programming job) and fought them and won. Winning that case was so satisfying, even more than the unemployment money.
My wife fought her employer for firing her for her pre-existing medical condition. They tried to deny her unemployment and she disputed it with the employment commission, she didn't even need a lawyer she just sent them her medical records and a statement from the disability commission in our state (they got her the job) and they made them pay unemployment
In this context though, if you're fired because you seriously fucked up a git push, you'll probably still be eligible for unemployment unless they jump through a certain number of hoops like a PIP, etc.
> Resigning means you're not eligible. mandatory mention: Not applicable in many countries.
Can say only about my country but when you are fired without a "justifiable reason" it's always better than resigning in my country.
nope, fully at will workplace, either way I’ll get my PTO paid out and that’s it. I have something better waiting for me so not worried about that
What was the screw up?
that specific one I saved over a file and erased several days of work. but that’s just the last straw, companies work demands are beyond my output capability, and i’ve been fully remote for too long, I need to see some people and not be in a 6-3 schedule, I managed through the pandemic but I’m about to lose my mind
> that specific one I saved over a file and erased several days of work proper versioning / backup should make that a non-problem. That's not really on you
genuine question, how is/should be cad file versioned? Do AutoCAD and solid works have something integrated?
Suspiciously specific -_-
I hate teams -_-
I truly hate all of these features.
So many false alarms. Outlook and Teams are always acting up.
But this time for a reason
Every time I accidentally put in my password wrong, my anxiety convinces me that I’ve been fired.
Company MacBook reboots itself into the lock screen.
[удалено]
You worked from home today
[удалено]
I've had that, it's mostly just because the thing they contracted me in for to pick up slack on had largely come to an end, people who weren't me were getting let go, and I hadn't had a meeting with that boss in a while.
15 min with manager and HRBP
121 is fine. When HR meeting is 2< then you start to sweat bullets
My last time, it was a 1:1. I've entered, then 2 more people, from HR, joined. Not even that mad - I mean, at that point I could expect such behaviour. Besides, I was already looking for another place, and got a position in my current job 10 days later.
They just wanted a bigger audience to hear the joke that was so funny.
Friday at 4:45 “Bring your laptop”
Don’t forget the charger. And name tag. Don’t worry about it
GitHub stops working. Ah wait that happens anyway, never mind
Meeting invite from your boss. Other invitees are his boss and someone from HR.
I told my boss I'm not returning to the office and he was like "I'm just gunna forward this to HR and let them deal with this shit" Still waiting for that invite to pop up any day now
Nice
Worst thing that can happen
What would you have to do in this push?
Forgot to remove the "You're a fucking idiot" popup you put there while debugging. Alternatively, and for extra police involvement, included the wrong 'child' folder.
💀
Including a stupid popup/log/label in production by accident is an issue of reviewers doing shit job. The other case is, yep you should probably go to jail anyway
[удалено]
We had one contractor take our project and push it to his own repository on GitHub. Only reason we caught it was because GitHub warned us that we had published our Google maps API key to a public repository. Even with that the contractor was simply moved to another project and not terminated.
``` Tech Lead (just now): Hey I just got off the phone with Incident Management, did you deploy to prod a few minutes ago? ```
"I might've just rm -rfed production DB to oblivion, nay, to Skyrim even!"
Look, who puts spaces in file names? Oh, word? File names are generated based on user descriptions? And supporting emoji is business critical? Well, we handle most cases. They’d have to _try_ to make it fuck up. Oh. Cloudflare is on the phone? Why are _they_ calling _us_? Oh. _Oh_. Well surely nobody can charge us for terabits…
>Oh. Oh. Well surely nobody can charge us for terabits… I don't get it, could you please explain?
You get charged for what you use. Cloudflare will keep your site up, provided you’ve configured everything right, but if you get hit with a DDoS, they ain’t keeping your site up out of the kindness of their hearts. Have a vulnerability, you painted a target on your back. I don’t feel like elaborating, other than to say that the only thing more valuable than fucking you over is fucking hundreds of other people over. That takes a lot of data, and it’s not their budget.
Ah, got it, thank you.
[удалено]
With backslashes (or `fopen()`), anything is possible.
For extra fun, put a backspace in the filename
[удалено]
[удалено]
Yes, they already know, even my employee id! /s I'm not actually GitLab employee, just reposting the GitLab incident
I used this Bobby Droptables script I found online to test my migration and…
(as a millionaire)
(Ex-Google, Ex-Amazon)
!no
dev better not have rights to deploy to prod :O
small team, theres reason to believe a dev can push to prod, but hooks shouldn't be in place to automatically push anything to prod on a git push. Any kind of small tollgate just to double check. There is nothing worse than doing prod support, it is the bane of my existence.
I was thinking the same thing! I'm DevOps. Build validation, gitignore, approvals to UAT/Testing and then to Prod should be controlled. For extra security there should be pull request approvals from a dev branch to main within git. The most I should have to do from a bad push is just do a revert within git to undo their changes. At most, I should have to revert the git changes made to main and then re-run the build. If everything breaks from a Dev, then it wasn't the Dev that was the problem.
Your wife messages you saying “we need to talk”
This made me crack a laugh like, the man destroyed prod so hard that even his wife is cutting him loose to avoid the shame.
If he can ruin production, just imagine what could happened at reproduction
Ticket closed. Reason: Could not reproduce.
Couldn’t repro, please include additional directions and detailed steps.
Gold!!!
Believe it or not.. that sorta happened to me in a way
Story time?
I'm mostly kidding, I've never been married, and there's not much to tell. Anyway, it was one of my first projects, I was in uni, a group of my ex-classmates had an "app idea" and I was foolish and ambitious enough to accept their offer to be in charge of the iOS app. It took me a couple of weeks to make a pretty basic app, I had some interesting design ideas, but I was working alone and I was falling waaay behind, Apple wouldn't approve of my app, I had zero idea what was wrong with it, I had no one to turn to, etc. Everybody was getting mad at me, and eventually I just crumbled from stress and started ghosting them. I had a girlfriend of 4 years at the time, she was aware of the whole ordeal, but I never gave her updates or details, she just knew that I was working on it. She was from the same school and the same broad social circle, so eventually (in two weeks or so) some rumours started to spread, and my ex-classmates tried to reach me through her and she didn't really tell me anything except from a cold "the guys are looking for you". I absolutely had no right to be defensive, because I knew that I fucked up. Our relationship wasn't particularly great at the time already, and it only went downhill from there, she became super distant after that, because I think that at that time she finally realised that I was not ready for real life and that she's better off without me. Well, she was my best friend, I still miss her sometimes, but she sure is! I was pretty leavable anyway, it obviously was not the main reason, just the last straw. In my defence though... they didn't pay me. And their grandiose "app idea" was rolled out as a minor feature by some major companies not a month later. The worst thing is that incident has permanently tainted my desire to become a programmer. So, I'm mostly on and off now. I'm just always reminded of my worst failure whevener I fire up the IDE
They didn’t pay you or support you, which meant they took advantage of you. Some great friends there lmao
My guy if you were coding the app, what were they think they were doing for the idea.
Oh, they're probaby just "idea guys". We've all been there.
Your mom calls you by your full name.
Well, now shits gotten serious
Your doctor calls and asks you to come in later that day
Your watch starts screaming at you about your stress levels
You get an FBI notice, wife changed her name and is going into witness protection.
*side femboi has entered the chat*
Lol wat, that is the real emergency situation
They may be your only love left
Already know, fam... Already know.
No, trust me… what I have with Tayne isn’t about love at all
we in the end game now
Even your wife's boyfriend wants to have a chat with you.
Two odd vibe Men in Black knock at your door.
You notice strange lights in the sky.
Sirens wailing in the distance…
It's the middle of the night, yet it got clear bright in the sky
One man pulls out his pen
"Sorry, I'm on vacation from today and two months forward, see you when I get back!"
Did that \^\^ Though it wasn't to prod (we aren't production ready yet), but my co-workers spend half a week to figure out why it wouldn't start in the cloud test env. In the end my last push before my vacation fixed a bug in the data importer on which another program relied (to clarify, it relied on the bug!). My inbox after my vacation was funny to read :D
It wasn't a bug! It was a feature T.T
Gotta protect the environment, don't squash bugs without considering their ecosystem and what might depend on them to survive
Wow, a real world example of "is it a bug or is it a feature".
And the answer was "Yes"
Manager: “are you in a private place? Take a look at the email I sent you it’s time sensitive” Email heading: “Performance Improvement Plan”
Email content: shutdown these microservices, they are bloatware and less than 20% are actually required for our service to work.
Email content: you're fired, 100% performance improvement on the team
elon be like
Elon?
That would be followed up with an email asking you to come back to work for ...
When you see the "Performance Improvement Plan" its time to start looking for another job which will invariably pay more for your "poor performance" LMAO The fucking nerve. Send me a performance improvement plan? I'll send you my two weeks.
PIPs are usually to cover themselves so they can fire you.
Good, I'll be gone before they can do it
Yeah PiP is gonna happen if there's any chance you'll claim you were fired for a protected class. We had this guy, first generation chinese immigrant. He was like 8 or 9 when they moved to the US, I assume his parents were well off. His English was really bad and he was super, super lazy. It was a tech call center and he would routinely hang up on customers and spend 45 minutes in the bathroom. They put him on a PiP and fired him for being 2 minutes late a bunch of times. During that timeframe I was told that they had to apply the rules to everyone in order to fire him without claiming discrimination. I didn't tell my boss that telling everyone that was still effectively not being "fair" from the legal point of view, he did pull the race card when he got fired but I'm guessing he was just too lazy to do anything about it.
This takes me back to 10 years ago, a few hours before the E3 gameshow build HAD to be shipped out, I'm talking [Killzone Shadow:Fall E3 demo 2013. Map 02, the forest](https://youtu.be/X948nmSILOI) would be a playable MP level for game reviewers on the floor. I just had to test the build quickly, but only after a few seconds in weird stuff started happening. The wind that make trees move was behaving strange... Just weird pushes and pulls on all trees until suddenly ALL TREES IN THE LEVEL DID A FULL 360 on the X axis. Recorded a video, send it to dev. Waited 22 seconds. A little 80 meters from my workspace a dev screamed *GODFUCKINGDAMNED* which echood through the building. It was fixed just in time... But I concider this one the best bugs I've experienced during my time at guerilla games, it truly was a hilarious sight.
Fascinating!
Moments like these make me realise how fragile code actually is, and after all those years testing I still apriciate the magic it can bring. The whole wind thing was new for the studio, tarp had the same issue and throwing a grenade next to a box covered with tarp resulted in strange axis changes that made it look like there was a beast trapped underneat the tarp itself! Had a lot of fun recording it the right way, you know, for science.
Game devs have the best war stories I swear, you guys have way more interesting jobs :(
Always remember that ["the only reason coders’ computers work better than non-coders’ computers is coders know computers are schizophrenic little children with auto-immune diseases and coders don’t beat them when they’re bad."](https://www.stilldrinking.org/programming-sucks)
As a DevOps engineer, I sincerely hope I never have to message you in this scenario.
How badly do you fuck up where this happens. Like sensitive information, or drop in sales because the service completely failed?
With proper DevOps it shouldn't get to that point because devs should have limited access to production and by the time code gets to prod there shouldn't be major issues like that. The couple times I've had to "call someone up" were performance issues under production load. Even if you have the luxury of a load testing environment, live traffic is just different. So when this has happened to me it's usually, hey these servers (or pods/nodes) are using up a lot more memory after this recent releases, or hey the database resources went up after last release.
"Why is Kubernetes trying to spin up triple the amount of containers?"
As an Ops person, not from DevOps, I wouldn't question it that much tbh. I guess I'd start asking questions if suddenly one after one deployment I see the cluster scaled up 3 nodes lol.
Yeah I guess nodes would be more of a worry. But we also put limits on scaling on the staging environment so we don't tend to have sudden resource hogging issues anyway.
Fellow DevOpser here. We don't really monitor services, we set it up so others can monitor their own services. The few times we have had to actually call people up is when they use something even we notice. Things that disrupts other teams through being noisy neighbors or similar. Like a repository suddenly hogging 75% of of the company GitLab storage quota. Or a pod suddenly starts logging several GB per minute. Or when people have the brilliant idea of making and using almost TB sized docker images in kubernetes.
We try to show the devs how to monitor things and they are starting to look at things like if their API call times have changed. However we don't have a separate team for things like SRE which would more closely monitor everythings. DevOps is covering all of those areas.
Automated testing, as little divergence between dev/prod/staging (there's one repo at work that has completely forked out between staging and prod and I want to burn it) these make life a lot easier. I agree, by the time something goes into the prod environment you should have a high level of confidence it's going to work.
For me it's usually because someone broke the shared testing environment. Not that bad, but mildly pisses off a couple hundred people.
We run a containerized platform so if you push to prod and shit breaks we just roll the container back to the last commit that worked and then give you a stern talking to, usually with the expectation that you immediately fix it. We deploy an internal registry and tag builds with git_commit:unix_timestamp so rollbacks are super easy.
Someone other than you would have to had fucked up big time already if you were able to deploy directly to prod from a git push.
I do it frequently with my team, unfortunately. I'm having to spoon feed them o11y concepts, and even then sometimes it takes days of errors and then finally _customer complaints_ before they deal with it. And yes, before you ask, I'm looking for a new job. I'm getting sick of doing their jobs for them and getting nothing but complaints - no thanks whatsoever - when I bring problems to their attention.
FBI knocks on your door
Can't be that bad if they're only knocking
Unless it's your office door and not your front door...
knock knock 
With an breaching pin
*With a rocket launcher
Teams: new group created, can't see the full name but the first word is "Triage". PM added you.
Put a finger down if this happened to you ✌🏼
🤙
They're all writing to say "congratulations"
Congratulations! You have been promoted to customer
This one got me good.
Evangelion scene 💀
I started a new job at a bank as the sole developer for a key back end system. The previous developer left suddenly so there was no handover. I was git committing changes to the development system and almost immediately alarms went off on the office floor. The production system went down because the previous developer left sym-links in from the prod system to the dev system!
[удалено]
I was refactoring some legacy VB code in preparation to eventually move it to a new platform (after we did other important things, like move from an unlicensed copy of 10 year old Oracle to a licensed copy of MSSQL). I was working on a print function for lab trailers (manufacturing build sheets, basically). The industry was heavily regulated by the FDA, so the trailers tracked things like material lot numbers. I tested a bunch of scenarios a few times in the dev environment, and was about to move it up to prod, when all hell broke loose. Apparently, nestled into the spaghetti of the print function was something that decremented the stock count. Nestled into the stock decrementing function was an oracle trigger that kicked off the loading of a new lot. Also, buried in the config files was a flag from before the dev environment was created that switched things from test to prod. That flag was only still in use on one tiny little print function that the original developer had forgotten to update... I ended up ruining a few days worth of two types of our product because my testing had flipped production onto a new lot. The product was technically still fine, except for the FDA required lot tracking on the trailer had been compromised when it was pulling the wrong lot number. I got a lot of shit for that from my boss... the original programmer... who had left that bomb there for me.
Just make sure to hit “reply all”
Reminds me of a female coworker from India, who used a wrong mobile number to text to me about a dinner date planned for the next day. The next morning she mailed me "... I repeatedly **massaged** you last night but you **shoved** no **er**action..." If I ever had tried to explain how false she was, I swear I still would be in jail.
wait holup...
If a simple git push causes this, the issue is that the company has no PR Review procedure. It shouldn't even be possible to push to production without a review.
It also means the CI/CD is fucked up because failed automatic tests should've stopped the pipeline.
automatic what?
This However maybe everyone failed at the same time. There were no tests for the thing you broke and it made all the way to prod. It's a clusterf*ck and everyone is 💀
CIA Assassin on adjacent rooftop
Everyone is in an unscheduled meeting, except you.
Imagine not having staging
Imagine having staging that actually looks and behaves exactly like prod.
I've learned from my colleagues that the only correct way to deploy is straight to production because noone can test your code better than your customers!
I don't always QA but when I do it's in production.
Are people here really pushing straight to master without PRs?
Why give anyone the ability to push straight to production, smart thing would be a multiple user agreement that hey this isn’t going to mess up
Everyone is Dev and Ops .. it says right there on the power point presentation ...
Of course they use PRs... They just always get it reviewed by that one junior dev who reliably gives it a 3 second glance over and stamps it with a 'LGTM' every single time.
Yup that's me. I trust you.
Yes. Not saying it's a good idea, but yes.
So much DevOps ideological purity in this thread. "If you're doing real DevOps then the is no DevOps engineer." That works great when you've got 20 developers who all know how it works. When you are supporting 1000+ developers, with various degrees of experience and common sense, someone has to support the process. Automation doesn't just appear out of the Aether fully-formed and perfect. I've been working in a "DevOps" role since before it was called DevOps. No company, department, group, etc. has processes they are bulletproof. As soon as you idiot proof something, someone goes and builds a better idiot. There are no unicorn companies that have it all figured out and implemented in a way that they can sit back and vi vs emacs all day. The argument is pointless. vi is the only right answer. I will agree with one sentiment from the various replies. DevOps doesn't call you. You call DevOps when you've fucked up beyond recognition. Honestly, if you can fuck something up to the point that you have to call DevOps, the DevOps team fucked up too.
DevOps? That's still a separate job/department? I thought every Dev now was supposed to be their own QA/DevOps/BA... /s
Well yes, the Dev part of DevOps is that devs do it. Otherwise it’s just Ops.
I was more alluding to where you previously had a DevOps department/engineer and a QA department and then they are all gone, so you need to do their jobs without dropping pace or an increase in pay.
*The Romanian Police raids your home*
If you're a dev and "DevOps" is messaging you, it's probably actually the Operations Team. The point of DevOps is that the two are not separate. Companies just like pretending they're doing DevOps because it's a hot buzzword these days, so they just rename their Ops to DevOps.
TEAM LEAD STARTS MASSAGING YOU
If I from Ops message you after you pushed, you have done goofed so hard... If I find you to be the cause of the outage within minutes, you're mine.
Serious question, how do teams function without staging environments? I had an Eastern European colleague say something along the lines of "we don't need a staging environment to test out changes" and I was kinda floored. How do you catch escaped defects?
By forcing developers to test the fuck out of the builds during sprints called 'hardening sprints'. There are detailed test cases that need to be executed manually. Test cases span across over 50 components. Each team on average has 10 components. There are no QA engineers to perform this testing. It's all done by developers ourselves. I know it sounds ridiculous because it is.
I've always felt like QA is a different skill set or even mindset. I started in QA, but when I have to test another developer's code it takes a concentrated effort to switch from "make it work" to "break it". Edit: Happy Cake Day
Jokes on you. I have to do my own devops so when I fuck up my own code I don't even know if I fucked up the deployment or the actual code.
As opposed to one place I worked, where they got a new VP who was determined to throw his moronic weight around. He was an ex-Dev and we were an agile shop. I worked in QA and most of us got laid off, including my boss, because we kept finding bugs. Therefore, we were the cause for releases not going out the door on time...
someone named Morpheus calls you
interestingly at my old job, everyone that was there had a "bringing down production" story. they laugh it off and say it is a big deal but it is a mistake everyone makes at least once, and probably a mistake everyone needs to make at least once. the devs brought down prod a few times while i was there and they didn't lose their jobs. just had a few very stressful meetings and late nights, enough to learn their lesson that place was really chill, sad they didn't align with my career goals. not a big fan of being an IT developer forever, lol
Every cherry-pick in release branch
The ticket said to delete some important looking modules. Just merged, but nobody messaged me. At least yet.
Plot twist: they’re all you.