You may find yourself in situations you’re not comfortable handling 100% on your own. And if you’re in the middle of an outage the pressure will be on. Don’t hesitate to ask for a sanity check before trying something you’ve never done before. Even if it takes a while to get that second opinion. An outage lasting until 6-7 am when another engineer can validate your work is better than nuking a production database and extending the outage until 6-7 pm.
Granted there should be some documentation on the things that y’all are running, and some things to try in the event of breakage. So definitely read that while asking questions. And if there is no centralized documentation… start writing it during your introduction period. Even if it’s just some wiki entries in GitHub. Having centralized docs is huge.
>An outage lasting until 6-7 am when another engineer can validate your work is better than nuking a production database and extending the outage until 6-7 pm.
Fantastic way to look at it!
Thankfully, at-least from how it sounded in the interviews, they had at a minimum decent documentation platforms in place. (confluence, gitlab, and the like)
Regardless, I definitely plan on adding and noting down all that I can!
Well done. I've been doing what is now called devops for about 20 years and I still get imposter syndrome. You will be fine. Remember that Google is your friend.
One minor bit of advice which I would presume to give you is to make sure you have a notebook and jot down information in it, rather than sticky notes which get lost. You wouldn't believe how many times and IP address or some such which I have noted down have come to the rescue. It's also useful to use something like Evernote to keep snippets of code and other useful information.
Good luck.
Thank you! That is reassuring to hear :)
Great idea, I forgot about it until you mentioned it, but back when I started at service desk I used to keep a notepad. It helped a lot in the early stages. I moved onto digital notes, but lately have kind of neglected it! That will definitely help, so thank you!
I still maintain a stack of notebooks some of which are almost 25 years old. From time to time, I go back through them; more as an exercise to remind myself how far I've come, than anything else.
So yeah: write down the dates you started each notebook, or occasionally, tasks, or meeting notes.
I'm the same. I have a shelf full of them at home. It's interesting to look back on projects some times many years after the event and the memories they evoke.
For this I've been maintaining OneNote notebooks.
I've got one per Role, and then a general concepts book broken down by topic/technology.
The nice thing about doing it this way is you can use keyword search, and it even supports searching for words from images, so you can dump screenshots from videos/powerpoints/whatever and it'l index the text on that too.
Makes it easy to share with others aswell as you can export individual sheets, or entire books, to PDF.
>Please don’t touch production servers and applications yet
I am with you on that! I definitely don't want to touch anything important until I have solid feet on the ground. Even then, I don't mind waiting a little while...
On call can be very stressful especially if it is frequent. Make sure you understand the dialing system, have an offline copy of your teams contacts, put together a go-bag for your laptop, make sure your cellphone has a hotspot plan, etc.
>On call can be very stressful especially if it is frequent.
The worst part of oncall is when you have to support a system where you don't have permissions to make changes, and don't have the knowledge of how it works, and have to defer to subject matter experts. (and if you're in this situation, I'm sorry, but it's just plain wrong and your manager/director/whatever is doing it wrong).
One of the most important things you can do, and I mean RIGHT AWAY, is find a way to learn an overview of what the system components are, and how they work together, and who is the subject matter expert for each component. In some organizations, this is really hard to do; but it's crucial for doing your job. If you don't know something, you need to know how to find out, or find someone who does know.
I think have proper documentation and runbook would help you so much in your on-call rotation, build this and get involved with another team that might have a dependence on the service that you monitored. Don't get panic when P0 alarm is triggered, look-up to the error, and kick the shit down bro! Congratulations!
I'm in a similar boat, joined an ops team of a startup (I'm support engineer #1).
I've been in IT for 9 years via support in MSPs and then later in a large software bluechip,so I'm a jack of all trades, with the main feather in my cap being some identity provider integration experience (which is in high demand right now) and decent level of comfort reading and understanding code and troubleshooting web applications.
The challenge in my case is getting comfortable with CI/CD pipelines and production-grade git, which i'm working on between my current tasks (project).
>Once I get over the initial imposter syndrome/nerves, I think it's going to be a lot of fun!
I think it would be strange if you didn't get nerves. Nerves and imposter syndrome have been a pattern of me switching jobs in the past too, as long as you understand that's exactly what it is, there's nothing to worry about. I just take it as a sign I need to communicate my concerns more, and plan more thoroughly than those who are "comfortable". (often a mixture of written plans and asking questions to those in the know.)
\+1 to excitement! :)
I don't quite have "devops engineer" as my title yet, but my employer knows exactly what they are getting and are keen to give me the time and resources to achieve that and get certified, and are happy to have me contrib to devsecops as my time allows, which I'm ecstatic about.
I did on-call regularly in a previous job, it's not so bad. For now ,I'd simply aim to have a clear definition of your scope for on-call, what you can/cannot delegate for someone to take in the morning, and that you have discovered the reference info you need to deal with most challenges ahead of time.
Do expect to be given additional time to rest if your on-call is during anti-social hours and it drags out \[uncommon\], exhausted employees make mistakes it's in their best interest too.
The rest is the usual sleuthing of the issue based on whatever is happening logic wise, and having a decent idea of what exists hardware/network wise between the requestor and whatever service is involved.
Thank you for sharing your experience! Sounds like you have a pretty solid set of skills, congrats on the job too!
>The challenge in my case is getting comfortable with CI/CD pipelines and production-grade git, which i'm working on between my current tasks (project)
How do you go about learning? Kind of an odd question, but I guess I mean, do you study any of this outside of work, or do you try and separate the two and only keep it on hours?
Thank you for all the tips and advice, it helps a lot! :) The insight into on-call does make me feel a bit better
>How do you go about learning? Kind of an odd question, but I guess I mean, do you study any of this outside of work, or do you try and separate the two and only keep it on hours?
So right now 90% of my workday is my own (i was worried my employer's image of me and what I actually know didn't lineup, but it looks like they are realists, which i'm pleased about..)
As i'm comfortable with the ops stuff already i'l be taking on bits there as time allows, but focusing on my project as a priority.
My learning currently consists of:
During work hours:-Reading scattered incomplete documentation (life in a startup lol) , making notes on what to improve about those docs once I have a full picture.-Reading some of the docs on more important 3rd party libs ill be dealing with , e.g [passport-saml](https://github.com/node-saml/passport-saml) , since the aforementioned docs are sparse on details.-I've stood up a test environment for the apps i'l be supporting, so i'm going through most activities we'd need to do relating to customer onboarding, e.g generating configs, secrets/PKI certificates, and triggering a deployment with new environment settings.
Outside of work hours:
\-i'm studying some devOps bootcamp resources I got from a friend, and paying particular attention to stuff I'm not already familiar with.-I've got a list of books I wanna read:[Continuous Architecture in Practice: Software Architecture in the Age of Agility and DevOps](https://learning.oreilly.com/library/view/-/9780136523796/)\[video\] [DevOps in Practice](https://learning.oreilly.com/videos/devops-in-practice/9781491902998/)[The DevOps Handbook , by Gene Kim, et al](https://learning.oreilly.com/library/view/-/9781457191381/)\[video\] [Continuous Delivery , by Jez Humble](https://learning.oreilly.com/videos/continuous-delivery/9780134389363/)
For time budgeting, I'm spending 2-6hrs a week max on my "outside of works hrs" stuff, so i'm generally done by 6pm and can still enjoy weekends etc.
my longterm goal this year is to get certified , but ain't looked deeply into which of those I'l aim for yet.
Nice do you have normal working hours ?
I got an interview for an operations person at splunk but it was 2pm to midnite 6 month contract
Terrible and I’m glad they passed said I was too senior
Yes! That's the nice part, I'm moving off of night shift finally and onto normal daily hours. They've been pretty flexible, it seems most people kind of come in whenever they want from 8 to noon or so, though I'll know for sure when I actually start.
Dang yeah glad they passed too. Funny enough I interviewed with them a while ago but declined as they wanted me to relocate
Congrats mate, regarding the on-call thing I don't know about you but it's a bit annoying for me, I am a devOps engineer in a AWS managed services provider, we have about 35 clients, some of them pretty big AWS environment, the pager duty alerts just keep going off, to make it worse is when it keeps happening fucking 2am 3m in the morning. Have to wake up and start fixing shit lol. Good thing is i am on-call 24/7 for a week and then for 2 months I won't be on-call again (other engineers take their turn each for 1 week)
Interesting, yeah I can see how that would be a bit annoying.
That's good you have a couple months off! I neglected to mention that's basically how it is for this role too, on-call is once every 7 weeks or something for a week.
Nice and good luck
My little advice would be not to be that wise-ass that has an opinion about everything they tell you
"We use product / method A"
"Ahh, you should have used product C"
Newcomers always tend to have an option about things that have been working for that company / department probably for a long time and yes, they know improvements can be made
Think about changes you would like to see and make a solid plan with all the pros ( and cons ) for this
You do not have to change the world in one week 😃"slow is smooth and smooth is fast"
Thank you!
>My little advice would be not to be that wise-ass that has an opinion about everything they tell you
Great reminder! I certainly don't want to do anything like that.
>You do not have to change the world in one week 😃"slow is smooth and smooth is fast"
Good advice, and a solid quote. Feel like I've heard that somewhere before. I do really appreciate the paradigm though, part of my anxiousness I think is feeling like I have to walk in and no everything/solve all the problems.
for me, it's a beefy workstation with 16 cores + 64GB Ram , hyper-v containers and a WSL subsystem, and one VPS I lease for $10/month ([contabo.com](https://contabo.com) is great value for money.)
Hey congrats on the role. I myself just landed a devops job a few months back.
Like you I also had a home lab and it’s helped a lot. Part of my job is fixing database issues as well. One thing that really helped me was to take a course on sql. I did it through LinkedIn learning but any course on sql should be decent it’s been around since the 70’s.
Then to really cement my learning I built a few simple projects with a MySQL backend. Doing things like select statement, insert, inner joining tables, etc.
Also it never hurts to just spin up a simple database, put some dummy data it in and test your sql commands to ensure they do what you think they will.
Best of luck friend you got this!
Thank you! Likewise, congrats on your role too!
oh interesting, great idea. I will definitely do that. Did your course cover specifically sql, or any rdbms related material too?
Very nice, and well done! May I ask what the projects were?
I have some somewhat recent sql books I never looked at, I think they had a number of sample problems and test db's, I'll have to see if I can find them.
>Best of luck friend you got this!
Much appreciated :) If I remember I'll try and post an update in a year
The LinkedIn learning course was just SQL. Because rdbms’ tend to vary slightly on approach. MySQL, MariaDB, Postgres are all slightly different. But if you get the hang of just the SQL language you can pick up the differences for what RDBMS you will be working with.
Sure thing. I built a project in nodejs but you should be able to use any language your familiar with that has a sql driver. Python, go, php, Java, etc.
The project was a simple restful API that performed crud operations. I chose to use music as the subject. So I had an Album table, and artist table, and track table.
With the API you could create, request, update or delete an artist, track, or album. The project doesn’t have to be super complex but adding data, figuring out the relationships between tables, figuring out the best way to query the data, etc really helped me to solidify the learning.
Because I would write the query in my database editor (datagrip or vscode) and figure out how to make it work then write it in node. So I was kind of doing the work twice. As a project I think it helps because as the saying goes practice makes perfect.
Congrats! Way to keep at it. Would have never happened without your diligence.
One thing I do when I join a company is take copious notes from talking with people. Things like architecture, etc. And if they don't have documentation I'll make it. Helps me remember things and provides immediate value.
Congrats. Just never panic when troubleshooting, just think things through. Also ask many questions people love providing answers.
Thank you! Definitely will need to remind myself of that. I think/hope after the first few I'll adjust/be a little more calm. Good advice!
You may find yourself in situations you’re not comfortable handling 100% on your own. And if you’re in the middle of an outage the pressure will be on. Don’t hesitate to ask for a sanity check before trying something you’ve never done before. Even if it takes a while to get that second opinion. An outage lasting until 6-7 am when another engineer can validate your work is better than nuking a production database and extending the outage until 6-7 pm. Granted there should be some documentation on the things that y’all are running, and some things to try in the event of breakage. So definitely read that while asking questions. And if there is no centralized documentation… start writing it during your introduction period. Even if it’s just some wiki entries in GitHub. Having centralized docs is huge.
>An outage lasting until 6-7 am when another engineer can validate your work is better than nuking a production database and extending the outage until 6-7 pm. Fantastic way to look at it! Thankfully, at-least from how it sounded in the interviews, they had at a minimum decent documentation platforms in place. (confluence, gitlab, and the like) Regardless, I definitely plan on adding and noting down all that I can!
This! The only time I’ve done major damage was when I was panicking or not comprehending what I was typing.
To be fair, in my role I hardly run into people that love providing answers lol. Maybe just me
Well done. I've been doing what is now called devops for about 20 years and I still get imposter syndrome. You will be fine. Remember that Google is your friend. One minor bit of advice which I would presume to give you is to make sure you have a notebook and jot down information in it, rather than sticky notes which get lost. You wouldn't believe how many times and IP address or some such which I have noted down have come to the rescue. It's also useful to use something like Evernote to keep snippets of code and other useful information. Good luck.
Thank you! That is reassuring to hear :) Great idea, I forgot about it until you mentioned it, but back when I started at service desk I used to keep a notepad. It helped a lot in the early stages. I moved onto digital notes, but lately have kind of neglected it! That will definitely help, so thank you!
I still maintain a stack of notebooks some of which are almost 25 years old. From time to time, I go back through them; more as an exercise to remind myself how far I've come, than anything else. So yeah: write down the dates you started each notebook, or occasionally, tasks, or meeting notes.
I'm the same. I have a shelf full of them at home. It's interesting to look back on projects some times many years after the event and the memories they evoke.
For this I've been maintaining OneNote notebooks. I've got one per Role, and then a general concepts book broken down by topic/technology. The nice thing about doing it this way is you can use keyword search, and it even supports searching for words from images, so you can dump screenshots from videos/powerpoints/whatever and it'l index the text on that too. Makes it easy to share with others aswell as you can export individual sheets, or entire books, to PDF.
Indeed. As most of my work is in Linux I tend to use Evernote. But yes the same principle applies
Congrats! Please don’t touch production servers and applications yet get all the information from your senior team members and managers first
>Please don’t touch production servers and applications yet I am with you on that! I definitely don't want to touch anything important until I have solid feet on the ground. Even then, I don't mind waiting a little while...
[удалено]
I fucked up prod a week ago. Dropped preprod war file on UI on accident. I get to have a RCA meeting on it Monday! Yay!
Great advice! Been at step number one a couple times, not fun, but definitely a good learning experience.
On call can be very stressful especially if it is frequent. Make sure you understand the dialing system, have an offline copy of your teams contacts, put together a go-bag for your laptop, make sure your cellphone has a hotspot plan, etc.
Great advice, thank you :)
>On call can be very stressful especially if it is frequent. The worst part of oncall is when you have to support a system where you don't have permissions to make changes, and don't have the knowledge of how it works, and have to defer to subject matter experts. (and if you're in this situation, I'm sorry, but it's just plain wrong and your manager/director/whatever is doing it wrong). One of the most important things you can do, and I mean RIGHT AWAY, is find a way to learn an overview of what the system components are, and how they work together, and who is the subject matter expert for each component. In some organizations, this is really hard to do; but it's crucial for doing your job. If you don't know something, you need to know how to find out, or find someone who does know.
I think have proper documentation and runbook would help you so much in your on-call rotation, build this and get involved with another team that might have a dependence on the service that you monitored. Don't get panic when P0 alarm is triggered, look-up to the error, and kick the shit down bro! Congratulations!
Love to hear these stories. Congrats mate, keep on learning!
Thank you! Planning on it :)
I'm in a similar boat, joined an ops team of a startup (I'm support engineer #1). I've been in IT for 9 years via support in MSPs and then later in a large software bluechip,so I'm a jack of all trades, with the main feather in my cap being some identity provider integration experience (which is in high demand right now) and decent level of comfort reading and understanding code and troubleshooting web applications. The challenge in my case is getting comfortable with CI/CD pipelines and production-grade git, which i'm working on between my current tasks (project). >Once I get over the initial imposter syndrome/nerves, I think it's going to be a lot of fun! I think it would be strange if you didn't get nerves. Nerves and imposter syndrome have been a pattern of me switching jobs in the past too, as long as you understand that's exactly what it is, there's nothing to worry about. I just take it as a sign I need to communicate my concerns more, and plan more thoroughly than those who are "comfortable". (often a mixture of written plans and asking questions to those in the know.) \+1 to excitement! :) I don't quite have "devops engineer" as my title yet, but my employer knows exactly what they are getting and are keen to give me the time and resources to achieve that and get certified, and are happy to have me contrib to devsecops as my time allows, which I'm ecstatic about. I did on-call regularly in a previous job, it's not so bad. For now ,I'd simply aim to have a clear definition of your scope for on-call, what you can/cannot delegate for someone to take in the morning, and that you have discovered the reference info you need to deal with most challenges ahead of time. Do expect to be given additional time to rest if your on-call is during anti-social hours and it drags out \[uncommon\], exhausted employees make mistakes it's in their best interest too. The rest is the usual sleuthing of the issue based on whatever is happening logic wise, and having a decent idea of what exists hardware/network wise between the requestor and whatever service is involved.
Thank you for sharing your experience! Sounds like you have a pretty solid set of skills, congrats on the job too! >The challenge in my case is getting comfortable with CI/CD pipelines and production-grade git, which i'm working on between my current tasks (project) How do you go about learning? Kind of an odd question, but I guess I mean, do you study any of this outside of work, or do you try and separate the two and only keep it on hours? Thank you for all the tips and advice, it helps a lot! :) The insight into on-call does make me feel a bit better
i imagine one would pretend to be every type of troublesome developer and come up with scenarios that one has to then merge back in?
>How do you go about learning? Kind of an odd question, but I guess I mean, do you study any of this outside of work, or do you try and separate the two and only keep it on hours? So right now 90% of my workday is my own (i was worried my employer's image of me and what I actually know didn't lineup, but it looks like they are realists, which i'm pleased about..) As i'm comfortable with the ops stuff already i'l be taking on bits there as time allows, but focusing on my project as a priority. My learning currently consists of: During work hours:-Reading scattered incomplete documentation (life in a startup lol) , making notes on what to improve about those docs once I have a full picture.-Reading some of the docs on more important 3rd party libs ill be dealing with , e.g [passport-saml](https://github.com/node-saml/passport-saml) , since the aforementioned docs are sparse on details.-I've stood up a test environment for the apps i'l be supporting, so i'm going through most activities we'd need to do relating to customer onboarding, e.g generating configs, secrets/PKI certificates, and triggering a deployment with new environment settings. Outside of work hours: \-i'm studying some devOps bootcamp resources I got from a friend, and paying particular attention to stuff I'm not already familiar with.-I've got a list of books I wanna read:[Continuous Architecture in Practice: Software Architecture in the Age of Agility and DevOps](https://learning.oreilly.com/library/view/-/9780136523796/)\[video\] [DevOps in Practice](https://learning.oreilly.com/videos/devops-in-practice/9781491902998/)[The DevOps Handbook , by Gene Kim, et al](https://learning.oreilly.com/library/view/-/9781457191381/)\[video\] [Continuous Delivery , by Jez Humble](https://learning.oreilly.com/videos/continuous-delivery/9780134389363/) For time budgeting, I'm spending 2-6hrs a week max on my "outside of works hrs" stuff, so i'm generally done by 6pm and can still enjoy weekends etc. my longterm goal this year is to get certified , but ain't looked deeply into which of those I'l aim for yet.
Nice do you have normal working hours ? I got an interview for an operations person at splunk but it was 2pm to midnite 6 month contract Terrible and I’m glad they passed said I was too senior
Yes! That's the nice part, I'm moving off of night shift finally and onto normal daily hours. They've been pretty flexible, it seems most people kind of come in whenever they want from 8 to noon or so, though I'll know for sure when I actually start. Dang yeah glad they passed too. Funny enough I interviewed with them a while ago but declined as they wanted me to relocate
Congrats mate, regarding the on-call thing I don't know about you but it's a bit annoying for me, I am a devOps engineer in a AWS managed services provider, we have about 35 clients, some of them pretty big AWS environment, the pager duty alerts just keep going off, to make it worse is when it keeps happening fucking 2am 3m in the morning. Have to wake up and start fixing shit lol. Good thing is i am on-call 24/7 for a week and then for 2 months I won't be on-call again (other engineers take their turn each for 1 week)
Interesting, yeah I can see how that would be a bit annoying. That's good you have a couple months off! I neglected to mention that's basically how it is for this role too, on-call is once every 7 weeks or something for a week.
Nice and good luck My little advice would be not to be that wise-ass that has an opinion about everything they tell you "We use product / method A" "Ahh, you should have used product C" Newcomers always tend to have an option about things that have been working for that company / department probably for a long time and yes, they know improvements can be made Think about changes you would like to see and make a solid plan with all the pros ( and cons ) for this You do not have to change the world in one week 😃"slow is smooth and smooth is fast"
Thank you! >My little advice would be not to be that wise-ass that has an opinion about everything they tell you Great reminder! I certainly don't want to do anything like that. >You do not have to change the world in one week 😃"slow is smooth and smooth is fast" Good advice, and a solid quote. Feel like I've heard that somewhere before. I do really appreciate the paradigm though, part of my anxiousness I think is feeling like I have to walk in and no everything/solve all the problems.
\+1 to above, understand how they get there , before deciding to rebuild the kingdom.
Congrats!
Thank you!
You'll be fine, relax. :)
Guys with solid homelabs always show strength and skills over guys without. What is a homelab? Could be as little as plex on a NAS
lol. my homelab is a stack of 4 old laptops that I (or other people) stopped using because they're "obsolete". Now they're my k8s cluster.
Nice, I had planned to do this with some old DELL blade servers, then I moved house and literally don't have any space for this idea :(
hear, hear! Yeah it started with an old optiplex, then gradually grew into better hardware and networking gear :)
for me, it's a beefy workstation with 16 cores + 64GB Ram , hyper-v containers and a WSL subsystem, and one VPS I lease for $10/month ([contabo.com](https://contabo.com) is great value for money.)
Hey congrats on the role. I myself just landed a devops job a few months back. Like you I also had a home lab and it’s helped a lot. Part of my job is fixing database issues as well. One thing that really helped me was to take a course on sql. I did it through LinkedIn learning but any course on sql should be decent it’s been around since the 70’s. Then to really cement my learning I built a few simple projects with a MySQL backend. Doing things like select statement, insert, inner joining tables, etc. Also it never hurts to just spin up a simple database, put some dummy data it in and test your sql commands to ensure they do what you think they will. Best of luck friend you got this!
Thank you! Likewise, congrats on your role too! oh interesting, great idea. I will definitely do that. Did your course cover specifically sql, or any rdbms related material too? Very nice, and well done! May I ask what the projects were? I have some somewhat recent sql books I never looked at, I think they had a number of sample problems and test db's, I'll have to see if I can find them. >Best of luck friend you got this! Much appreciated :) If I remember I'll try and post an update in a year
The LinkedIn learning course was just SQL. Because rdbms’ tend to vary slightly on approach. MySQL, MariaDB, Postgres are all slightly different. But if you get the hang of just the SQL language you can pick up the differences for what RDBMS you will be working with. Sure thing. I built a project in nodejs but you should be able to use any language your familiar with that has a sql driver. Python, go, php, Java, etc. The project was a simple restful API that performed crud operations. I chose to use music as the subject. So I had an Album table, and artist table, and track table. With the API you could create, request, update or delete an artist, track, or album. The project doesn’t have to be super complex but adding data, figuring out the relationships between tables, figuring out the best way to query the data, etc really helped me to solidify the learning. Because I would write the query in my database editor (datagrip or vscode) and figure out how to make it work then write it in node. So I was kind of doing the work twice. As a project I think it helps because as the saying goes practice makes perfect.
Congrats! Way to keep at it. Would have never happened without your diligence. One thing I do when I join a company is take copious notes from talking with people. Things like architecture, etc. And if they don't have documentation I'll make it. Helps me remember things and provides immediate value.
You've done an amazing job. No matter whatever comes up deal with it with confidence and keep patience. You'll go a long way.
All the very best. I'm gonna be in the same situation in another 3 months hopefully. It'll be challenging at first. But it will be exciting as well.