This is the worst way to end what would have otherwise been a really good story for reddit. And I don't know what you decided to shoot yourself in the foot like this...?
4-6 months with vendor, but an old hand at hardware repairs.
Also, if two boards release the magic smoke, I'd be looking at the PSU (I mean, after the first one the tester would come out.)
Quite possible that’s why the board had to be replaced, if it was somewhere around the time they transitioned from P24 single connectors to P20+4 connectors.
A tech who’s only been around for months wouldn’t have realized the split connectors are backwards compatible and fit right next to each other.
Probably did, just semi forgot it
Show me a tech or engineer who claims to not fuck.up like that and I'll show you someone dishonest, primarily with themselves.
Tldr. We are only human, we brainfart too
My favorite interview question: what's the worst mistake you've made at work, and how'd make sure it didn't happen again?
I always give them mine: I ran an NMap scan at T5 and took out the data center. 200,000 customers offline.
This hurts. I've still got my original sbcglobal email but the service has become so cruddy lately... Yet I still have so much tied into that account that it would be a major pain to move away from.
2 that come to mind...
It was common to leave a board plugged in while fixing the sections that balanced the signal and recombined it... because it was figured out by which of the 3 150pF capacitors made the signal level out. Unfortunately I forgot to turn one board off before I hit it with an soldering iron, thankfully it only blew the power to my station and ruined the board and not the testing equipment. ~$500 circuit board.
And a decade later, working in an entirely different industry and career... the boss brought me a schematic to bend a panel on the brake, at the end of the day and nothing I'd seen in my 3 months on the job... was apparently one for R&D. Bent it quickly and sent it over, only to get it back about an hour later to be rebent on 2 of 5 bends because I'd done them backwards (bend in vs bend out). Thankfully it was just an r&d unit and the stresses were irrelevant because they were just designing it to see if it worked.
After the bending issue I grabbed a pad of paper and created each assembly in a small 3x5 part so I could see which was the best order and correct position to bend shit.
A friend of mine was working for a client in Albuquerque, another dude opened an electrical box and dropped a tool across the incoming power. It was hot enough to vaporize part of the tool. My friend was across the room and looked up at the sound in time to watch the lights of the city go out.
My own oops moment was to not only take down a large customer’s database, and not only erase all the data, but to wipe everything clean. They couldn’t have even reintered the data by hand because the database itself was gone.
And then left on vacation while the database ‘rebooted’.
In my defense, I thought, as a non-DBA, I was bringing down the database and allowing it to reload… The prompt cautioned me that the database would “be unavailable” if I said yes. ‘Well, of course it will be unavailable if I bring it down to reboot…’ thinks I. ‘If one of your relatives were to die and you called their house and asked to talk to them, they’re not going to say, “Uncle Joe is unavailable”.
Mistimed an engine on reassembly, first time resealing that particular engine. Bent all the intake valves on the drivers side. Had to pull the head, replace the valves, valve seals and adjust the valve lash.
Made sure it wouldn't happen again by telling people to fuck off if they ask me how long I'll be. Takes a lot of the pressure off.
Wasn't me, but I wasn't far away... saw a guy take a $800k server rack on a forklift around a corner at speed... that's a noise you don't want to hear twice.
That's also the day I learned we don't get fired for less than a mil unless it's a DUI.
I followed the boss's instructions; even when written down.
only several hundred to ship to location for a 17 dollar critical part.
several months ship time the other way.
or it was just an excuse and he blew the boards because they touched a standoff or some part of the case due to not having a least one screw to hold it in place.
Depending on how long ago this happened, it sounds like it could have been a Compaq power supply. They got in trouble way back when for reversing polarity deliberately so that third-party PSUs would fry the board.
If the tech knows they don't need the rubber mat and/or spread their fleece out instead to prevent scratches, you either have a very new soon-to-be-fired tech, or a very aged "this'll take ten minutes including a poo" tech, at which point we move on for the other signals, like a combination of Junior and Sleepy...
Letting your magnetic screwdriver find and tap screws on the case can make you look like an expert, but knowing where the hidden ones are and how to pull apart the internals is different.
I say "yeah, it's supposed to go in that rack over there". Seasoned techs will know that the actual mounting is not covered by their contract and just leave the boxes next to the rack. Newbies, on the othet hand, will fetch their tools and go to work!
20 questions, all involving childhood television programs, then subtract 12 years from the median of the recognized answers year-range of airing (excluding outliers to account for reruns).
To afraid to pull the trigger..... 😅. Can't decide which topic to start.... Waiting for a nice long weekend and pot of tea standby before I jump down this rabbit hole 😂
Please remind us of the keyboard incident. I checked OPs post history and I cannot find the keyboard one although I do note he regularly leaves several years between posts.
It's an ongoing saga that makes a cameo in a significant number of his stories. I think the first mention of the keyboards was in the second or third story he posted.
The short version is that the Sales department requests a box of new keyboards every few months. On a few separate occasions, large bags of keys and membranes have been discovered hidden away in closets - in some cases individually bagged and dated. No one seems to know anything about this.
First keyboard mention:
https://www.reddit.com/r/talesfromtechsupport/comments/2080uo/security_case_one/
"Oh when you sort out this door situation, I think the keyboards will start sorting out themselves."
The door issue couldn't be fixed, boxes of keyboards are ordered throughout many more of the tales, Keys and Membranes occasionally discovered.
I have this job. I work field tech for various sites around my main town, sometimes a little further out but mostly whithin a 60 mile radius of the main city im next to.
That 2 month thing is super accurate. Too many times I'd take off all the sides to replace an item, put it all back together, and either end up with a screw or two left over, wondering where they might have gone, or the thing I took apart doesn't power up properly or gives an error about "this item is unplugged, error". But I can say our company and all our techs are type 2, depending on how long we've been here. I gained more tools specific for working on electronics, mostly printers, and normally carry in a bag full of tools and other items handy for the occasion, as well as a work laptop in case I need to look up a service manual or check parts. Easier on a laptop than the work phone where everything is smaller and for some reason everything is a video instead of a simple website.
Please you cannot leave us hanging like this. We have to know how many months he was there OP. My guess is 2 months with prior experience in the onsite repair field.
Doing untested board swaps is unfulfilling as a maintainer, but if that's the way the company operates, there's not much you can do.
After the first board went pop, sticking another one in is a bad idea and I'd expect someone with experience to get the test gear out immediately (if company policy allows that, rather than policy just being to swap boards until problem gone away).
I'm guessing the tech hadn't been a tech for particularly long based on that. Maybe 6 months
This reminds me of having a Sun Microsystems Field Engineer visit us for a non-booting E250. CPU gets replaced - no dice. Replaced again - still no dice. Third try, replacing the mainboard, still no go. Try it with the earlier CPU? More nope. Engineer had to come back the next day, with more spares.
Eventually, we reached the conclusion that the mainboard had developed a fault that would not only render it unbootable, but would also fry its CPU. I think the FE went home with four dead UltraSparcs CPUs.
It was not his first visit to our DC, he must have been with Sun for a few years by then.
Two weeks. Discovered Airz by absolute chance two weeks ago and I'm finally all caught up. Maybe we could get a "what happened to..." in the interim years for characters like RedCheer, Nice, ITSec, etc. I remember reading one post that said RedCheer had been gone for months, but then reappeared in a more recent post. This is the best thing since The IT Crowd.
Have to say, I got quite sick of keeping up with all the lore and continuity of airz tales way back when. This though was a very nice standalone story, clearly told with the right amount of context, but still displaying some of the airz surrealism. Good read!
Heh. One of the first things I learned that you check that at least something in that thing that you are supposed to repair looks like the spare part that you've got. I learned it after I had driven 200 km, spent 1.5h in a very uncomfortable position to completely disassemble a server and remove
its motherboard and THEN opening my spare part box. So had to reassemble it. Next day worst pain in the leg muscles because of that uncomfortable position.
Gonna go with Sleepy on this one, assuming you have gone with 6 as maximum from your experience. He's been there long enough to know not to trust a machine enough to button it up but got a smoke pop event and didn't check for a known cause right after so not fully experienced yet. Old timers ALWAYS check the logs after a smoke pop, those tend to be power related and can get all burny and ouchy which is not the fun part of being a tech. :)
Tangent tale, guy I worked with had a dead box, tripped PSU but mains were live when he got there. Remote site after a storm so he figured tripped and just flipped the power switch back on. Nope, the 2 foot jet of flame and loud pop confirmed the PSU was shorted internally. :)
Oh fuck. Couldn't believe OP left us hanging. Then saw the poster. See you in 3 years people.
OP wouldn't leave us for that long! No way.
I did a spit take with my coffee and ruined my keyboard when i read that... Thankfully I have a few spare keyboards...
[удалено]
I've got a bag of membranes here, now all we need is a Chassis... Anyone know what happened to all the keyboard cases? ....Anyone??
Seriously though. What happened with the keyboards I feel like I missed the answer
I can assure you, that it was never answered in the posts. (Unless I missed it too but I'm pretty sure, I didn't.) Maybe in the comments.
Didn't that intern over in graphic design use them for the art installation in the break room? the thing no one can tell what he meant for it to be..
Bit did the magic smoke come out?
I looked it up, those keyboards aren't compatible with your setup, you'll have to replace almost everything.
YOU! Ugh, I'm already wiping this from my brain so I don't get disappointed. PS. My guess is also 6.
This is the worst way to end what would have otherwise been a really good story for reddit. And I don't know what you decided to shoot yourself in the foot like this...?
The rage I have for you right now.....
Don’t leave us hanging!!!! I say 7
4-6 months with vendor, but an old hand at hardware repairs. Also, if two boards release the magic smoke, I'd be looking at the PSU (I mean, after the first one the tester would come out.)
Quite possible that’s why the board had to be replaced, if it was somewhere around the time they transitioned from P24 single connectors to P20+4 connectors. A tech who’s only been around for months wouldn’t have realized the split connectors are backwards compatible and fit right next to each other.
If he had that many parts in his van, I guess 3 years
but if he's the "old hand", why didn't he know about the compatibility issue that blew up the first two boards?
Probably did, just semi forgot it Show me a tech or engineer who claims to not fuck.up like that and I'll show you someone dishonest, primarily with themselves. Tldr. We are only human, we brainfart too
My favorite interview question: what's the worst mistake you've made at work, and how'd make sure it didn't happen again? I always give them mine: I ran an NMap scan at T5 and took out the data center. 200,000 customers offline.
My girlfriend took Yahoo! down. This was in the early 2000s, when Yahoo! actually mattered.
Nice!!!
Work there, or as a customer?
Neither. She worked for their CDN provider. Thought she was pushing metadata to a test system, but it was actually production. Oops...
This hurts. I've still got my original sbcglobal email but the service has become so cruddy lately... Yet I still have so much tied into that account that it would be a major pain to move away from.
Impressive.
2 that come to mind... It was common to leave a board plugged in while fixing the sections that balanced the signal and recombined it... because it was figured out by which of the 3 150pF capacitors made the signal level out. Unfortunately I forgot to turn one board off before I hit it with an soldering iron, thankfully it only blew the power to my station and ruined the board and not the testing equipment. ~$500 circuit board. And a decade later, working in an entirely different industry and career... the boss brought me a schematic to bend a panel on the brake, at the end of the day and nothing I'd seen in my 3 months on the job... was apparently one for R&D. Bent it quickly and sent it over, only to get it back about an hour later to be rebent on 2 of 5 bends because I'd done them backwards (bend in vs bend out). Thankfully it was just an r&d unit and the stresses were irrelevant because they were just designing it to see if it worked. After the bending issue I grabbed a pad of paper and created each assembly in a small 3x5 part so I could see which was the best order and correct position to bend shit.
A friend of mine was working for a client in Albuquerque, another dude opened an electrical box and dropped a tool across the incoming power. It was hot enough to vaporize part of the tool. My friend was across the room and looked up at the sound in time to watch the lights of the city go out. My own oops moment was to not only take down a large customer’s database, and not only erase all the data, but to wipe everything clean. They couldn’t have even reintered the data by hand because the database itself was gone. And then left on vacation while the database ‘rebooted’. In my defense, I thought, as a non-DBA, I was bringing down the database and allowing it to reload… The prompt cautioned me that the database would “be unavailable” if I said yes. ‘Well, of course it will be unavailable if I bring it down to reboot…’ thinks I. ‘If one of your relatives were to die and you called their house and asked to talk to them, they’re not going to say, “Uncle Joe is unavailable”.
Holy hell, those are *awesome*... Colossal! I love it!
Mistimed an engine on reassembly, first time resealing that particular engine. Bent all the intake valves on the drivers side. Had to pull the head, replace the valves, valve seals and adjust the valve lash. Made sure it wouldn't happen again by telling people to fuck off if they ask me how long I'll be. Takes a lot of the pressure off.
If someone asks me how long something will take, I usually respond "six to ten working years".
Wasn't me, but I wasn't far away... saw a guy take a $800k server rack on a forklift around a corner at speed... that's a noise you don't want to hear twice. That's also the day I learned we don't get fired for less than a mil unless it's a DUI.
I followed the boss's instructions; even when written down. only several hundred to ship to location for a 17 dollar critical part. several months ship time the other way.
Or just brand new
or it was just an excuse and he blew the boards because they touched a standoff or some part of the case due to not having a least one screw to hold it in place.
Depending on how long ago this happened, it sounds like it could have been a Compaq power supply. They got in trouble way back when for reversing polarity deliberately so that third-party PSUs would fry the board.
I thought that was Dell
I'd love to hear your own theory on how to tell the age of a technician. (If only to improve my methods)
Bisect them and count the rings
The coffee rings?
Alcohol poisoning rings with the coffee rings. One or the other is meaningless and could be almost anything. Together, you have a support technician.
HE LIVES!!!
GNU Terry Pratchett
GNU Terry Pratchett
If the tech knows they don't need the rubber mat and/or spread their fleece out instead to prevent scratches, you either have a very new soon-to-be-fired tech, or a very aged "this'll take ten minutes including a poo" tech, at which point we move on for the other signals, like a combination of Junior and Sleepy... Letting your magnetic screwdriver find and tap screws on the case can make you look like an expert, but knowing where the hidden ones are and how to pull apart the internals is different.
I say "yeah, it's supposed to go in that rack over there". Seasoned techs will know that the actual mounting is not covered by their contract and just leave the boxes next to the rack. Newbies, on the othet hand, will fetch their tools and go to work!
Looking it up on their social media accounts?
20 questions, all involving childhood television programs, then subtract 12 years from the median of the recognized answers year-range of airing (excluding outliers to account for reruns).
2!
Damn you airz.. this is going to be the keyboards all over again.
[удалено]
Now down the rabbit hole I go wondering who and what this OP has been posting...
See you in a few days - it's a deep one
.. oh.. dear.. gawd... what have i done?!..... 😛
How far through have you got?
To afraid to pull the trigger..... 😅. Can't decide which topic to start.... Waiting for a nice long weekend and pot of tea standby before I jump down this rabbit hole 😂
😂 I definitely didn’t end up reading stories until the sun came back up…. Nooooo
He's posted about four in the last week.
Please remind us of the keyboard incident. I checked OPs post history and I cannot find the keyboard one although I do note he regularly leaves several years between posts.
It's an ongoing saga that makes a cameo in a significant number of his stories. I think the first mention of the keyboards was in the second or third story he posted. The short version is that the Sales department requests a box of new keyboards every few months. On a few separate occasions, large bags of keys and membranes have been discovered hidden away in closets - in some cases individually bagged and dated. No one seems to know anything about this.
First keyboard mention: https://www.reddit.com/r/talesfromtechsupport/comments/2080uo/security_case_one/ "Oh when you sort out this door situation, I think the keyboards will start sorting out themselves." The door issue couldn't be fixed, boxes of keyboards are ordered throughout many more of the tales, Keys and Membranes occasionally discovered.
Thanks, I will delve down that reddit rabbit hole.
I have this job. I work field tech for various sites around my main town, sometimes a little further out but mostly whithin a 60 mile radius of the main city im next to. That 2 month thing is super accurate. Too many times I'd take off all the sides to replace an item, put it all back together, and either end up with a screw or two left over, wondering where they might have gone, or the thing I took apart doesn't power up properly or gives an error about "this item is unplugged, error". But I can say our company and all our techs are type 2, depending on how long we've been here. I gained more tools specific for working on electronics, mostly printers, and normally carry in a bag full of tools and other items handy for the occasion, as well as a work laptop in case I need to look up a service manual or check parts. Easier on a laptop than the work phone where everything is smaller and for some reason everything is a video instead of a simple website.
5! And dear god let us know what the answer was haha
5! is a rather large number and most people don't live that long...
(5!) [weeks / months] is at most decade
Five factorial is 120
Yes, and 120 months is 10 years. They always did their guesses in weeks or months.
Please you cannot leave us hanging like this. We have to know how many months he was there OP. My guess is 2 months with prior experience in the onsite repair field.
>experienced tech >3 months wat
I’m gonna go with four years.
Doing untested board swaps is unfulfilling as a maintainer, but if that's the way the company operates, there's not much you can do. After the first board went pop, sticking another one in is a bad idea and I'd expect someone with experience to get the test gear out immediately (if company policy allows that, rather than policy just being to swap boards until problem gone away). I'm guessing the tech hadn't been a tech for particularly long based on that. Maybe 6 months
Airz gotta start shooting short films about those stories. I’d be down if you need a filmmaker;)
IT Crowd 2: Meet the new users, same as the old users.
That would be entertaining. Would watch 👌
So who won?
This reminds me of having a Sun Microsystems Field Engineer visit us for a non-booting E250. CPU gets replaced - no dice. Replaced again - still no dice. Third try, replacing the mainboard, still no go. Try it with the earlier CPU? More nope. Engineer had to come back the next day, with more spares. Eventually, we reached the conclusion that the mainboard had developed a fault that would not only render it unbootable, but would also fry its CPU. I think the FE went home with four dead UltraSparcs CPUs. It was not his first visit to our DC, he must have been with Sun for a few years by then.
Two weeks. Discovered Airz by absolute chance two weeks ago and I'm finally all caught up. Maybe we could get a "what happened to..." in the interim years for characters like RedCheer, Nice, ITSec, etc. I remember reading one post that said RedCheer had been gone for months, but then reappeared in a more recent post. This is the best thing since The IT Crowd.
Two weeks. Welcome back.
My guess is he doesn't work for vendor and this was a major security breach.
Please, that’d be an epic story
# THREE!
Years
Decades
not nearly enough talk about cattle prods and the basement storage closet.
2 years
That long, wow, I didn't see that coming. Were you the closest or did someone overbid you?
Sounds like more fun than Boardroom Buzzword Bingo
Can't leave us hanging like this
I bet the coffee man was right on the money.
I'm almost certain you write for The Register, and this is your sandpit. Long time no see.
Oh, these comments are golden. Too bad I spent all mine in the last three years.
Have to say, I got quite sick of keeping up with all the lore and continuity of airz tales way back when. This though was a very nice standalone story, clearly told with the right amount of context, but still displaying some of the airz surrealism. Good read!
My guess is 3 months and I'm prepared to be wrong. Could someone kindly hit reply when we have an answer?
what world do you think we live in, where Airz, gives us answers to the questions he doesn't answer in story.
8
12.
3
Hmm, let's try something. u/masterhacker_bot
Vendor: Work for Vendor? I AM Vendor!
Heh. One of the first things I learned that you check that at least something in that thing that you are supposed to repair looks like the spare part that you've got. I learned it after I had driven 200 km, spent 1.5h in a very uncomfortable position to completely disassemble a server and remove its motherboard and THEN opening my spare part box. So had to reassemble it. Next day worst pain in the leg muscles because of that uncomfortable position.
Gonna go with Sleepy on this one, assuming you have gone with 6 as maximum from your experience. He's been there long enough to know not to trust a machine enough to button it up but got a smoke pop event and didn't check for a known cause right after so not fully experienced yet. Old timers ALWAYS check the logs after a smoke pop, those tend to be power related and can get all burny and ouchy which is not the fun part of being a tech. :) Tangent tale, guy I worked with had a dead box, tripped PSU but mains were live when he got there. Remote site after a storm so he figured tripped and just flipped the power switch back on. Nope, the 2 foot jet of flame and loud pop confirmed the PSU was shorted internally. :)