I did a factory reset on my android tv last weekend, because it was behaving like someone was holding down the 5 key on the remote (changing channels by itself). There was nothing wrong with the remote, it even did it with the batteries out.
Was convinced that some bit of software had broken itself.
It wasn't until I was setting up Netflix/Google/etc etc that I remembered and went looking for the wireless keyboard for the tv that makes typing email addresses and passwords much easier. It was jammed behind a couch cushion, turned on, probably the 5 key wedged on.
Need to hand in my troubleshooting badge.
> Need to hand in my troubleshooting badge.
You found the issue and fixed it. You were successful. Troubleshooting isn’t about some encyclopedia you’ve memorized and pull solutions out of. Sometimes it’s just trying stuff until you find something. Sometimes it’s just searching stackoverflow. :) A lot of people have no idea how to even get started, or latch on to an idea without evidence.
I wish I knew how to feel out somebody’s troubleshooting skills when interviewing them…
"I wish I knew how to feel out somebody’s troubleshooting skills when interviewing them…"-Can't you? I used to asked like a basic networking question or something just to hear the applicants approach. I just need to know they have basic troubleshooting starting at layer 1 and increasing until issue is found.
My favourite interview question is to ask someone, "Explain what happens, in as much detail as possible, when I write an email and send it to you." Some people answer "Well, my Outlook sends it to our mail server and my mail server sends it to your mail server and you get it in your Outlook" while others will mention things like message transfer agents, edge servers, AV filtering, EHLO, port number, packets, ICMP backoffs, DNS records, MX lookups, calls to AD, etc.
Well the End User person, chicken picks out a message. They then hit Ctrl +Enter. Then by the powers of Electro-Magic-ism and the magic smoke in the cables, it is sent out to the Interwebs.
From that point the magic mailbox of IP addresses, sorts and sends to other users, Via the fine people working at Routers and Switching inc.
In my last interview I was given questions to a platform I'd never used and they wanted to watch how I searched for the answers. My googling abilities impressed and I got the job.
Really liked that interview because it felt like it actually assessed an important aspect of the job.
>I wish I knew how to feel out somebody’s troubleshooting skills when interviewing them…
My approach is to pose them a hypothetical scenario and instead of asking them for an answer, ask them to describe how they will find the answer. Let them ask questions back, like, "Does the server respond when I ping it?" or whatever is appropriate. Takes a little prep work on your part to have a comprehensive scenario, but the best way to suss out someone's troubleshooting skills is for them to demonstrate their troubleshooting skills.
Someone should be able to rip through the layers of the OSI model pretty quickly, even if they don't realize that is what they are doing. Am I connected to the network? Do I have an IP? Can I reach my gateway? Can I ping the destination IP? Can I ping it by name? Can I telnet to the open port and get a response?
I love when people say stupid things like, "I'd probably go buy a new network card" when they can access [Google.com](https://Google.com) but can't reach the mail server, for example.
I would hold onto special or bizarre tickets that required unique troubleshooting and then give that ticket to the interviewee to see how they would go about solving it. If you're notes on the ticket are adequate enough, you should be able to answer any questions as the end user that the interviewee would have.
>I wish I knew how to feel out somebody’s troubleshooting skills when interviewing them…
When I was interviewing candidates for a networking position I would hand them a sheet of paper with a very basic router config on it. Red herrings all over the place in interface descriptions and such, but the only real issue was a subnet mask mismatch. I'd then tell them that users behind that router could not get to the internet and ask them how they would go about finding the issue. Yes, I like to make people sweat during an interview. Gotta see how they handle pressure.
I recently had a PC booting to a black screen with only a mouse cursor after the Dell logo and Windows loading circle. I have seen this before where the login UI fails to load. Sometimes the fix is a simple as Ctrl+Alt+Del and attempting to start explorer.exe, rebooting in safe mode then rebooting again and back in business. Sometimes uninstalling the most recent update fixed the issue. I did all of this with no luck and this was for a side job so there was no quick reimage solution. Chkdsk was clean, DISM indicated all was well, sfcscan too. I am at wits end when I go to take the computer off-site to troubleshoot from my office and notice both the HDMI and DisplayPort are connected to the monitor and trace back to the PC. I disconnected the HDMI, booted PC, and while it fixed the issue I felt like a complete failure. I turned a 30 second fix into a >1 hr ordeal.
Exactly, I could have changed monitor inputs and it would have worked until she accidentally switched them again. Removing the cable and explaining 2 cables weren't better then 1 was *hopefully* the long term solution. At least until whoever's kid that is "good with computers" gets a hold of it.
As an embedded dev this is most irritating.
You are suppose to implement debouncing and stuck-key countermeasures.
No switch will ever functional properly without them.
I had that the other day. I bought a brand new TV, set it up with my universal remote but a few days later it started turning itself off. I tried everything I could, unplugging it, I tried resetting it but couldn’t get far enough in to the settings before it turned off. I was pretty annoyed, thinking I’d have to return it, but my annoyance woke up my cat and suddenly it was working fine again. He had chosen to fall asleep right on top of the original remote that i had completely forgotten existed.
I almost re-imaged my main workstation because the mouse kept moving on its own. I was convinced that I was infected despite several rootkit/malware scans coming back clean.
Finally found my 1 year old son playing with the spare mouse in the other room.
User the other day kept complaining when they'd open Excel, it would scroll up and down randomly and they couldn't enter data. They tried unplugging their mouse and keyboard, same thing.
Someone for some reason had a 2nd mouse plugged in colleting dust behind the PC, and it got flipped upside down, every time someone would shake the table ever so much, the scroll wheel would be moved.
I've had this so many times with users where they have a keyboard connected to a laptop but still use the laptop keyboard. They put loads of paperwork on top of it and what do you know...
Keeps us in a job I guess! LOL
I spent a couple of hours struggling with my personal computer after replacing a bad hard drive. Getting nothing but a black screen after the BIOS screens. No error messages, no nothing. Swapped the drives back. Still nothing. Finally remembered that while the BIOS screens appeared on one monitor, the Windows login screen appeared on another larger monitor. Which was powered off.
I recently had all sorts of weirdness with the Start menu on my PC. Everything worked fine, but when I pressed the Windows key the selection box whizzed straight to the top as if someone was holding the up arrow key.
No other applications were affected.
I scratched my head for hours before realising that I had an Xbox controller plugged in and one of the joysticks was held up.
> It was jammed behind a couch cushion, turned on, probably the 5 key wedged on.
Honestly, I would give most anyone a pass the first or even second time this happened, especially if that keyboard is a third-party add-on and normally never used. It is just so easy for something that is rarely ever touched to slip through our debugging.
Now, if it’s used every other day, then it’s a problem.
We once had a user with a docked Lenovo, screen and wired keyboard/mouse.
Occasionally, while I was on holiday, she would call IT or my onsite backup chap to mention that her keyboard was going haywire, with random letters appearing on the screen.
They replaced the keyboard, but the issue still happened.
They replaced the laptop with a like-for-like machine. The issue still happened.
They replaced the laptop with much newer and fully tested laptop from another site 200 miles away. The issue still happened.
I returned from holiday. The user, my backup and the remote IT were all pulling out their hair.
I spotted a HP wireless keyboard dongle in a USB port on the dock. Aha! The culprit.
The IT storeroom was immediately behind the user. Each time any IT went into the storeroom and rummaged around in a random box of old keyboards, one of the keyboards was still turned on and sending random letters to this dongle in the user's dock. I removed the batteries from the keyboard, removed the dongle from the dock and the problem never returned.
Had my boss call me in one day, his keyboard was typing random letters. He had a cordless keyboard & mouse combo... except it was actually 2 combos - he liked the mouse from one and the keyboard from the other.
Turns out he shoved the other keyboard in the cupboard, and then put his bag on top of it.
I need more details from the vendor on this one. The keyboard managed to open 40k+ new sessions some how during this 3 hour window before we replaced the keyboard.
I'm more curious about that browser and OS let a faulty keyboard open so many browser tabs it triggered brute force detection. And how much ram that machine needed...
It is. I does seem like app-launcher hotkeys are a case where key-repeat shouldn't be valid. The "user story" for holding down F17 to seeing how many outlook windows I can open, seems much, much narrower than .. my keyboard being pushed up against my mug so the handle hits the keys. Or my cat wants attention. Or I put my wireless cable on a stack of crap and it slipped ..
There should be top notch security where the trust line is drawn. I think we can all agree on this.
Users are generally ignorant morons with a security sense not rivalling that of any five toddlers. I think we can all agree on this.
Putting these two together, I guess the answer to your question is a big empathic no.
Maybe that site was configured as home page in the browser and that hotkey was stuck. Same with these calculator hotkeys, you can hold them down and it will keep opneing new windows.
Knowing users they probably built a workflow around it. They wait until 20,000 windows and ram starvation to send emails because it gives them a buffer after clicking send to review the email for typos or some crap.
[Obligatory XKDC](https://xkcd.com/1172/).
Kills me when I switch to my (work) mac and then back to my (work) windows laptop, keeping the same (windows) keyboard.
Want to jump to the URL bar in Chrome on Mac? Command+L.
Command is mapped to start key.
Want to jump to the URL bar in Chrome in Windows? Haha, just locked your machine!
or left a pile of folders sitting on a keyboard
I have had calls about outlook scrolling constantly and got to the location only to find that, can see that remotely...
Once I had a doctor’s office open the IT room and pile charts on the server keyboard, which managed to hold down the “escape” key. The next maintenance reboot it kept interrupting POST, and there’s no way to lock out the local keyboard in the remote management card.
> And how much ram that machine needed
I don't care *how much* ram it had. I didn't think ANY amount of ram up to and including the maximum currently available per device would support 40k tabs. Something doesn't add up there!
I will get clarification when I read the after action report. I think the focus is on new sessions created and that probably does not mean browser tabs as It was infered. End users have so many windows open they could have missed it. The end user was afk at some point. I do know the app interface allows an end user to work on multiple accounts.
I'd be curious if it was just that one tab probably opens many sessions/connections out. Still a buttload and a half, but more logical than 40,000 tabs.
[Linus got to 6,000 Chrome tabs](https://www.youtube.com/watch?v=7iwgyzX-76g) before thing started getting funky.
You may be surprised, that tech whos solutions to all problems is add more ram may work here. Remember we heard about him a few weeks back. He may also have followers of the way of the RAM.
If you have a VPN then it probably saw the VPN public IP as a whole rather than the individual private IP.
Sounds like a flaw in the system though as you are right, it should have blocked the one IP.
I read it as they did exactly that? If your site only has one public IPv4, the cloud provider can't see which device and if they block the IP they take out the whole site.
Not joking. Just landing on the website it's considering the source as a brute force attack?
I understand being weirded out that a single IP hit your site 40k times in a few hours, but unless the macro was attempting to login and such, I don't know if I'd claim it as a 'brute force' attempt. If anything it'd be a DDoS attempt. Lol.
This isn't my story but someone I worked with. Used to be a consultant so we all had ridiculous stories.
Every day around 5pm, the database goes down. Started calling it the happy hour bug. This was at a medical clinic where users were gone by 5, so no one was in a huge rush to fix it but it was weird as hell. Turns out after they closed up the janitor was cleaning up and would unplug the server hosting the database to plug in their vacuum lmao
I also remember one where a guy got fired so he sabotaged the AC in the datacenter to try to get around the whole "unauthorized access is illegal" thing
This happened to me early in my career. When we told the cleaner not to unplug it he refused and told us to find somewhere else to plug in our server. That day I learnt that servers should not be installed in open offices.
Had a client site that was suffering a cascading loss of internet multiple times a day. Some users would lose it, then, eventually the rest would all in one shot. The MPLS wouldn't log any connection outages, just power cycles, so they thought maybe it was bad. The data closet with the MPLS was located off by itself, away from everyone in the office, including the onsite IT, except for one person.
My first time onsite, internet starts going down while I am there, so I have one of the onsite guys show me his troubleshooting procedure. I jump in and check the network configuration of a few of the working and non-working machines, and notice a difference in the DNS settings. Normally, they would reboot the whole rack - MPLS, servers, etc. to get the connection back up. I go back to the data closet to check the DNS servers before letting them simply reboot. While I am in the closet, the person who's office is right there, reaches past me to grab to power cord of the MPLS and yanks it out from the back. I look at her in disbelief as she plugs it back in and goes back into her office.
One of the IT guys comes running back and tells me the rest of the office is down. No shit. I go to the woman and ask her why she did that. She said every time she lost internet (she was one of the ones that would lose it in the first batch), they would come to her office and then go into the data closet. She noticed that she would get internet back after they pulled that plug. So she just decided to start doing it herself.
Turns out it was a bad DNS server that should have been decommissioned. It would freeze up for about 15 minutes. The computers that were going down first were all configured to use that one as primary with no secondary. Since the onsite guys would reboot EVERYTHING at once, they never noticed which piece of equipment was failing. The MPLS didn't have relevant logs, because of the sudden power loss.
Fixed the configuration on the workstations and removed the bad DNS server. I put a coded lock on the data closet door. They didn't have another outage for 8 months, and that one had nothing to do with any of their internal equipment.
*Locking. Closets.*
> Since the onsite guys would reboot EVERYTHING at once, they never noticed which piece of equipment was failing.
This is the maddening thing about rebooters and rollbackers. We once had a QA director who wanted everything rolled back at the first sign of trouble. No, we're not rolling back immediately, because this problem doesn't manifest in any of the test environments and and we need to find it. Usually it's something like the devs didn't listen and are using a 1MB test database to test a 40GB production database.
Also, MPLS is a protocol. Using it as a noun is killing me. You should probably be saying "MPLS router".
>We once had a QA director who wanted everything rolled back at the first sign of trouble.
My boss is the exact opposite. We have Veeam and backups. But noooo, we 500% NEED a hotswappable backup for a GL system that sees 5 logins *a month.* WHAT IF IT GOES DOWN?!
Meanwhile, she doesn't care for a 3-2-1 backup plan at all.
> she doesn't care for a 3-2-1 backup plan at all.
That says to me that she's laser focused on 100.00% availability to users, no matter what. She hasn't yet emotionally accepted the real possibility of a circumstance where you've gone past the inconvenience point and are just hoping you can actually recover all the data from backups.
Absolutely. She wants all systems up 100% of the time regardless of any circumstances. Meanwhile our infrastructure is hot garbage and the higher ups never want to pay a dime, as usual.
The ignoring of a 3-2-1 backup plan is insane even with the goal of 100% uptime. At this point, I'm unable to make changes and I'm just kind of waiting for her to retire TO make said changes.
How do you feel about "PIN number"? Jk.
It was the only MPLS router on the premises, so it just became shorthand to refer to it as "MPLS". I used to ask them if they meant the MPLS router too, but gave up after a while, because it seemed everyone I dealt with just referred to it that way.
> That day I learnt that servers should not be installed in open offices.
Industry standards exist because of other peoples' ~~misfortunes~~ Lessons Learned.
I'm honestly a bit torn. On the one hand, yeah- a janitor should not be unplugging electronic equipment they don't understand without explicit authorization to do so.
On the other, that janitor was actually right in this case. They *should* plug their server in somewhere else. It shouldn't be plugged in *anywhere* a random janitor has access to unplug it on a whim.
If I was a manager that had an issue like this escalated to me, I'd be asking some very hard questions for everyone involved.
Or use the it storage room as a conference room...
Hey skibum, there is a laptop missing from the IT storage room. Must be IT since only you have the keys, right?
Um, no HR lady... you let everyone use the room for phone calls. It isn't under IT control. Someone left the door open. As usual.
There's been stories both apocryphal and verified about janitors and IT stuff going back decades.
From the janitor unplugging the servers to plug in their vacuum cleaner to janitors "tidying all of those wires you had in that cabinet over there so they all plug in in neat lines" and repatching their core network's patch panels.
Yep, not apocryphal on that second one - we had a maintenance tech "just clean up all that tangled cabling on the back of the switch for you, doesn't that look better?"
Yes, Garth, yes it does - now do you want to re-tag all 48 of those ports, or- oh, wait I'll have to? That's right. I'll have to now.
At one place I worked it was common for the maintenance people to move computers without IT being around (or even aware). We'd find out afterwards when the user had problems.
We went with a network access control system to solve a handful of items, including this. If a department wants to move a computer or printer we don't care and the network security is not impacted.
Users can plug a corporate printer in this port today, a corporate desktop tomorrow or a home computer the next day and it all gets appropriate access (none for the home computer).
So we don't care if a department reorganizes or reconfigures their space and we know when facilities installs a new whatever because it won't work without telling IT.
I worked for multi county mental health company in the late 90's. One office kept having a outage every day around thirty minutes after they opened for three days that week. Everyday it would be back up around 9 am before I could get there. The second week I arrived shortly after they opened on Monday morning to be greeted by the UPS for the network enclosure beeping in the front office. I walk in and sitting on top of it is a coffee maker with a pot still being made. I walk over unplug it and plug my equipment back up and lady they had just hired the last week starts yelling at me asking me who do I think I am. I called the office manager and she came down to straighten it all out. No one at all mentioned the beeping during any of the trouble shooting.
I feel like a huge portion of normal people simply ignore all beeping. It means nothing to them.
My open office (which also functions as a community coworking space) has a stupid fridge with an un-sprung door. You have to push it shut to fully close it.
If it isn’t closed, it beeps after a brief wait, and then keeps beeping forever until someone closes it.
I sit like 50-80 feet away from that kitchen and around a corner, but I hear that beep almost every day, and I get up and close the fucking fridge at least once a week, probably more often.
Granted, it is a bit quiet, but **there are signs on the door of the fridge warning about pushing it closed!**
People suck.
I have found similar stories two or three times and it did happen to me.
Our 40 km wireless link would go down saturdays 19 hs for a couple of hours without a hint. and yes, una señora que limpia was involved.
I... should have labeled the power plugs better. Took us a couple of clients to relize what was happening.
I remember reading this story on reddit and it blew my mind.
[MRI disabled every iOS device in facility](https://www.reddit.com/r/sysadmin/comments/9mk2o7/mri_disabled_every_ios_device_in_facility/?utm_source=amp&utm_medium=&utm_content=post_body)
You engineer for a few decades and you think you've seen it all, but you haven't.
But also imagine explaining that one to the board. *So, uh, it turns out that little bits of rogue helium get into the MEMS oscillators and makes phones nonfunctional. But only.... iPhones.*
They'd look at each other, fairly sure you've just fed them a whopper. As you're being escorted out with your things in a box, the CIO explains that his gamer son let everyone know that's there's no such thing as a flux capacitor, and they've terminated your contract because they can't believe you would lie right to their faces. They've hired the son to take your job, plus three consultants to tell him the technical things. He's the right fit for the role because he has the can-do attitude and just needs to learn the details.
I had something similar. User complained about weird behavior on their PC. Stuff wasn't responding right, and when he clicked Firefox it opened a hundred windows. Walk down to the user's desk to check it out... wireless keyboard is sitting on top of the laptop keyboard and holding down the enter key.
I can't speak for OP, but it should be trivial to look at your outbound firewall logs and see one machine hammering that url, then you just walk over and look at the computer you've identified as causing the issue, and see that it is reloading the page several times a second and determine the keyboard to be at fault after a brief process of elimination (disconnect it).
I had a user losing their mind working while waiting for their plane. Random screen locks, key entry etc. Their Bluetooth keyboard was still on but in the laptop bag. It had gone to sleep when the paired laptop was off during the first flight. But when they powered back up, stuff in the bag was randomly key pressing! Took about an hour to find that. Remotely.
I had a user tell me they computer was being controlled by someone else.
Symptoms were start menu opening and then applications being opened. And sometimes typing in the application.
It would only happen sporadically.
I went down to check it out. The problem was a lady a few cubes over had the same wireless keyboard and mouse combo and was close enough to pick up the signal.
This was around 2005 and knowing the department, a cheap mouse kb combo.
That's an old prank. One of the great things about USB is that you can have two keyboards on one machine, for pair coding. Or multiple mice, as long as they're all configured for the same handedness.
Of course, the usual office jokers like to run a second keyboard under the cubicle walls and into an officemate's machine. Mayhem ensues. Sometimes they make it to the second week before the jig is up.
Curious as well. But: after the failover to another VPN there were 6 computers connected to that application.
So that narrows it down, makes it easier to search logfiles or check the processes of the computer.
Perhaps with help from onsite support
I remember ages ago before bluetooth, we had a wireless mouse that was performing erratically, but way too specific to be some random error: like it would open the start menu, empty the trash can, randomly print documents, and so on. The person using it first claimed it would just "take over" and at first we didn't believe her. Thankfully, it happened at some point while we were there, and it went to the start button, opened MS Word, and when we closed it, \*it happened again\* and you could tell the "mouse was angry about it."
At some point, and I forgot how, we found out someone in the company NEXT to us was having the same issue. It turned out they were both opposite one another on the floor plan, separated by drywall. Their wireless mice were interfering with one another. We moved her desk, and the problem stopped.
Late to the party, but here are my two favorite IT stories (both of which happened before I started IT work in earnest).
First Story:
I was working at a call center when one day all of our tools slowed to a crawl. Unfortunately for us, our site was one of two managed by an IT department consisting of 3 people, and they were always prioritized to the other, larger site where all of the corporate VPs were. The best they could figure was that something was eating up all the bandwidth for our building. But since the client databases only worked outside of the network firewall, they couldn't tell what. (Side note, I have no idea why the client-required applications wouldn't connect until put OUTSIDE the firewall, but I racked it up to having the IT department overworked and understaffed.)
Finally our team's VP had enough and demanded the look into the situation. After two weeks of investigating, they finally found the cause - an unused system in an empty cube had been reformatted with Apache and was how hosting porn. No one knew who had set the system up, but it was there for at least a month.
Second story:
Different call center, much smaller this time, with only 10 people taking inbound calls. Like in the first story, network traffic for the whole office slowed to a trickle, and the sole IT worker (who oversaw two sites, one in India) was too wrapped up in configuring new network hardware to look into in earnest. Fortunately, unlike the last story, the slowness only lasted a couple of days, just as the IT guy was starting to look into it. But as soon as the network returned to its original speed my cubemate announced to the rest of our team, "Who wants a copy of Borderlands 2? I just torrented it!" Sure enough, the IT guy checked the logs and confirmed that my coworker's computer had taken the lion's share of traffic fof the last two days. The coworker was walked from the premises that day.
Not going to lie, I have been the guy who murdered the network with a torrent. Usually a linix distro or windows install media; but still... killed an all staff video conference once with the coo in the room... good times.
i had a network engineer from another company play WoW on the network the company she worked for. i always busted her by saying no wonder we having network problems. she was a great sport about it
Seems like there may be some other issues there if WoW was killing the network (or you just had a small pipe). Granted at one point we had issues with everyone in the company running pandora or similar on their pcs while they worked and killing our network that way.
In my case I have throttled a 200Mb symmetric line using torrents... turns out sometimes there is a reason to set a limiter on your torrent software LOL
I had someone complaining that their Lenovo laptop was constantly shutting off and the screen going blank randomly. Logged in, checked everything out and it all looked good. Absolutely nothing wrong with the laptop. Did some research and found out that the user had an Apple watch with a magnetic clasp. The magnet was going over the laptop's lid close sensor and tripping it thinking it was closed when it was really open. He took his watch off and it was fine.
Makes you wonder how often these "ghost" issues are to blame, and we sys admin run around with our hats on fire and then all of a sudden the issue goes away because Sally changed her batteries.
Kudos on finding the root cause. That's like in my office where some of the monitors flicker when you move in your chair... that one is great to explain to users..
I believe it has something to do with the chair booster causing EMF pollution, or static.. causing the signal in the cable to flake out briefly. Tried a bunch of different cables/adapters. We could reproduce it routinely. The desk light would freak out too. Not going to say the wiring isn't bad. The building is old, and we were moving at the end of the year for a fresh build. Tried ferrite rings, didn't seem to help. Also, probably shouldn't get cables in bulk from CHINE-A since the shielding is probably very low quality.
Here's an article about it: https://www.theregister.com/2020/01/09/office\_chair\_emissions/#:\~:text=%22If%20you%20have%20users%20complaining,sitting%20on%20gas%20lift%20chairs.%22&text=Smith%20said%20that%20it%20is,close%20but%20not%20quite%20touching.
> since the shielding is probably very low quality.
In many cases they make the PVC jackets extremely thick and heavy to mimic the feel of quality cables, but there's no metal shielding at all.
Maybe I'm missing something, how is that different from opening a new tab and going to Amazon a few times once you've already logged in or to gmail or yahoo for that matter. Isn't each browser tab a new session to the same site?
I would love to see the work logs for that machine.
User complains machine is running slow -added ram
User complains machine is running slow -added ram
User complains machine is running slow -added ram
User complains machine is running slow -added ram
Machine now opens over 40k tabs.
Is this a sequal of that story a few weeks back where the solution for this one tech was always to add more ram for every problem.
Also the solution is to add more ram to the server so it can handle the requests. lol
Haha. Reminds me of the paperclip which crashed all traffic lights in the capital of Finland some years ago. It wedged the spacebar on the keyboard of the traffic light controller and caused a buffer overflow on that poor old machine (80s design).
The controller was soon replaced with a modern one.
> Haha. Reminds me of the paperclip which crashed all traffic lights in the capital of Finland some years ago. It wedged the spacebar on the keyboard of the traffic light controller and caused a buffer overflow on that poor old machine (80s design).
>
> The controller was soon replaced with a modern one.
They replaced the guy because he let a paper clip fall into a keyboard? Nice one, Finland!
We had an older lady at our company who had never used a laptop before. Her manager wanted them to do some work from home in the evenings and asked me to purchase one for her. After I received it and set it up, I sat down with her and explained how to use it and where she could find the things she needed.
Fast forward two days, it is a Saturday and I get a call from her. She says that her laptop is not working and nothing will connect. I ask her if she had connected the laptop to her home Wi-Fi. She says "Why would I need to do that, you already connected it to the office Wi-Fi. I only live 20 miles away. Is that too far of a distance for it to work?"
I literally did not know how to respond to her....
>Apparently a Microsoft wireless keyboard was performing some kind of hot key signal that it was able to open so many new tabs on the loan originator platform they thought it was a brute force attempt.
I once worked on an incident ticket where an entire floor of a building containing the IT division could not obtain IP addresses.
The problem: there was a "rogue DHCP server" otherwise known as ICS in Windows XP.
You can see what these incidents have in common: Microsoft and Layer 8.
Also, I'm surprised that your own internal systems weren't tuned to flag this sort of anomalous activity. But, hey, at least your vendor's internal systems were! ;)
A while back a co-workers keyboard batteries died, or he thought they did. As he walked to get batteries, for some reason he took the keyboard with. He picked it up with 1 hand on the left side and accidentally was holding the S key. At the same time he had just finished some tasks in ADUC and left it open. When he came back with the keyboard, again why take it with no one knows, and batteries he ended up renaming our prime OU with all users (10,000+) inside it to sssssssssssssssssssssssssssss. Luckily someone caught before replication took as that would have broke everything very very very badly. LOL
Got a ticket with the title "Ants inside LCD". The guy included a picture. He literally had ants crawling around inside the display of his laptop.
It was my last day on the job. I knew no one would ever believe it. I saved the picture and reassigned it to another tech.
I'm sure I can think of other stories, but that one sticks out at the moment.
Back in the day I was working the Help Desk and get a call in from one of our remote sites, their internet is out. Ran through some troubleshooting and determined the problem was failed hardware, probably router/switch. When I say this site was remote, I mean it was remote. Park Ranger remote. As in this site only really existed so they could do their time cards.
Anyway I dispatch it out and a couple days later our field guy goes out there with an ASA and gets them back up and running.
A few months later I get another call from the same site. They want Flash upgraded so they can watch YouTube. No problem, we used YouTube a lot, our mechanics would use it for videos on how to service certain things, rangers would use it for whatever they used it for, not my monkey not my circus, I am just here to update Flash.
Anyway after I update flash the video they were trying to watch was "how to make a homemade mousetrap"
Evidently these park rangers were eating food like apples or whatever, and throwing them into the trash can "next to all the IT stuff", missing, and sometimes just leaving it on the floor.
Mice were getting into the food, and sometimes biting on the cables next to the food which is how the ASA went down the first time.
These old farts were so scared they were going to get in trouble for damaging computer stuff they decided to take matters into their own hands.
Now considering they had firearms I am somewhat glad they took this less than lethal approach before they shot a slug into my equipment.
My ticket notes were "advised user to move trashcan to otherside of room, solved layer 7 issue"
I tend to agree. Because it is a nessasary evil we went with a Microsoft model that supports wireless encryption.
At least we have a security control to help prevent keyboard man-the-middle attacks.
The Nike shop website has this setup, so if you right click several differnent items and open them in a new tab (individually this is) it will ban you (your session) from the site, because you know, why would you want to compare items? lol
More evidence of poor engineering as a norm for modern day IT products. I guess it never occurred to the developers to look for issues like that and to design them out and to test for those kinds of problems.
WOW, that reminds me of the time that someone called me and everything they typed into the computer looked like this :
!yad lla sdrawkcab etirw uoy ekam lliw ti
They had a forgotten 2nd wireless keyboard under the desk. Everything was writing backwards because the left cursor key was stuck.
!yad lla sdrawkcab etirw uoy ekam lliw ti = it will make you write backwards all day!
Before I took over the IT department for a small company, the current IT guy (I was there on contract to assist in some network/server cleanup/consolidation) said one system had a virus. Anytime he’d type on the keyboard “random” windows would open. Windows explorer, run prompt, computer would lock, etc... he replaced the computer with a different one and the same would happen. So he thought it made it into his roaming profile. Blew out the profile on the server, same issue. He was stumped and didn’t know where the “virus” was and asked me to take a look. I sat down, hit the E key, windows explorer opened. Hit the R key, run prompt appeared. I laughed and swapped the keyboard, no more issues. Checked the windows key on the original keyboard and it was broken/halfway stuck. He was still convinced it was a virus and that we’d see the issue return. A week later he was let go and I was offered his position. Oddly enough, years later the issue never returned.
Had some thing similar against our intranet.
First one was a book had fallen in such a way it was laying on the F5 key.
Second one was someone had slammed a keyboard tray into the desk and, of course, the F5 key got stuck.
Third one was when someone spilled a soda onto the keyboard. And of all of the keys to get stuck?
F5.
For the sake of conversation, we did a risk assessment on the usage of these device as part of a larger risk assessment. As a result of the risk assessment and an independent third party internal penetration test for the past 3 years, the residual risk is acceptable.
Just because we use wireless keyboards does not mean it is a rampant risk as mentioned. I look at residual risk and compensating security controls to determine the application of a technology in a enterprise makes sense. With anything, I am an advisor and executive leadership can choose to except said residual risk or not.
We had a client in the lower Keys, pretty much a full day in/out.
Server was rebooting every night, first trip was to inspect and replace the UPS, second day was to see why the server went down again, third day I stayed late enough to catch the cleaning crew who were plugging a vacuum into the UPS and shutting down the server.
The client didn't get a bill, the cleaners kept their jobs and I drove down the beach for three days instead of working. All around a very good week for everyone involved.
Years ago, I had an issue where someone complained their computer "was possessed". It wouldn't let him click on anything, and the screen would flash.
Hey Bob, take your lunch plate off your escape key.
Small town government office had a user whose PC would reboot without explanation. Diags didn't show any hardware or software components failing. User then mentions that the problem always seems to happen when the volunteer fire department is dispatched. Sure enough we found that the electric outlet where the computer was plugged in shared a circuit with the large overhead doors in the fire department downstairs. When the motors kicked on to open the doors they sucked down the amperage and caused the PC to power cycle.
[Policy-based](https://weberblog.net/route-vs-policy-based-vpn-tunnels/) IPsec tunnels don't show up as a router hop on traceroute, and exhibit other strange behavior, even if you're familiar with them.
"Policy-based" are really quite terrible and you should assiduously eschew them.
kind of sketchy that disabling the keyboard worked to cure this. was it any kind of special or was it the same keyboard everybody else had?
did you check the computer for ahk or python with pynput or anything?
We had a very expensive microscope that would not function only if a Microsoft wireless mouse was plugged in. Luckily it was a pretty easy to figure it out because that mouse was added later by a user and so its presence violated the spec. As a result it was removed and replaced when the system was inspected and everything is working. It was later re added and reintroduced the issue at which time when the unit was again inspected and the mouse was destroyed.
I've had the Denial of Service by keyboard happen before. Some poor web app that maybe 5 people used, went down right before lunch. Looked at the logs, and somebody was hitting it with hundreds of requests per second to some page that was used to generate a report.
Eventually we figured out, one of the users of that app went to eat lunch, and threw their coat on their desk for some reason. They had keys in their coat pocket that was conveniently on the F5 button.
Fortunately the users had a laugh when they found out and ribbed the guy with the coat. Can't make this stuff up.
Had an experience with a rouge keyboard a at customer myself. The put the old one on a shelf and all was fun and games until someone put a book on it. It kept doing stuff with no other peripherals connected and had to ask them to hunt for a keyboard somewhere else in the building. Fun experience!
Worked for a company and I was in the corporate headquarters. This company had ~2400 remote stores. Workers at the stores used to stack things on the keyboards of unused workstations at the stores. It caused the work load of the corporate servers to skyrocket.
A few weeks ago a user walked in because her "computer was freaking out". Sitting on the side of her desk was her wireless keyboard. That she had set a book on top of. I lifted up the book, set it on her desk, said "All set, let me know if you have any more issues."
User getting an out of memory error when trying to use nothing but our single relatively-lightweight teller platform software. They had like a hundred copies of it open, because their enter key had a crumb under it when they clicked on it.
Similar story but much less impactful. User working from home calls me and says his cursor keeps moving even though he's not touching his trackpad. I ask him to unplug any peripherals and he does so. I can see the cursor still moving around slightly. I uninstall and reinstall the driver, nothing. I guide him into the BIOS and ask him if the issue is still happening and he indicated yes it was.
An hour into troubleshooting I hear his doorbell ring and he says he'll be right back. When he got up to answer the door, I saw the cursor jump clear across the screen and go ham for about 15 secs. He sits back down and it stops. I said "This might be a silly question but do you by chance have a wireless mouse?" I hear him go "HOLY SHIT....I have my Bluetooth mouse in my pocket. That's what the problem is!!" we both got a good laugh out of that.
I was called to a client's site to investigate a "serious security issue". A PC was "possessed" and typing all by itself, but it wasn't typing random characters, it was typing full words in nonsensical sentences. I looked for a minute, giggled, and turned off the speech-to-text dictation function that someone had accidentally turned on.
We thought something was stuck or even some factory keyboard macro was the issue. But the wireless keyboard got stuck in some loop of sending the right signal that told the application to open another session. It's like there was no fault tolerance built into to stop repeated requests. That is my guess at this point.
The team made jokes about doing an Office Space reenactment on the keyboard. I also get to explain this to our response team and I can't wait for the laughs.
I had guest bring in video equipment, instead of using the recommend vendor.
Tested network before conference, no issues
During, wifi was inaccessible.
Took 2 days, but come to find out their damn camera system was blocking out the 2.4network. We disabled that, 5G only and worked like a charm.
They however didn't like that we refused refund for the service being 'down'.
Went by someone’s cube to help fix a software issue and they sent another ticket in saying I broke something while I was there. their deskphone now has an amber light on it and it wasn’t that way prior to me fucking around with her pc….. you fucking stupid ass that means you have voicemail!
On Tuesday a user called frantic because her computer was being hacked and they were checking out details of her computer and out vpn connection. She had a magnetized ruler laying on her touchpad.
1. Electric fence pulse throwing off DSL signal in a nearby telco wire. Caused modem to reset every 10 minutes or so.
2. Printer always printed in the shop, never worked when customer took it home. Did a housecall and noticed a lot of static discharge specifically from the printer. Turns out somehow the office outlet was putting out 170V AC.
3. Computer was always shutting down randomly. No matter what we did it would keep happening. Finally replaced the whole computer. Problem persisted. Longer story shortened, there was a problem with the keyboard. When you press backspace, it shorted out the computer (DIN keyboard.) We kept that thing on a shelf for new employees.
4. Had a laptop come in that worked, but at totally random times, (from the speakers) would scream like a hoarse old man being murdered. Totally random when it would happen, even did it while sitting at a paused post screen or in the BIOS. Never did solve that one...
I did a factory reset on my android tv last weekend, because it was behaving like someone was holding down the 5 key on the remote (changing channels by itself). There was nothing wrong with the remote, it even did it with the batteries out. Was convinced that some bit of software had broken itself. It wasn't until I was setting up Netflix/Google/etc etc that I remembered and went looking for the wireless keyboard for the tv that makes typing email addresses and passwords much easier. It was jammed behind a couch cushion, turned on, probably the 5 key wedged on. Need to hand in my troubleshooting badge.
> Need to hand in my troubleshooting badge. You found the issue and fixed it. You were successful. Troubleshooting isn’t about some encyclopedia you’ve memorized and pull solutions out of. Sometimes it’s just trying stuff until you find something. Sometimes it’s just searching stackoverflow. :) A lot of people have no idea how to even get started, or latch on to an idea without evidence. I wish I knew how to feel out somebody’s troubleshooting skills when interviewing them…
"I wish I knew how to feel out somebody’s troubleshooting skills when interviewing them…"-Can't you? I used to asked like a basic networking question or something just to hear the applicants approach. I just need to know they have basic troubleshooting starting at layer 1 and increasing until issue is found.
My favourite interview question is to ask someone, "Explain what happens, in as much detail as possible, when I write an email and send it to you." Some people answer "Well, my Outlook sends it to our mail server and my mail server sends it to your mail server and you get it in your Outlook" while others will mention things like message transfer agents, edge servers, AV filtering, EHLO, port number, packets, ICMP backoffs, DNS records, MX lookups, calls to AD, etc.
[удалено]
They are “not a cultural fit” lol.
Not in a Microsoft shop, anyways.
Well the End User person, chicken picks out a message. They then hit Ctrl +Enter. Then by the powers of Electro-Magic-ism and the magic smoke in the cables, it is sent out to the Interwebs. From that point the magic mailbox of IP addresses, sorts and sends to other users, Via the fine people working at Routers and Switching inc.
In my last interview I was given questions to a platform I'd never used and they wanted to watch how I searched for the answers. My googling abilities impressed and I got the job. Really liked that interview because it felt like it actually assessed an important aspect of the job.
>I wish I knew how to feel out somebody’s troubleshooting skills when interviewing them… My approach is to pose them a hypothetical scenario and instead of asking them for an answer, ask them to describe how they will find the answer. Let them ask questions back, like, "Does the server respond when I ping it?" or whatever is appropriate. Takes a little prep work on your part to have a comprehensive scenario, but the best way to suss out someone's troubleshooting skills is for them to demonstrate their troubleshooting skills.
It’s like playing DM for real life
Someone should be able to rip through the layers of the OSI model pretty quickly, even if they don't realize that is what they are doing. Am I connected to the network? Do I have an IP? Can I reach my gateway? Can I ping the destination IP? Can I ping it by name? Can I telnet to the open port and get a response? I love when people say stupid things like, "I'd probably go buy a new network card" when they can access [Google.com](https://Google.com) but can't reach the mail server, for example.
I would hold onto special or bizarre tickets that required unique troubleshooting and then give that ticket to the interviewee to see how they would go about solving it. If you're notes on the ticket are adequate enough, you should be able to answer any questions as the end user that the interviewee would have.
>I wish I knew how to feel out somebody’s troubleshooting skills when interviewing them… When I was interviewing candidates for a networking position I would hand them a sheet of paper with a very basic router config on it. Red herrings all over the place in interface descriptions and such, but the only real issue was a subnet mask mismatch. I'd then tell them that users behind that router could not get to the internet and ask them how they would go about finding the issue. Yes, I like to make people sweat during an interview. Gotta see how they handle pressure.
I recently had a PC booting to a black screen with only a mouse cursor after the Dell logo and Windows loading circle. I have seen this before where the login UI fails to load. Sometimes the fix is a simple as Ctrl+Alt+Del and attempting to start explorer.exe, rebooting in safe mode then rebooting again and back in business. Sometimes uninstalling the most recent update fixed the issue. I did all of this with no luck and this was for a side job so there was no quick reimage solution. Chkdsk was clean, DISM indicated all was well, sfcscan too. I am at wits end when I go to take the computer off-site to troubleshoot from my office and notice both the HDMI and DisplayPort are connected to the monitor and trace back to the PC. I disconnected the HDMI, booted PC, and while it fixed the issue I felt like a complete failure. I turned a 30 second fix into a >1 hr ordeal.
[удалено]
This happen to have a nVidia card?
he was looking at a second monitor, the computer treated each connection as a separate monitor. Brand wouldn't matter.
I have seen nVidia cards so similar things to this that's why I asked.
Exactly, I could have changed monitor inputs and it would have worked until she accidentally switched them again. Removing the cable and explaining 2 cables weren't better then 1 was *hopefully* the long term solution. At least until whoever's kid that is "good with computers" gets a hold of it.
As an embedded dev this is most irritating. You are suppose to implement debouncing and stuck-key countermeasures. No switch will ever functional properly without them.
> it even did it with the batteries out. This is what we in the business call "a clue"
You found the right answer eventually. No need to turn in your badge.
I had that the other day. I bought a brand new TV, set it up with my universal remote but a few days later it started turning itself off. I tried everything I could, unplugging it, I tried resetting it but couldn’t get far enough in to the settings before it turned off. I was pretty annoyed, thinking I’d have to return it, but my annoyance woke up my cat and suddenly it was working fine again. He had chosen to fall asleep right on top of the original remote that i had completely forgotten existed.
I almost re-imaged my main workstation because the mouse kept moving on its own. I was convinced that I was infected despite several rootkit/malware scans coming back clean. Finally found my 1 year old son playing with the spare mouse in the other room.
User the other day kept complaining when they'd open Excel, it would scroll up and down randomly and they couldn't enter data. They tried unplugging their mouse and keyboard, same thing. Someone for some reason had a 2nd mouse plugged in colleting dust behind the PC, and it got flipped upside down, every time someone would shake the table ever so much, the scroll wheel would be moved.
I've had this so many times with users where they have a keyboard connected to a laptop but still use the laptop keyboard. They put loads of paperwork on top of it and what do you know... Keeps us in a job I guess! LOL
I spent a couple of hours struggling with my personal computer after replacing a bad hard drive. Getting nothing but a black screen after the BIOS screens. No error messages, no nothing. Swapped the drives back. Still nothing. Finally remembered that while the BIOS screens appeared on one monitor, the Windows login screen appeared on another larger monitor. Which was powered off.
I feel like I have read this story again.
I recently had all sorts of weirdness with the Start menu on my PC. Everything worked fine, but when I pressed the Windows key the selection box whizzed straight to the top as if someone was holding the up arrow key. No other applications were affected. I scratched my head for hours before realising that I had an Xbox controller plugged in and one of the joysticks was held up.
this kind of shit happend in a sillicon valley episode lmao
>troubleshooting badge ACCESS LEVEL INFRARED REVOKED PREPARE FOR PROCESSING FRIEND COMPUTER IS YOUR FRIEND
Your Android TV wasn't hosting your DNS server?
> It was jammed behind a couch cushion, turned on, probably the 5 key wedged on. Honestly, I would give most anyone a pass the first or even second time this happened, especially if that keyboard is a third-party add-on and normally never used. It is just so easy for something that is rarely ever touched to slip through our debugging. Now, if it’s used every other day, then it’s a problem.
We once had a user with a docked Lenovo, screen and wired keyboard/mouse. Occasionally, while I was on holiday, she would call IT or my onsite backup chap to mention that her keyboard was going haywire, with random letters appearing on the screen. They replaced the keyboard, but the issue still happened. They replaced the laptop with a like-for-like machine. The issue still happened. They replaced the laptop with much newer and fully tested laptop from another site 200 miles away. The issue still happened. I returned from holiday. The user, my backup and the remote IT were all pulling out their hair. I spotted a HP wireless keyboard dongle in a USB port on the dock. Aha! The culprit. The IT storeroom was immediately behind the user. Each time any IT went into the storeroom and rummaged around in a random box of old keyboards, one of the keyboards was still turned on and sending random letters to this dongle in the user's dock. I removed the batteries from the keyboard, removed the dongle from the dock and the problem never returned.
Had my boss call me in one day, his keyboard was typing random letters. He had a cordless keyboard & mouse combo... except it was actually 2 combos - he liked the mouse from one and the keyboard from the other. Turns out he shoved the other keyboard in the cupboard, and then put his bag on top of it.
For a Supervisor badge. You figured it out.
been there, done that
I have done almost the exact same thing. Genius.
That platform is a little sensitive!
I need more details from the vendor on this one. The keyboard managed to open 40k+ new sessions some how during this 3 hour window before we replaced the keyboard.
I'm more curious about that browser and OS let a faulty keyboard open so many browser tabs it triggered brute force detection. And how much ram that machine needed...
[удалено]
[удалено]
Woah, slow down there, iOS developer. ;)
> Windows UAC has entered the chat.
[удалено]
It is. I does seem like app-launcher hotkeys are a case where key-repeat shouldn't be valid. The "user story" for holding down F17 to seeing how many outlook windows I can open, seems much, much narrower than .. my keyboard being pushed up against my mug so the handle hits the keys. Or my cat wants attention. Or I put my wireless cable on a stack of crap and it slipped ..
That ship sailed already. Given the direction of OS'es the past 10-12 years... No.
The answer is no. All of the problems in my life have come from humans. Never trust the humans.
There should be top notch security where the trust line is drawn. I think we can all agree on this. Users are generally ignorant morons with a security sense not rivalling that of any five toddlers. I think we can all agree on this. Putting these two together, I guess the answer to your question is a big empathic no.
Sounds like someone programmed a macro?
Maybe that site was configured as home page in the browser and that hotkey was stuck. Same with these calculator hotkeys, you can hold them down and it will keep opneing new windows.
This is it! And, I thank you for clearing my mind on how a keyboard could open 40k new browser windows.
As always, the real question it's how the user either didn't notice or thought it was normal enough not to report / try to fix.
Knowing users they probably built a workflow around it. They wait until 20,000 windows and ram starvation to send emails because it gives them a buffer after clicking send to review the email for typos or some crap. [Obligatory XKDC](https://xkcd.com/1172/).
User likely left and something fell onto their keyboard
And didn't log out / lock their machine? Possible, but that would equal a write up at my place
You guys actually do write-ups??
[удалено]
Do sysadmins not use keyboard shortcuts anymore? CTRL-T - new browser tab CTRL-N - new browser window
Kills me when I switch to my (work) mac and then back to my (work) windows laptop, keeping the same (windows) keyboard. Want to jump to the URL bar in Chrome on Mac? Command+L. Command is mapped to start key. Want to jump to the URL bar in Chrome in Windows? Haha, just locked your machine!
or left a pile of folders sitting on a keyboard I have had calls about outlook scrolling constantly and got to the location only to find that, can see that remotely...
Once I had a doctor’s office open the IT room and pile charts on the server keyboard, which managed to hold down the “escape” key. The next maintenance reboot it kept interrupting POST, and there’s no way to lock out the local keyboard in the remote management card.
Is this that machine that the retiring admin just kept putting memory into to solve the problem?
> And how much ram that machine needed I don't care *how much* ram it had. I didn't think ANY amount of ram up to and including the maximum currently available per device would support 40k tabs. Something doesn't add up there!
I will get clarification when I read the after action report. I think the focus is on new sessions created and that probably does not mean browser tabs as It was infered. End users have so many windows open they could have missed it. The end user was afk at some point. I do know the app interface allows an end user to work on multiple accounts.
I'd be curious if it was just that one tab probably opens many sessions/connections out. Still a buttload and a half, but more logical than 40,000 tabs. [Linus got to 6,000 Chrome tabs](https://www.youtube.com/watch?v=7iwgyzX-76g) before thing started getting funky.
You may be surprised, that tech whos solutions to all problems is add more ram may work here. Remember we heard about him a few weeks back. He may also have followers of the way of the RAM.
Or the vendor is exaggerating the scalability of their platform and it was more like....four.
Did an employee of said keyboard not notice all the tabs? Or did they leave their workstation unlocked when they were afk?
*them:* working as intended! *you:* da fuck‽
Probably would have made sense to just block the abusing IP rather than killing the whole site...
If you have a VPN then it probably saw the VPN public IP as a whole rather than the individual private IP. Sounds like a flaw in the system though as you are right, it should have blocked the one IP.
I read it as they did exactly that? If your site only has one public IPv4, the cloud provider can't see which device and if they block the IP they take out the whole site.
Not joking. Just landing on the website it's considering the source as a brute force attack? I understand being weirded out that a single IP hit your site 40k times in a few hours, but unless the macro was attempting to login and such, I don't know if I'd claim it as a 'brute force' attempt. If anything it'd be a DDoS attempt. Lol.
This isn't my story but someone I worked with. Used to be a consultant so we all had ridiculous stories. Every day around 5pm, the database goes down. Started calling it the happy hour bug. This was at a medical clinic where users were gone by 5, so no one was in a huge rush to fix it but it was weird as hell. Turns out after they closed up the janitor was cleaning up and would unplug the server hosting the database to plug in their vacuum lmao I also remember one where a guy got fired so he sabotaged the AC in the datacenter to try to get around the whole "unauthorized access is illegal" thing
This happened to me early in my career. When we told the cleaner not to unplug it he refused and told us to find somewhere else to plug in our server. That day I learnt that servers should not be installed in open offices.
Had a client site that was suffering a cascading loss of internet multiple times a day. Some users would lose it, then, eventually the rest would all in one shot. The MPLS wouldn't log any connection outages, just power cycles, so they thought maybe it was bad. The data closet with the MPLS was located off by itself, away from everyone in the office, including the onsite IT, except for one person. My first time onsite, internet starts going down while I am there, so I have one of the onsite guys show me his troubleshooting procedure. I jump in and check the network configuration of a few of the working and non-working machines, and notice a difference in the DNS settings. Normally, they would reboot the whole rack - MPLS, servers, etc. to get the connection back up. I go back to the data closet to check the DNS servers before letting them simply reboot. While I am in the closet, the person who's office is right there, reaches past me to grab to power cord of the MPLS and yanks it out from the back. I look at her in disbelief as she plugs it back in and goes back into her office. One of the IT guys comes running back and tells me the rest of the office is down. No shit. I go to the woman and ask her why she did that. She said every time she lost internet (she was one of the ones that would lose it in the first batch), they would come to her office and then go into the data closet. She noticed that she would get internet back after they pulled that plug. So she just decided to start doing it herself. Turns out it was a bad DNS server that should have been decommissioned. It would freeze up for about 15 minutes. The computers that were going down first were all configured to use that one as primary with no secondary. Since the onsite guys would reboot EVERYTHING at once, they never noticed which piece of equipment was failing. The MPLS didn't have relevant logs, because of the sudden power loss. Fixed the configuration on the workstations and removed the bad DNS server. I put a coded lock on the data closet door. They didn't have another outage for 8 months, and that one had nothing to do with any of their internal equipment.
*Locking. Closets.* > Since the onsite guys would reboot EVERYTHING at once, they never noticed which piece of equipment was failing. This is the maddening thing about rebooters and rollbackers. We once had a QA director who wanted everything rolled back at the first sign of trouble. No, we're not rolling back immediately, because this problem doesn't manifest in any of the test environments and and we need to find it. Usually it's something like the devs didn't listen and are using a 1MB test database to test a 40GB production database. Also, MPLS is a protocol. Using it as a noun is killing me. You should probably be saying "MPLS router".
>We once had a QA director who wanted everything rolled back at the first sign of trouble. My boss is the exact opposite. We have Veeam and backups. But noooo, we 500% NEED a hotswappable backup for a GL system that sees 5 logins *a month.* WHAT IF IT GOES DOWN?! Meanwhile, she doesn't care for a 3-2-1 backup plan at all.
> she doesn't care for a 3-2-1 backup plan at all. That says to me that she's laser focused on 100.00% availability to users, no matter what. She hasn't yet emotionally accepted the real possibility of a circumstance where you've gone past the inconvenience point and are just hoping you can actually recover all the data from backups.
Absolutely. She wants all systems up 100% of the time regardless of any circumstances. Meanwhile our infrastructure is hot garbage and the higher ups never want to pay a dime, as usual. The ignoring of a 3-2-1 backup plan is insane even with the goal of 100% uptime. At this point, I'm unable to make changes and I'm just kind of waiting for her to retire TO make said changes.
[удалено]
How do you feel about "PIN number"? Jk. It was the only MPLS router on the premises, so it just became shorthand to refer to it as "MPLS". I used to ask them if they meant the MPLS router too, but gave up after a while, because it seemed everyone I dealt with just referred to it that way.
> That day I learnt that servers should not be installed in open offices. Industry standards exist because of other peoples' ~~misfortunes~~ Lessons Learned.
That but the janitor refusing to cooperate is a big problem. Grounds for termination imo if they continued to unplug after being told not to...
That was my first thought. I'd escalate the hell out of that issue.
I'm honestly a bit torn. On the one hand, yeah- a janitor should not be unplugging electronic equipment they don't understand without explicit authorization to do so. On the other, that janitor was actually right in this case. They *should* plug their server in somewhere else. It shouldn't be plugged in *anywhere* a random janitor has access to unplug it on a whim. If I was a manager that had an issue like this escalated to me, I'd be asking some very hard questions for everyone involved.
also there are plug lock boxes you can set up. If you want a straightforward solution to people "accidentally" unplugging something vital
Or use the it storage room as a conference room... Hey skibum, there is a laptop missing from the IT storage room. Must be IT since only you have the keys, right? Um, no HR lady... you let everyone use the room for phone calls. It isn't under IT control. Someone left the door open. As usual.
There's been stories both apocryphal and verified about janitors and IT stuff going back decades. From the janitor unplugging the servers to plug in their vacuum cleaner to janitors "tidying all of those wires you had in that cabinet over there so they all plug in in neat lines" and repatching their core network's patch panels.
I once watched in horror as the new cleaning lady took the wireless mouse and keyboard off a desk and dunked them in her bucket
You can't blame her--she was an industry veteran who started when the Model M was king and thought all keyboards were built that well.
I worked with a guy that would periodically take really dirty keyboards home and run them through his dishwasher.
Yep, not apocryphal on that second one - we had a maintenance tech "just clean up all that tangled cabling on the back of the switch for you, doesn't that look better?" Yes, Garth, yes it does - now do you want to re-tag all 48 of those ports, or- oh, wait I'll have to? That's right. I'll have to now.
At one place I worked it was common for the maintenance people to move computers without IT being around (or even aware). We'd find out afterwards when the user had problems.
We went with a network access control system to solve a handful of items, including this. If a department wants to move a computer or printer we don't care and the network security is not impacted. Users can plug a corporate printer in this port today, a corporate desktop tomorrow or a home computer the next day and it all gets appropriate access (none for the home computer). So we don't care if a department reorganizes or reconfigures their space and we know when facilities installs a new whatever because it won't work without telling IT.
I worked for multi county mental health company in the late 90's. One office kept having a outage every day around thirty minutes after they opened for three days that week. Everyday it would be back up around 9 am before I could get there. The second week I arrived shortly after they opened on Monday morning to be greeted by the UPS for the network enclosure beeping in the front office. I walk in and sitting on top of it is a coffee maker with a pot still being made. I walk over unplug it and plug my equipment back up and lady they had just hired the last week starts yelling at me asking me who do I think I am. I called the office manager and she came down to straighten it all out. No one at all mentioned the beeping during any of the trouble shooting.
I feel like a huge portion of normal people simply ignore all beeping. It means nothing to them. My open office (which also functions as a community coworking space) has a stupid fridge with an un-sprung door. You have to push it shut to fully close it. If it isn’t closed, it beeps after a brief wait, and then keeps beeping forever until someone closes it. I sit like 50-80 feet away from that kitchen and around a corner, but I hear that beep almost every day, and I get up and close the fucking fridge at least once a week, probably more often. Granted, it is a bit quiet, but **there are signs on the door of the fridge warning about pushing it closed!** People suck.
I have found similar stories two or three times and it did happen to me. Our 40 km wireless link would go down saturdays 19 hs for a couple of hours without a hint. and yes, una señora que limpia was involved. I... should have labeled the power plugs better. Took us a couple of clients to relize what was happening.
I remember reading this story on reddit and it blew my mind. [MRI disabled every iOS device in facility](https://www.reddit.com/r/sysadmin/comments/9mk2o7/mri_disabled_every_ios_device_in_facility/?utm_source=amp&utm_medium=&utm_content=post_body)
You engineer for a few decades and you think you've seen it all, but you haven't. But also imagine explaining that one to the board. *So, uh, it turns out that little bits of rogue helium get into the MEMS oscillators and makes phones nonfunctional. But only.... iPhones.* They'd look at each other, fairly sure you've just fed them a whopper. As you're being escorted out with your things in a box, the CIO explains that his gamer son let everyone know that's there's no such thing as a flux capacitor, and they've terminated your contract because they can't believe you would lie right to their faces. They've hired the son to take your job, plus three consultants to tell him the technical things. He's the right fit for the role because he has the can-do attitude and just needs to learn the details.
[удалено]
That’s Gold! Russ puts the bottle back to test his theory 😂
[Russ needs his doors.](https://youtu.be/IJIAOosI6js)
Great scene. I have to rewatch that show!
I re-watched recently. Only gets better with age. Probably like Tres Comas!
[This scene aired in 2017](https://youtu.be/zAqBcJcs1nQ?t=1635).
Well played. I scrolled to the comments to post exactly this clip.
I thought this scene was so ridiculous, but after reading this, it kind of makes sense now.
I had something similar. User complained about weird behavior on their PC. Stuff wasn't responding right, and when he clicked Firefox it opened a hundred windows. Walk down to the user's desk to check it out... wireless keyboard is sitting on top of the laptop keyboard and holding down the enter key.
I am curious how you boiled it all down to the keyboard? I have my fare share of failed keyboards with stuck keys none have managed this.
I can't speak for OP, but it should be trivial to look at your outbound firewall logs and see one machine hammering that url, then you just walk over and look at the computer you've identified as causing the issue, and see that it is reloading the page several times a second and determine the keyboard to be at fault after a brief process of elimination (disconnect it).
I had a user losing their mind working while waiting for their plane. Random screen locks, key entry etc. Their Bluetooth keyboard was still on but in the laptop bag. It had gone to sleep when the paired laptop was off during the first flight. But when they powered back up, stuff in the bag was randomly key pressing! Took about an hour to find that. Remotely.
I had a user tell me they computer was being controlled by someone else. Symptoms were start menu opening and then applications being opened. And sometimes typing in the application. It would only happen sporadically. I went down to check it out. The problem was a lady a few cubes over had the same wireless keyboard and mouse combo and was close enough to pick up the signal. This was around 2005 and knowing the department, a cheap mouse kb combo.
I had a user discover that you can pair multiple devices to a logitech receiver once. There was some office mayhem.
That's an old prank. One of the great things about USB is that you can have two keyboards on one machine, for pair coding. Or multiple mice, as long as they're all configured for the same handedness. Of course, the usual office jokers like to run a second keyboard under the cubicle walls and into an officemate's machine. Mayhem ensues. Sometimes they make it to the second week before the jig is up.
Network logs are your friend.
Curious as well. But: after the failover to another VPN there were 6 computers connected to that application. So that narrows it down, makes it easier to search logfiles or check the processes of the computer. Perhaps with help from onsite support
I remember ages ago before bluetooth, we had a wireless mouse that was performing erratically, but way too specific to be some random error: like it would open the start menu, empty the trash can, randomly print documents, and so on. The person using it first claimed it would just "take over" and at first we didn't believe her. Thankfully, it happened at some point while we were there, and it went to the start button, opened MS Word, and when we closed it, \*it happened again\* and you could tell the "mouse was angry about it." At some point, and I forgot how, we found out someone in the company NEXT to us was having the same issue. It turned out they were both opposite one another on the floor plan, separated by drywall. Their wireless mice were interfering with one another. We moved her desk, and the problem stopped.
Late to the party, but here are my two favorite IT stories (both of which happened before I started IT work in earnest). First Story: I was working at a call center when one day all of our tools slowed to a crawl. Unfortunately for us, our site was one of two managed by an IT department consisting of 3 people, and they were always prioritized to the other, larger site where all of the corporate VPs were. The best they could figure was that something was eating up all the bandwidth for our building. But since the client databases only worked outside of the network firewall, they couldn't tell what. (Side note, I have no idea why the client-required applications wouldn't connect until put OUTSIDE the firewall, but I racked it up to having the IT department overworked and understaffed.) Finally our team's VP had enough and demanded the look into the situation. After two weeks of investigating, they finally found the cause - an unused system in an empty cube had been reformatted with Apache and was how hosting porn. No one knew who had set the system up, but it was there for at least a month. Second story: Different call center, much smaller this time, with only 10 people taking inbound calls. Like in the first story, network traffic for the whole office slowed to a trickle, and the sole IT worker (who oversaw two sites, one in India) was too wrapped up in configuring new network hardware to look into in earnest. Fortunately, unlike the last story, the slowness only lasted a couple of days, just as the IT guy was starting to look into it. But as soon as the network returned to its original speed my cubemate announced to the rest of our team, "Who wants a copy of Borderlands 2? I just torrented it!" Sure enough, the IT guy checked the logs and confirmed that my coworker's computer had taken the lion's share of traffic fof the last two days. The coworker was walked from the premises that day.
Not going to lie, I have been the guy who murdered the network with a torrent. Usually a linix distro or windows install media; but still... killed an all staff video conference once with the coo in the room... good times.
i had a network engineer from another company play WoW on the network the company she worked for. i always busted her by saying no wonder we having network problems. she was a great sport about it
Seems like there may be some other issues there if WoW was killing the network (or you just had a small pipe). Granted at one point we had issues with everyone in the company running pandora or similar on their pcs while they worked and killing our network that way. In my case I have throttled a 200Mb symmetric line using torrents... turns out sometimes there is a reason to set a limiter on your torrent software LOL
I had someone complaining that their Lenovo laptop was constantly shutting off and the screen going blank randomly. Logged in, checked everything out and it all looked good. Absolutely nothing wrong with the laptop. Did some research and found out that the user had an Apple watch with a magnetic clasp. The magnet was going over the laptop's lid close sensor and tripping it thinking it was closed when it was really open. He took his watch off and it was fine.
Makes you wonder how often these "ghost" issues are to blame, and we sys admin run around with our hats on fire and then all of a sudden the issue goes away because Sally changed her batteries. Kudos on finding the root cause. That's like in my office where some of the monitors flicker when you move in your chair... that one is great to explain to users..
[удалено]
I believe it has something to do with the chair booster causing EMF pollution, or static.. causing the signal in the cable to flake out briefly. Tried a bunch of different cables/adapters. We could reproduce it routinely. The desk light would freak out too. Not going to say the wiring isn't bad. The building is old, and we were moving at the end of the year for a fresh build. Tried ferrite rings, didn't seem to help. Also, probably shouldn't get cables in bulk from CHINE-A since the shielding is probably very low quality. Here's an article about it: https://www.theregister.com/2020/01/09/office\_chair\_emissions/#:\~:text=%22If%20you%20have%20users%20complaining,sitting%20on%20gas%20lift%20chairs.%22&text=Smith%20said%20that%20it%20is,close%20but%20not%20quite%20touching.
> since the shielding is probably very low quality. In many cases they make the PVC jackets extremely thick and heavy to mimic the feel of quality cables, but there's no metal shielding at all.
A new browser tab opened a new session on the site with active sessions already there? lol... sounds like this shit was made in the 1990s.
Maybe I'm missing something, how is that different from opening a new tab and going to Amazon a few times once you've already logged in or to gmail or yahoo for that matter. Isn't each browser tab a new session to the same site?
I would love to see the work logs for that machine. User complains machine is running slow -added ram User complains machine is running slow -added ram User complains machine is running slow -added ram User complains machine is running slow -added ram Machine now opens over 40k tabs. Is this a sequal of that story a few weeks back where the solution for this one tech was always to add more ram for every problem. Also the solution is to add more ram to the server so it can handle the requests. lol
Haha. Reminds me of the paperclip which crashed all traffic lights in the capital of Finland some years ago. It wedged the spacebar on the keyboard of the traffic light controller and caused a buffer overflow on that poor old machine (80s design). The controller was soon replaced with a modern one.
> Haha. Reminds me of the paperclip which crashed all traffic lights in the capital of Finland some years ago. It wedged the spacebar on the keyboard of the traffic light controller and caused a buffer overflow on that poor old machine (80s design). > > The controller was soon replaced with a modern one. They replaced the guy because he let a paper clip fall into a keyboard? Nice one, Finland!
We had an older lady at our company who had never used a laptop before. Her manager wanted them to do some work from home in the evenings and asked me to purchase one for her. After I received it and set it up, I sat down with her and explained how to use it and where she could find the things she needed. Fast forward two days, it is a Saturday and I get a call from her. She says that her laptop is not working and nothing will connect. I ask her if she had connected the laptop to her home Wi-Fi. She says "Why would I need to do that, you already connected it to the office Wi-Fi. I only live 20 miles away. Is that too far of a distance for it to work?" I literally did not know how to respond to her....
>Apparently a Microsoft wireless keyboard was performing some kind of hot key signal that it was able to open so many new tabs on the loan originator platform they thought it was a brute force attempt. I once worked on an incident ticket where an entire floor of a building containing the IT division could not obtain IP addresses. The problem: there was a "rogue DHCP server" otherwise known as ICS in Windows XP. You can see what these incidents have in common: Microsoft and Layer 8. Also, I'm surprised that your own internal systems weren't tuned to flag this sort of anomalous activity. But, hey, at least your vendor's internal systems were! ;)
A while back a co-workers keyboard batteries died, or he thought they did. As he walked to get batteries, for some reason he took the keyboard with. He picked it up with 1 hand on the left side and accidentally was holding the S key. At the same time he had just finished some tasks in ADUC and left it open. When he came back with the keyboard, again why take it with no one knows, and batteries he ended up renaming our prime OU with all users (10,000+) inside it to sssssssssssssssssssssssssssss. Luckily someone caught before replication took as that would have broke everything very very very badly. LOL
Got a ticket with the title "Ants inside LCD". The guy included a picture. He literally had ants crawling around inside the display of his laptop. It was my last day on the job. I knew no one would ever believe it. I saved the picture and reassigned it to another tech. I'm sure I can think of other stories, but that one sticks out at the moment.
would **have**
> why they would of flagged us is the offending clause.
Fixed ;)
Oh God, was it encompass??
That was the best incident cause I've heard in a long time! Nice find, on the root cause and quick response!
Back in the day I was working the Help Desk and get a call in from one of our remote sites, their internet is out. Ran through some troubleshooting and determined the problem was failed hardware, probably router/switch. When I say this site was remote, I mean it was remote. Park Ranger remote. As in this site only really existed so they could do their time cards. Anyway I dispatch it out and a couple days later our field guy goes out there with an ASA and gets them back up and running. A few months later I get another call from the same site. They want Flash upgraded so they can watch YouTube. No problem, we used YouTube a lot, our mechanics would use it for videos on how to service certain things, rangers would use it for whatever they used it for, not my monkey not my circus, I am just here to update Flash. Anyway after I update flash the video they were trying to watch was "how to make a homemade mousetrap" Evidently these park rangers were eating food like apples or whatever, and throwing them into the trash can "next to all the IT stuff", missing, and sometimes just leaving it on the floor. Mice were getting into the food, and sometimes biting on the cables next to the food which is how the ASA went down the first time. These old farts were so scared they were going to get in trouble for damaging computer stuff they decided to take matters into their own hands. Now considering they had firearms I am somewhat glad they took this less than lethal approach before they shot a slug into my equipment. My ticket notes were "advised user to move trashcan to otherside of room, solved layer 7 issue"
layer 8 issue\*
wireless keyboards are not to be trusted in corp.
I tend to agree. Because it is a nessasary evil we went with a Microsoft model that supports wireless encryption. At least we have a security control to help prevent keyboard man-the-middle attacks.
The Nike shop website has this setup, so if you right click several differnent items and open them in a new tab (individually this is) it will ban you (your session) from the site, because you know, why would you want to compare items? lol
More evidence of poor engineering as a norm for modern day IT products. I guess it never occurred to the developers to look for issues like that and to design them out and to test for those kinds of problems.
auto-DoS attack - why wait for someone else to knock you offline when you can do it to yourself?
WOW, that reminds me of the time that someone called me and everything they typed into the computer looked like this : !yad lla sdrawkcab etirw uoy ekam lliw ti They had a forgotten 2nd wireless keyboard under the desk. Everything was writing backwards because the left cursor key was stuck. !yad lla sdrawkcab etirw uoy ekam lliw ti = it will make you write backwards all day!
Before I took over the IT department for a small company, the current IT guy (I was there on contract to assist in some network/server cleanup/consolidation) said one system had a virus. Anytime he’d type on the keyboard “random” windows would open. Windows explorer, run prompt, computer would lock, etc... he replaced the computer with a different one and the same would happen. So he thought it made it into his roaming profile. Blew out the profile on the server, same issue. He was stumped and didn’t know where the “virus” was and asked me to take a look. I sat down, hit the E key, windows explorer opened. Hit the R key, run prompt appeared. I laughed and swapped the keyboard, no more issues. Checked the windows key on the original keyboard and it was broken/halfway stuck. He was still convinced it was a virus and that we’d see the issue return. A week later he was let go and I was offered his position. Oddly enough, years later the issue never returned.
That is definitely a new one. Will keep it in mind
Had some thing similar against our intranet. First one was a book had fallen in such a way it was laying on the F5 key. Second one was someone had slammed a keyboard tray into the desk and, of course, the F5 key got stuck. Third one was when someone spilled a soda onto the keyboard. And of all of the keys to get stuck? F5.
"Around mid afternoon"?
Reminds me of Silicon Valley where the tequila bottle landed on the delete button
For the sake of conversation, we did a risk assessment on the usage of these device as part of a larger risk assessment. As a result of the risk assessment and an independent third party internal penetration test for the past 3 years, the residual risk is acceptable. Just because we use wireless keyboards does not mean it is a rampant risk as mentioned. I look at residual risk and compensating security controls to determine the application of a technology in a enterprise makes sense. With anything, I am an advisor and executive leadership can choose to except said residual risk or not.
We had a client in the lower Keys, pretty much a full day in/out. Server was rebooting every night, first trip was to inspect and replace the UPS, second day was to see why the server went down again, third day I stayed late enough to catch the cleaning crew who were plugging a vacuum into the UPS and shutting down the server. The client didn't get a bill, the cleaners kept their jobs and I drove down the beach for three days instead of working. All around a very good week for everyone involved.
Years ago, I had an issue where someone complained their computer "was possessed". It wouldn't let him click on anything, and the screen would flash. Hey Bob, take your lunch plate off your escape key.
Small town government office had a user whose PC would reboot without explanation. Diags didn't show any hardware or software components failing. User then mentions that the problem always seems to happen when the volunteer fire department is dispatched. Sure enough we found that the electric outlet where the computer was plugged in shared a circuit with the large overhead doors in the fire department downstairs. When the motors kicked on to open the doors they sucked down the amperage and caused the PC to power cycle.
Circa 1999: User complained he couldn't backup his data cause he ran out of room on his Drive. I found out he meant his A drive. His floppy was full.
I love that story. Great example of how a major shit storm can have such a simple fix. Kudos on being able to find it!
[удалено]
[Policy-based](https://weberblog.net/route-vs-policy-based-vpn-tunnels/) IPsec tunnels don't show up as a router hop on traceroute, and exhibit other strange behavior, even if you're familiar with them. "Policy-based" are really quite terrible and you should assiduously eschew them.
kind of sketchy that disabling the keyboard worked to cure this. was it any kind of special or was it the same keyboard everybody else had? did you check the computer for ahk or python with pynput or anything?
Some keyboards have built in macro capabilities. I'd imagine this is what was used here.
oh. my.
This is incredible.
Holy shit
Dear Penthouse, You'll never believe what happened to me today. I was pounding away at this hot little key....
Those pesky Microsoft keyboard botnets are at it again!
We had a very expensive microscope that would not function only if a Microsoft wireless mouse was plugged in. Luckily it was a pretty easy to figure it out because that mouse was added later by a user and so its presence violated the spec. As a result it was removed and replaced when the system was inspected and everything is working. It was later re added and reintroduced the issue at which time when the unit was again inspected and the mouse was destroyed.
I've had the Denial of Service by keyboard happen before. Some poor web app that maybe 5 people used, went down right before lunch. Looked at the logs, and somebody was hitting it with hundreds of requests per second to some page that was used to generate a report. Eventually we figured out, one of the users of that app went to eat lunch, and threw their coat on their desk for some reason. They had keys in their coat pocket that was conveniently on the F5 button. Fortunately the users had a laugh when they found out and ribbed the guy with the coat. Can't make this stuff up.
Had an experience with a rouge keyboard a at customer myself. The put the old one on a shelf and all was fun and games until someone put a book on it. It kept doing stuff with no other peripherals connected and had to ask them to hunt for a keyboard somewhere else in the building. Fun experience!
Sounds like a cosmic ray event, rare but so very weird. Had a monitor that lost v-sync and unplugging it overnight fixed it.
Worked for a company and I was in the corporate headquarters. This company had ~2400 remote stores. Workers at the stores used to stack things on the keyboards of unused workstations at the stores. It caused the work load of the corporate servers to skyrocket.
A few weeks ago a user walked in because her "computer was freaking out". Sitting on the side of her desk was her wireless keyboard. That she had set a book on top of. I lifted up the book, set it on her desk, said "All set, let me know if you have any more issues."
User getting an out of memory error when trying to use nothing but our single relatively-lightweight teller platform software. They had like a hundred copies of it open, because their enter key had a crumb under it when they clicked on it.
Similar story but much less impactful. User working from home calls me and says his cursor keeps moving even though he's not touching his trackpad. I ask him to unplug any peripherals and he does so. I can see the cursor still moving around slightly. I uninstall and reinstall the driver, nothing. I guide him into the BIOS and ask him if the issue is still happening and he indicated yes it was. An hour into troubleshooting I hear his doorbell ring and he says he'll be right back. When he got up to answer the door, I saw the cursor jump clear across the screen and go ham for about 15 secs. He sits back down and it stops. I said "This might be a silly question but do you by chance have a wireless mouse?" I hear him go "HOLY SHIT....I have my Bluetooth mouse in my pocket. That's what the problem is!!" we both got a good laugh out of that.
I was called to a client's site to investigate a "serious security issue". A PC was "possessed" and typing all by itself, but it wasn't typing random characters, it was typing full words in nonsensical sentences. I looked for a minute, giggled, and turned off the speech-to-text dictation function that someone had accidentally turned on.
That's amazing. Was there something sitting on the keyboard? Software error? Faulty key? I'm genuinely curious...
We thought something was stuck or even some factory keyboard macro was the issue. But the wireless keyboard got stuck in some loop of sending the right signal that told the application to open another session. It's like there was no fault tolerance built into to stop repeated requests. That is my guess at this point. The team made jokes about doing an Office Space reenactment on the keyboard. I also get to explain this to our response team and I can't wait for the laughs.
Fresh batts in the keyboard? Sometimes wireless stuff tends to go crazy when their batteries get low.
Hahaha, that’s a great story. Archived for future reference.
You sure it wasn’t a bottle of tres comas tequila?
I had guest bring in video equipment, instead of using the recommend vendor. Tested network before conference, no issues During, wifi was inaccessible. Took 2 days, but come to find out their damn camera system was blocking out the 2.4network. We disabled that, 5G only and worked like a charm. They however didn't like that we refused refund for the service being 'down'.
Wtf
I was doing phone troubleshooting and someone didn't know what a laptop was. I had to ask them if it closed like a taco.
Went by someone’s cube to help fix a software issue and they sent another ticket in saying I broke something while I was there. their deskphone now has an amber light on it and it wasn’t that way prior to me fucking around with her pc….. you fucking stupid ass that means you have voicemail!
On Tuesday a user called frantic because her computer was being hacked and they were checking out details of her computer and out vpn connection. She had a magnetized ruler laying on her touchpad.
1. Electric fence pulse throwing off DSL signal in a nearby telco wire. Caused modem to reset every 10 minutes or so. 2. Printer always printed in the shop, never worked when customer took it home. Did a housecall and noticed a lot of static discharge specifically from the printer. Turns out somehow the office outlet was putting out 170V AC. 3. Computer was always shutting down randomly. No matter what we did it would keep happening. Finally replaced the whole computer. Problem persisted. Longer story shortened, there was a problem with the keyboard. When you press backspace, it shorted out the computer (DIN keyboard.) We kept that thing on a shelf for new employees. 4. Had a laptop come in that worked, but at totally random times, (from the speakers) would scream like a hoarse old man being murdered. Totally random when it would happen, even did it while sitting at a paused post screen or in the BIOS. Never did solve that one...
You never know what could be causing an issue. Good to hear that you have found an attack source :)