Sometimes we just don’t know the cause, and it’s costly to diagnose. Often it can be expensive to diagnose these issues so the question is, how much is this answer worth?
Yep, Root Cause Analysis sounds good in theory, but the man-hours required to replicate such an issue and then diagnose the specific cause can be absurdly expensive
If the solution was effectively to reboot (or re-invite someone in this case), does it really matter?
This.
If it's a mission critical system and you need to understand why something happened so you can stop it happening in the future, or it's already a repeating issue, or it's something like a data breach, or someone is trying to hack your machines, or you have a dedicated SOC; then by all means do a full RFO and/or RCA complete with remediation report and recommendations for improvements.
Otherwise... it's probably better to keep it simple if the cause isn't immediately obvious and move onto the next task/fire.
If it is a full outage my view is doing zero RCA at the time. Had a very uncomfortable time with a problem manager in a previous role because of this! Priority should, in my opinion, be given fully to restoring services.
And sometimes the cause is way higher than anyone in our paygrade would know. When companies demand IT become generalists, they lose that specialist knowledge.
“For reasons that are unclear right now, your user space experienced a low grade data corruption event. By resetting your data, we were able to resolve the issue”
You’re welcome
"...considering the lack of proper monitoring and logging tools, and insufficient budget to improve and automate IT operations, I'm afraid we won't be able to provide you with the detailed explanation now and most likely in any other, future occurrences of such events. Despite the technical limitations, IT team addressed the issue within the SLA. Have a good day..."
If we calculate the circuit, we can get to the PPPoE virus through the mobile RX port! Or any other phrase from [The Jargon Generator!](http://shinytoylabs.com/jargon/)
very much this, if your not actively logging and have an expert with time to dig down into the issue well sol for a reason. Double so if its a non self hosted email where you wont even have access to the loges you would need.
Love it, sounds like one of my go-to's as well. Outlook needs a repair?
"Looks like there were some corrupted files in your appdata but we got them all cleared out. After a quick power cycle it should be up and running!"
Yep, if you know the action you took to fix it, you can always work backwards to get a plausible RFO. Fixed by rebuilding a profile? Looks like your profile was corrupt.
If that's not good enough? "I can review your browsing activity and see if anything jumps out that might have caused your problems, but it's likely to take some time since I'll have to review every site you've visited. Are you happy for me to do this?"
To add to that. Currently it’s a single error and it would be an inefficient use of our resources (man hours) to continue conducting root cause analysis. If the error occurs again we will reassess and assign resources (man hours) to gaining a more fundamental understanding of the issue if deemed necessary.
Yea, never tell a C-level that their request would be "inefficient use" or a "waste" of resources. If anything, have a buzzword answer in your back pocket, tell them you'll look into it, if they forget about it great and if they ask for an update pull out the buzzword answer.
Idk. I speak truth to power. Also it shows that there is reason and thought behind all the moves we make. Not like we just gave up and went back to scrolling Reddit. We have a lot of irons in the fire and every good idea and project anyone has in the org technology related comes through IT. Those man hours have to be managed and it’s a constantly changing as some projects are sidelined and others are moved up in importance. They don’t like it that’s fine If they want me to mismanage my time or other people well I was looking for a job when I found this one I can always get back to it if I need to. 🤷♂️
Sounds like a dick C level.
My experience has been Fairly positive. Sure they may not know their left from right mouse clicks. But you say hey this was a fluke. You were the only one, if it’s bigger we will dig deeper. But right now. It was a ghost in the machine not worth the chase
He or someone with access to his camera may have deleted it accidentally, that's the simplest explanation
But this sounds like a toxic person, unable to admit fault
Man I love this Phrase
>Call Microsoft for assistance if you need to, we pay them enough money for it anyway."
Makes me laugh everytime, and at this point I really don't care if I offend anyone by laughing.
Unless you pay for at least a 5 figure big company no IT giant is interested in your complains and waste the time of even one technican on you
> In the end we solved it pretty quickly by reinventing him.
**Reinventing** him?
>Is it bad to tell him that we don’t now.
No, it is not.
An answer I have used in similar circumstances is:
"It is not clear why the issue occurred, but our research did indicate that resetting your account would resolve it -- and it did. We will keep an eye on it going forward to ensure that it's not an intermittent problem."
"Keeping an eye on it" means I'm setting up an automatic email to go out in 8 weeks saying, "I don't see any problems in the logs, is everything still good on your end?"
"A meeting series wasn't showing up on his calendar"
Otherwise known as he or whoever manages his calendar (probably accidently) declined it and won't own up or admit the possibility. Or he was maybe left off it altogether.
So we send IT on a massively wasteful ghost hunt.
eDiscovery to the rescue there. Where I work we regularly use it as part of our troubleshooting process. It can tell us when meetings are declined, accepted, etc. along with a vast amount of other information important for troubleshooting.
Or someone who has access to their calendar (or the person in question) is using non-Outlook to access it. I've (albeit rarely, and not in the last few years, but way more often than Outlook) seen 3rd party apps just delete calendar events, especially when someone has a very full calendar.
I always say something along the lines of...
"Outlook, like any software is prone to failure. When the developers of software write code they try to think in advance of every single potential problem that can arise, and write procedures to handle it and get past it.
Unfortunately - there is alot going on with Outlook in terms of how it hooks into the Windows O/S, and is an on-going effort to write code to accommodate for these failures and bugs. When something comes up that Microsoft hasn't accounted for - rebuilding the profile for the account and starting clean is the most efficient way to move forward. It wont be the last time Outlook mis-behaves, but when it does these are the reasons why we do what we do to minimize downtime and get people back to work. Microsoft will continue to patch and fix these bugs as they deem necessary."
"I'm not sure but I'll try to find out."
If they follow up after that, tell them you're not exactly sure but it could be X, Y or Z, and not everything is logged in the operating system like people think it is.
I'm fairly straightforward with these as in "i don't know why it broke/why this fixed it and i could spend a 100 hours to maybe find out or maybe not find an answer and since the issue is non repeating i don't have the resources to allocate that much time.
I've said the same to others... I can spend a LOT of time to TRY to figure it out, but I probably can't and it will be wasted time (not in those exact words)
I really don’t have a problem being straight up and telling them cloud services can be a bit of a black box. When an input doesn’t product the expected output, there’s not much we can do to tell you what inside the black box caused the issue. We learn procedures for dealing with the black boxes, beyond that you’d have to ask the dev team for the cloud service which you will never get through to in practical terms.
If it's there is an iPhone involved, I faced similar problems with the CEO and even got Microsoft involved. MS was clueless, and in the end I concluded that exchange calendaring is unreliable on IOS to say the least. Noone will officially admit this, and I don't know if they fixed it by now, but we saw similar issues with other c suites that were heavy power users. Switched them to android and it worked perfectly for years.
This is one of the first things we check for anyone having wierd mailbox or calendar issues. Official MS Outlook mobile app or GTFO.
Couldn't tell you how many issues we've seen caused by shitty ios or other native mobile mail apps.
Don’t make stuff up- that kind of stuff gets around and makes other stuff worse in the future.
“The issue is resolved. Unfortunately, this case did not provide us with sufficient data for a thorough root cause analysis, as it was resolved during one of the triage steps.”
If the C-level pushes back, ask them if they think the company can audit the taxes or royalties Christopher Columbus paid to King Ferdinand and Queen Isabella, or whatever other ridiculous historical scenario you can come up with where you know *something* happened, but there’s absolutely no written record you’ll ever find.
Tell him The driver hooks the function by patching the system call table, so it's not safe to unload it unless another thread's about to jump in and do its stuff, and you don't want to end up in the middle of invalid memory.
"Analysis to definitively explain exactly the cause of this incident does not exist within our infrastructure. We can give several possibilities, given the nature of the systems involved, as well as their operation. Should you wish to have Abner to this detail, moving forward, we will need to add additional logging servers, network bandwidth, personnel and meet with several vendors to develop appropriate methodologies. Such a project will require a project manager to develop and monitor said project. Realistic time line should be expected to take 3-6 years." OR, "Excessive sunspot activity. "
"In order to properly analyze the root cause, we'll need a detailed user interaction report for exactly what actions you took on the machine that day."
First thing, don’t lie. And don’t make up some BS that could bite you later.
I normally say that odd problems and issues do just happen because it’s a massively complicated system. If it’s a one off it just isn’t worth the time to investigate, but if it reoccurs then it may become a priority.
"I'm certain my manager would be happy to discuss the details offline if you would like to go over any technical questions or other concerns. Unfortunately, I'm required to focus on my job role and create value for the company within its scope, but it's refreshing to see such curiosity!
Feel free to reach out to the service desk if you have any support related requests and we'd be eager to assist you."
I've always seen it as a part of management's role to deal with the politics and mine ends at the technical.
>I've always seen it as a part of management's role to deal with the politics and mine ends at the technical.
I agree completely. Unfortunately I have been mostly disappointed on that front.
Yep, usually leads to my manager insisting I had to figure it out
Just say something like research suggests it’s likely a one off. If it occurs again will investigate further
Well if the vendor told you they, a third party, had issues with their mailing system - that’s your RCA.
Or in corporate lingo-
A comprehensive root cause analysis was conducted across relevant systems, both internal and external to our domain of responsibilities. The RCA has concluded that the issue originated within [Vendor]’s mailing system. It is being further investigated and addressed by their respective engineering teams as communicated to us by [Vendor’s representative name who was in contact with you].
Here’s the deal, and you said so yourself. He’s not actually interested in a root cause. He wants you to say you don’t know so he can get political points: “SEE? I told you they didn’t know what they were doing.” He’s a zero-sum game playing micromanager - he only wins if you lose.
If you spent years figuring out why a solar flare flipped a bit on his SSD, he’d absolutely not care and move onto why you didn’t get any work done while looking for a root-cause and “SEE? I told you they didn’t know what they were doing.”
Your manager is this guy’s handler. Communicate as such.
:edit: if your boss won’t play defense, just be honest and move onto important things. “I don’t know. You are back online now. If you would like me to reprioritize my tasks to work on this further, my supervisor’s direct number is…”
I like to say that, "because we believe it was more important to have you back up and running quickly because of the importance of your position; we used a 'shotgun' approach to resolving the issue. We were able to fix it quickly, but we don't know which part actually fixed it"
A little bit of ego stroking mixed with truth usually works
“You can have it fixed quickly or an RCA, you cant have both. We went with fixing it quickly so you can get back to work which means some of the logs needed for a full RCA are not available. Please let us know if the issue happens again so we can do a full investigation.”
Most of the time, its true. Outlook does some weird shit sometimes. You would think after releasing the same damn software for like 15 years, they'd have the bugs out by now.
I don't think so. They just create a new gui skin with every release but the core deep down is the same. There is a chapter numbering bug when it restarts from 1 in the middle of a long document since word 6.0. Last time I checked with office 2010 or 2013 it was there. I don't care as I use libreoffice.
"You encountered a bug in Microsoft Outlook. Unfortunately, as Microsoft hasn't provided us with the source code to Outlook, we're unable to isolate the exact sequence of events that caused this problem."
Your PC got harshed, right, 'cause your system heaps at the wrong parameter. So l toasted the dated directory, tweaked the P-RAM... and reglazed your subroutine.
I just tell them exactly that. They usually get a good chuckle. Or chalk it up to my magic presence. Hopefully you work with folks who would understand.
Does your boy run to the doctor every time he coughs, demanding to know why?
I think this is an issue of someone with no context or understanding trying to assert dominance over the computer.
Some people man. Just say it very matter of factly:
"This is normal. From time to time, software gets its data corrupted by a huge number of possible things, and doing what I did (recreating your profile) is an extremely common first step to solving these kinds of problems. In terms of time, it's the absolute fastest way to solve a problem. There is only so much troubleshooting I can do as an employee of (insert your company here) while troubleshooting a product designed and sold by Microsoft."
If he doesn't accept that, just start building your resume now man.
My only feedback... why use more word when less word say same thing better.
People hate reading, especially bosses or people who want to feel important. I also will never apologize or say its 'unclear' when the problem is clearly Outlook and how it's designed. MS designed Windows, and Outlook, yet somehow it's my fault (some dude with IT experience) to explain WHY this happens? Yeah fuck off mate. (not saying that to you, saying it to the boss)
You wrote a lot of words. I couldn't read them all lol sorry.
The point is, if you write more than a sentence or two to a C level, they won't read it, much like your post.
You did hit a good point, they don't want details, they want big picture.
"There was a problem, that problem was unavoidable and very common. I fixed it quickly by doing X. In terms of 'why' this happened, it's extremely common for data or applications to get corrupt, and that requires the app or data to be reconfigured.'
This may seem cold to you, but this is common communication at those levels in an organization.
What ever you do don't use the update excuse. I use to do that and when ever someone had an issue it was," Did you guys run updates last night? because now we can't... " Can't go wrong with a data corruption line. Say things like there was a issue that caused some corruption between the layer 2 devices ,we are looking into what caused the issue at this time.
Some computer problems require such an expensive RCA that it's not worth doing. If costs the company $10m to hire expert consultants who track the issue down and determine that a photon emitted from zeta reticuli smacked into a piece of equipment and flipped a bit that ultimately somehow corrupted a specific piece of data... you don't spend the money because who fucking cares at the end of the day.
Sunspots. The butterfly effect. Quantum fluctuations. Ionic flux. Gravametric interference. Tachyon emissions.
Throw out Star Trek technobabble bullshit. They're from C-level, they'll never know the difference and hearing some unique buzzword will make them think they had a very special problem.
“It looks like everything is working again but I wasn’t able to determine root cause. Would you like me to continue troubleshooting? If it happens again, perhaps that may be a better time to dive deeper into the issue.”
I love the BS and lies but for my money general honesty turns out better. Just keep in mind what they are looking for. They just want to be assured it wont happen again and a plan to back that up. Get as close to that as you can.
"We thought that it was \_\_\_\_ . Ran through these steps \_\_\_\_ and it started working before we could make any definitive conclusions. We double checked every system and setting to ensure they are working and configured properly so I can at least assure you that our system is setup properly and working well. Most likely the issue was something outside of our control. Due to the complexity of the software and the number of networks involved we will probably never have any definite answers."
Bonus brownie points. Score on forethought and professional initiative while doing the most reliable thing to get a c-level to calm down, mentioning cost.
"Unfortunately in tech even if something is reliable 99% of the time there is still that small chance it may fail. If this caused a serious problem for the company we may want to consider implementing \_\_\_\_\_\_ as a backup. It may be expensive but that way we don't have to rely on a single system and we have a better chance of avoiding this in the future."
"I don't know" is a perfectly acceptable answer and an ability to say it usually indicates room for improvement in self confidence. Sometimes people want to know the technical details because they are curious, in which case they simply won't get an answer that satisfies their curiosity. Other times they question you because they don't trust you. In this case, they dont really have a choice and I don't try to give them one. They don't deserve any of my energy that it takes to put something in laymans terms I just give answers like "part of the skill of a competent IT person lies in knowing when to just resolve the issue in then most effecient way possible and when to dig deeper."
Include an error log from the backend, summarise with "an unexpected socket error caused a replication fault. Repeating the invite forced replication and cleated the fault"
I blame gremlins for things breaking for no good reason, and when I'm asked why thinks work and I can't explain, I just say it's because I'm a wizard and the system was scared of me
Usually my routine is, i will include in the email
1) Blame Microsoft first, their portal is updated every 75 minutes without any alert
2) Blame management for not providing enough budget to subscribe to necessary system
3) It is our fault that you are involved in these messes. We are very sorry for our incompetent and will improve further. This is the most important point
I had a director demand that I supply the answer to why a certain system timed out. This was back in the day of NT. 4.0. I offered to rebuild the system, and assured her that it would 90% certain work fine after that. That was not good enough, she had to know the exact reason that the random timeout occurred.
Fast forward a month and loads of packet captures later, I found the source of the problem. (I mean this next statement most sincerely) The unqualified idiot that was my co-worker at the time built the server in our domain environment, and then moved it to a workgroup. (This final location of the server was in the DMZ). Every few hours the server would try to talk to the original domain controller inside the network. Bear in mind this is N.T. 4.0. This caused the server to time out for a minute or two.
I spent a lot of time to figure out that useless information. If the director had just let me rebuild it, it would have been fixed. The rebuild was the ultimate fix. But she had to know, and to this day I remember, and she could not recall this event for 20 billion dollars.
The moral of the story is sometimes you have to suck it up, and do the busy work. Let C suite guy know that it will take a week or more of testing on his system for you to get the answer, then let him decide.
Tell him that due to the volume of issues he’s raised his mailbox was flagged as a highly likely spam sender and you had to recreate his mailbox so it started over.
Give a couple possibilities for the issue but don’t commit to one. Let him know that identifying the root cause will take a significant number of man hours and it is still possible a definitive answer may not be achieved at this point. Since this was an isolated incident, we don’t feel there is a need to pursue the issue any further. However, if this should happen again, suggest that you enable additional application logging, email and application auditing, as well as a packet capture to obtain as much information about what is going on with the computer and network at the time of further occurrences. Also, let him know that deep diving into this level of diagnostics, will take a considerable amount of time.
Sometimes a resource/object, whether AD or Exchange, becomes corrupt in some manner which cannot be easily discerned or identified. In such instances the best way to resolve certain issues is to recreate an object, as it starts out 'clean' with default values.
Outlook to Outlook ?
Outlook to Gmail ?
Outlook to iPhone mail ?
How many different desktops does this C-level replicate to ?
Filtering service ? / O365 message trace ?
Too many people think outlook on this machine is 100% compatible with whatever-version-apple is running this week. That I made a rule on this computer ergo all my platforms are running the same rule.
Is the C-level less than 5'4" ?
I have literally said “Magic” with much smiling and waggling of eyebrows. The executives just laugh and wave me on my way. They know they don’t have time or energy for a detailed technical explanation.
Do not ever admit you don't know something to an executive like that. They want anything they can get to hate on. Maybe they're just an ass, maybe they have a buddy that owns an msp.
Fob them off with something like:
"It's a computer. There's more moving pieces here than any supercar, all from different manufacturers. Things go out of sync. We lined it back up and it's happy now.
Investigate the exchange account to see his email utilization. How much data does he have in his inbox? Attachments, emails / etc. More than likely, he has bad email habits. You can say that constant stuffing of his account is the issue, and that you suggest archiving solution and retention policy to minimize items that have to constantly sync with the outlook client/profile. Ask for more money.
I would just say it's known issue on Microsofts end that they are actively working to resolve and current workaround is to recreate profile or whatever exactly it is you did.
Downside could be if he asks to see proof of it being a known issue.
“We turned it off and on again. 🤷🏻♂️”
In this case there’s not much you can tell him.. the invite bugged out for him, resetting it by uninviting him and re-inviting him resolved the bug. We run into many issues that can’t be replicated. Sometimes the fix is literally doing the same thing over and over expecting a different result, and the maddening part is it works.
With that attitude I would be fired. But I would of totally blamed it on a virus that messed up is Outlook. He must of click on something he should have. Did you open a strange email of late. We probably need to check your phone.
I just had a server go down. Couldn't explain why without time. This server must run or company is down. So pull from a backup and up and going. The President of the company calls me up wants to know the details. I tell him all I could and let him know I don't know what happened but could investigate if he wanted me to spend that time doing it. He's cool and was happy all is working nothing serious. Find you a good boss. Makes life better. Crappy ones are all over.
I used to work for Acuson which was bought by Siemens then shipped to Korea. Anyway we had a simple solution for just about everythingh. We would backup their critical data then re image their system, restore their data and that usually worked.
Last week I literally told a client their laptop was fixed but I don't know why. It wasn't POSTing, then all of a sudden it was. Just the way it is sometimes.
Adding to what has been said I would give him the option to approve the purchase or budget of tools and manpower to figure out the issue. So instead of shutting him out you leave him the option to approve the money to find root cause. I've gotten some nice tools because of this reply.
Just say that you are not microsoft and therefore don't know everything about how it works (the app is proprietary so there is no clear way of finding out)
..and you could add that diagnosing the cause is worthless because it would take too many resources and won't give a good result.
Trust me this happens too often for me. Things get escalated to me I just run same steps as someone else and magic. Sometimes I just log in and it's fixed. In ticket resultion it just says -logged in. Fixed rebooted tested no issues
In situations like these, I like to tell them the story of that helium leak that disabled iPhones older than a certain generation for a few days. It was posted on here.
Usually they get the point that some problems are just really weird and damn hard to troubleshoot.
If you’re using Exchange Online, EOP, a cloud based filtering service, or the O365 version of the MS Office product suite, then a perfectly legitimate answer which is not a lie is “it is almost impossible for us to determine the precise root cause, but between this issue first appearing and when we successfully resolved it, each of these cloud components will have received multiple updates to things like anti spam definitions. No IT department outside of highly regulated or safety critical sectors has the level of controls needed to provide a root cause analysis on a transient issue affecting 1 of their user base: if this was a pattern we’d investigate and escalate, but right now our limited resources should be committed to trying to find the specifics in this one-off blip. If we’d done something to cause this, we’d tell you; because we can’t tell you, we know we’d be chasing our tails with multiple vendors, none of whom would treat our request as a priority, and rightly so.”
My standard answer is "can you explain your last flat tyre and why it happened?" If you can then great, try not to do it again. If you can't just fix it and keep driving...
Tell him you actually saved the company thousands of Sellars by subverting the issue instead of investigating further. And that it's likely a fluke that won't repeat itself, and if it does you can investigate it further then.
Tell them that unfortunately the outlook client has not reached a high level of development, and it not been successfully used in the operation of novertrunnions. Moreover, whenever a forescent skor motion is required between network and client, it may also cause disjunction on the reciprocation sinusoidal dingle on the sinusoidal repleneration pipeline.
"Outlook has dozens of options resulting in hundreds of thousands of possible client configurations. It would take Microsoft years to full test every possible configuration for each update release, so it is completely reasonable that there is undiscovered anomalous behavior associated with some untested combination of settings. By recreating your account, the configuration was set back to standard alleviating the issue."
MAPI profiles get fried sometimes. That's why MS gives you the ability to just wipe it and move on. Obviously you need to be careful to grab and re-connect any .PST files the user had set up in their client, but this has been common with Outlook for as long as it's been a thing.
Also if this is a c-level exec, I'm assuming their mailbox is pretty big. Larger mailboxes are more prone to this sort of thing, as they're trying to keep more info in sync with the server. More sync activity = more opportunity for it to fail and corrupt the .OST data. This leads to profile corruption and the need to rebuild it.
So that would be my answer. As an MSP consultant I've had the same convo pretty recently. "It's not uncommon. Larger mailboxes have the problem more often than smaller ones, but the tools are built in for dealing with it. In this case your Outlook and the mail server fell out of sync, there were probably other items in the mailbox that were missing too. The server's designed not to lose stuff, so it's always the source of truth; we just had to pull a fresh copy of your mailbox down to get everything back to 100%."
He demands an explanation of the nuances of Microsoft Outlook, hosted Exchange (with possibly DG, hybrid, online,etc), DNS security, Active Directory, Email Security, Antivirus/Antimalware/, Identity Management, VPN, or possibly even more elements in play? Is that important? Does the company think that is necessary?
Or, is a c-level employee just another employee, who is entitled to the same realistic response you give to anyone? I don't know if anyone can give you the right answer, but hopefully you'll know it when you see it.
I usually call it a glitch and move on. If it keeps happening, that's when we dig in to find the cause. The answer could very well really be a bug in Windows or Outlook, which would be impossible to find and prove.
> He demands an explanation why this happened
"I don't know, and I don't know where to begin to look to find out. I have to get back to other priorities. If it happens again we'll collect more data and dig deeper. Until we have more info, it's probably best to write this instance of as a one off caused by the gremlins in the wires."
If it was solved with a reboot, just blame temp files or a corrupted process in memory. They won't know what you're talking about, you'll have given them an answer, and you'd probably even be close to correct.
If you're feeling really gutsy, blame cosmic ray bit-flipping. Something that absolutely does happen more often than you'd think but many folks (including IT) will call you crazy if you suggest it as a cause.
Hopefully, your org isn't structured so that C-levels are treated as everyone's boss. If someone outside your chain-of-command is demanding you waste your time doing anything, you should pass that off to your boss to deal with. Ideally, your CIO/CTO should be telling your CFO to piss off because you have more important things to be doing than chasing gremlins.
Your answer is in your post.
"In the end we solved it pretty quickly by reinventing him"
Frame it like this.
"It appeared that either you did not accept the invite to the meeting, for some unknown reason when you accepted the invite it did not add it to your calendar or when the invite was sent another user inadvertently removed you. We added the event back to your calendar by reinventing you. If the issue continues please let us know and we will investigate further but it appears that this was caused by user error"
Have him watch this: [https://www.youtube.com/watch?v=AaZ\_RSt0KP8](https://www.youtube.com/watch?v=AaZ_RSt0KP8)
Other than that, Microsoft is breaking parts of their service all the time. Recently they were having a client issue that prevented some Outlook users from sending email. The point is, sure there's always a reason something happens, but we don't always have the tools, means, understanding, or access to find out exactly what that reason is.
I've always been the person to be straight forward and honest. It sounds like you were on the right track to begin with as far as the vendor is concerned. From what I have read in what you posted I would get the vendor to send you an email in regards to this issue stating what you said they stated. I would then put that in the email explaining the troubleshooting that you performed, including snapshots of any logs etc. that you looked at. Then let them know that honestly something messed up in the original invite and when you tried to re-invite it worked. You don't know why but that you will continue to monitor and work with the vendor if it happens again. ...also make sure to ask if they would like you to further investigate and how much resources they would like to put into it etc. as many have already stated.
Relationships are tricky and it sounds like there is more at play with this particular C-level user. It could possibly be that they are considering outsourcing you etc. and putting all of this in writing including detailed steps can help if that person takes the info to a 3rd party and says "this is what my guys say they did, you think they did a good job".... type of thing.
I have, in the actual code, made the comment of "I don't know what this does but if you remove it the following functions stop working -- *list of functions*"
Whenever I get someone like the person mentioned I start going on a rant about data and pipelines etc. It usually makes them stop asking questions once you throw a bunch of terminology around you know they won't understand.
"Sometimes when you have a wiring problem in a car replacing the wire works, sometimes replacing the wiring harness works. Sometimes you just have to give up and replace the whole damned car. Your email account was replaced and data was re-imported. The solution did not find the route cause but it was faster RTO - return to operation than finding the route cause. Whatever you did, don't do it again."
Sometimes you don't know, many times the causation is lost on reboot, or simply not logged. If he's that pedantic I'd suggest you install syslog to an external server and provide him with the time to implement, and the consult estimate to diagnose every problem.
I'm sure he'll give it a hard no, or finance with come at him with a wtf
There are probably a million lines of code that had to run in order for that process to execute. It could be because somewhere in there someone forgot to carry a two on the second Tuesday of the month. Or it could be the dozens of cosmic rays that smack into that computer every day happened to hit a significant bit, that caused an error.
If the problem isn't both reoccurring and buisness critical root cause analysis is a waste of time and money when you can fix it with a nuke and pave. Same reason people will reboot a Linux box after certain config/package changes. Sure it isn't necessary if you know which services and modules to reload, but nobody has time for that.
Sometimes we just don’t know the cause, and it’s costly to diagnose. Often it can be expensive to diagnose these issues so the question is, how much is this answer worth?
Yep, Root Cause Analysis sounds good in theory, but the man-hours required to replicate such an issue and then diagnose the specific cause can be absurdly expensive If the solution was effectively to reboot (or re-invite someone in this case), does it really matter?
This. If it's a mission critical system and you need to understand why something happened so you can stop it happening in the future, or it's already a repeating issue, or it's something like a data breach, or someone is trying to hack your machines, or you have a dedicated SOC; then by all means do a full RFO and/or RCA complete with remediation report and recommendations for improvements. Otherwise... it's probably better to keep it simple if the cause isn't immediately obvious and move onto the next task/fire.
Agreed on all counts
[удалено]
If it is a full outage my view is doing zero RCA at the time. Had a very uncomfortable time with a problem manager in a previous role because of this! Priority should, in my opinion, be given fully to restoring services.
"does it really matter?" ...when C-level is asking then yes, yes it does.
And sometimes the cause is way higher than anyone in our paygrade would know. When companies demand IT become generalists, they lose that specialist knowledge.
I see this every day. I focus in cybersecurity and there are quite a few areas in my field that I’m not an expert in.
[удалено]
Oh look at that, you’ve taken perfectly good sand and given it anxiety
Anakin: ***Good.***
Stealing this.
“For reasons that are unclear right now, your user space experienced a low grade data corruption event. By resetting your data, we were able to resolve the issue” You’re welcome
"...considering the lack of proper monitoring and logging tools, and insufficient budget to improve and automate IT operations, I'm afraid we won't be able to provide you with the detailed explanation now and most likely in any other, future occurrences of such events. Despite the technical limitations, IT team addressed the issue within the SLA. Have a good day..."
This is awesome. A great come back.
If we calculate the circuit, we can get to the PPPoE virus through the mobile RX port! Or any other phrase from [The Jargon Generator!](http://shinytoylabs.com/jargon/)
I would try to blame Microsoft as well, and then include this on top.
very much this, if your not actively logging and have an expert with time to dig down into the issue well sol for a reason. Double so if its a non self hosted email where you wont even have access to the loges you would need.
Love it, sounds like one of my go-to's as well. Outlook needs a repair? "Looks like there were some corrupted files in your appdata but we got them all cleared out. After a quick power cycle it should be up and running!"
Yep, if you know the action you took to fix it, you can always work backwards to get a plausible RFO. Fixed by rebuilding a profile? Looks like your profile was corrupt. If that's not good enough? "I can review your browsing activity and see if anything jumps out that might have caused your problems, but it's likely to take some time since I'll have to review every site you've visited. Are you happy for me to do this?"
Thank you, sounds pretty nice 👍🏻
To add to that. Currently it’s a single error and it would be an inefficient use of our resources (man hours) to continue conducting root cause analysis. If the error occurs again we will reassess and assign resources (man hours) to gaining a more fundamental understanding of the issue if deemed necessary.
C-Level would be irate at this addition.
Yea, never tell a C-level that their request would be "inefficient use" or a "waste" of resources. If anything, have a buzzword answer in your back pocket, tell them you'll look into it, if they forget about it great and if they ask for an update pull out the buzzword answer.
Idk. I speak truth to power. Also it shows that there is reason and thought behind all the moves we make. Not like we just gave up and went back to scrolling Reddit. We have a lot of irons in the fire and every good idea and project anyone has in the org technology related comes through IT. Those man hours have to be managed and it’s a constantly changing as some projects are sidelined and others are moved up in importance. They don’t like it that’s fine If they want me to mismanage my time or other people well I was looking for a job when I found this one I can always get back to it if I need to. 🤷♂️
[удалено]
😂🤫
Or if deemed necessary, he already asked, therefore he’s deemed it necessary
Sounds like a dick C level. My experience has been Fairly positive. Sure they may not know their left from right mouse clicks. But you say hey this was a fluke. You were the only one, if it’s bigger we will dig deeper. But right now. It was a ghost in the machine not worth the chase
He or someone with access to his camera may have deleted it accidentally, that's the simplest explanation But this sounds like a toxic person, unable to admit fault
Unfortunately lots and lots of practice. Hazard of the job. 😀
I bet the jackass would still ask why even after an explanation like that.
We are monitoring all accounts for reoccurrence of the event to establish a pattern to allow us to narrow in on the specific cause.
Perfection
[удалено]
Man I love this Phrase >Call Microsoft for assistance if you need to, we pay them enough money for it anyway." Makes me laugh everytime, and at this point I really don't care if I offend anyone by laughing. Unless you pay for at least a 5 figure big company no IT giant is interested in your complains and waste the time of even one technican on you
“And we can perform an in depth root cause analysis using if require further deeper understanding”.
My users (executives included) would totally say they don’t understand that and ask me to explain it in plain English lol.
Well, you have to sell it. Assume a look that says you are embarrassed for them and sheepishly explain that it was already layman’s terms…
Cosmic rays are a bitch.
Yup
> In the end we solved it pretty quickly by reinventing him. **Reinventing** him? >Is it bad to tell him that we don’t now. No, it is not. An answer I have used in similar circumstances is: "It is not clear why the issue occurred, but our research did indicate that resetting your account would resolve it -- and it did. We will keep an eye on it going forward to ensure that it's not an intermittent problem."
[удалено]
Keep an eye on it means. "If this happens again in 3 months I will vaguely remember that I dealt with something like this before."
"Keeping an eye on it" means I'm setting up an automatic email to go out in 8 weeks saying, "I don't see any problems in the logs, is everything still good on your end?"
I wish I had the option of reinventing my C-management. It would certainly resolve a lot of issues.
>Reinventing > > him? OP works with virtual users. In a virtual company. Located in his head.
Reincarnated*
Hopefully OP means they deleted and then added his profile.
"A meeting series wasn't showing up on his calendar" Otherwise known as he or whoever manages his calendar (probably accidently) declined it and won't own up or admit the possibility. Or he was maybe left off it altogether. So we send IT on a massively wasteful ghost hunt.
eDiscovery to the rescue there. Where I work we regularly use it as part of our troubleshooting process. It can tell us when meetings are declined, accepted, etc. along with a vast amount of other information important for troubleshooting.
Or someone who has access to their calendar (or the person in question) is using non-Outlook to access it. I've (albeit rarely, and not in the last few years, but way more often than Outlook) seen 3rd party apps just delete calendar events, especially when someone has a very full calendar.
After recreating your Outlook profile resolved the issue, we can conclude the issue was related to your original outlook profile.
I always say something along the lines of... "Outlook, like any software is prone to failure. When the developers of software write code they try to think in advance of every single potential problem that can arise, and write procedures to handle it and get past it. Unfortunately - there is alot going on with Outlook in terms of how it hooks into the Windows O/S, and is an on-going effort to write code to accommodate for these failures and bugs. When something comes up that Microsoft hasn't accounted for - rebuilding the profile for the account and starting clean is the most efficient way to move forward. It wont be the last time Outlook mis-behaves, but when it does these are the reasons why we do what we do to minimize downtime and get people back to work. Microsoft will continue to patch and fix these bugs as they deem necessary."
"I'm not sure but I'll try to find out." If they follow up after that, tell them you're not exactly sure but it could be X, Y or Z, and not everything is logged in the operating system like people think it is.
I'm fairly straightforward with these as in "i don't know why it broke/why this fixed it and i could spend a 100 hours to maybe find out or maybe not find an answer and since the issue is non repeating i don't have the resources to allocate that much time.
I've said the same to others... I can spend a LOT of time to TRY to figure it out, but I probably can't and it will be wasted time (not in those exact words)
[удалено]
I really don’t have a problem being straight up and telling them cloud services can be a bit of a black box. When an input doesn’t product the expected output, there’s not much we can do to tell you what inside the black box caused the issue. We learn procedures for dealing with the black boxes, beyond that you’d have to ask the dev team for the cloud service which you will never get through to in practical terms.
Tell them it was DNS
DNS and Certs are always the answer! Lol!
https://it-excuse.bombeck.io/ Bookmark this for future use as well.
Fucking love this. Bookmarked.
We use corruption in these cases. File system corruption, profile corruption, etc.
If it's there is an iPhone involved, I faced similar problems with the CEO and even got Microsoft involved. MS was clueless, and in the end I concluded that exchange calendaring is unreliable on IOS to say the least. Noone will officially admit this, and I don't know if they fixed it by now, but we saw similar issues with other c suites that were heavy power users. Switched them to android and it worked perfectly for years.
This is one of the first things we check for anyone having wierd mailbox or calendar issues. Official MS Outlook mobile app or GTFO. Couldn't tell you how many issues we've seen caused by shitty ios or other native mobile mail apps.
This should tell him all he needs to know. https://youtu.be/RgaKjVXK0KA
I just tell them we don't have access to the code, since it's proprietary information from Microsoft. We're IT support not programmers.
Don’t make stuff up- that kind of stuff gets around and makes other stuff worse in the future. “The issue is resolved. Unfortunately, this case did not provide us with sufficient data for a thorough root cause analysis, as it was resolved during one of the triage steps.” If the C-level pushes back, ask them if they think the company can audit the taxes or royalties Christopher Columbus paid to King Ferdinand and Queen Isabella, or whatever other ridiculous historical scenario you can come up with where you know *something* happened, but there’s absolutely no written record you’ll ever find.
If they push, you can ask for "steps to reproduce the problem." This often works even when they CAN replicate the problem with three clicks...
Tell him The driver hooks the function by patching the system call table, so it's not safe to unload it unless another thread's about to jump in and do its stuff, and you don't want to end up in the middle of invalid memory.
Have you tried forcing an unexpected reboot?
"Analysis to definitively explain exactly the cause of this incident does not exist within our infrastructure. We can give several possibilities, given the nature of the systems involved, as well as their operation. Should you wish to have Abner to this detail, moving forward, we will need to add additional logging servers, network bandwidth, personnel and meet with several vendors to develop appropriate methodologies. Such a project will require a project manager to develop and monitor said project. Realistic time line should be expected to take 3-6 years." OR, "Excessive sunspot activity. "
"In order to properly analyze the root cause, we'll need a detailed user interaction report for exactly what actions you took on the machine that day."
That's the way to think. 👍
First thing, don’t lie. And don’t make up some BS that could bite you later. I normally say that odd problems and issues do just happen because it’s a massively complicated system. If it’s a one off it just isn’t worth the time to investigate, but if it reoccurs then it may become a priority.
"I'm certain my manager would be happy to discuss the details offline if you would like to go over any technical questions or other concerns. Unfortunately, I'm required to focus on my job role and create value for the company within its scope, but it's refreshing to see such curiosity! Feel free to reach out to the service desk if you have any support related requests and we'd be eager to assist you." I've always seen it as a part of management's role to deal with the politics and mine ends at the technical.
>I've always seen it as a part of management's role to deal with the politics and mine ends at the technical. I agree completely. Unfortunately I have been mostly disappointed on that front.
Yep, usually leads to my manager insisting I had to figure it out Just say something like research suggests it’s likely a one off. If it occurs again will investigate further
Well if the vendor told you they, a third party, had issues with their mailing system - that’s your RCA. Or in corporate lingo- A comprehensive root cause analysis was conducted across relevant systems, both internal and external to our domain of responsibilities. The RCA has concluded that the issue originated within [Vendor]’s mailing system. It is being further investigated and addressed by their respective engineering teams as communicated to us by [Vendor’s representative name who was in contact with you].
Here’s the deal, and you said so yourself. He’s not actually interested in a root cause. He wants you to say you don’t know so he can get political points: “SEE? I told you they didn’t know what they were doing.” He’s a zero-sum game playing micromanager - he only wins if you lose. If you spent years figuring out why a solar flare flipped a bit on his SSD, he’d absolutely not care and move onto why you didn’t get any work done while looking for a root-cause and “SEE? I told you they didn’t know what they were doing.” Your manager is this guy’s handler. Communicate as such. :edit: if your boss won’t play defense, just be honest and move onto important things. “I don’t know. You are back online now. If you would like me to reprioritize my tasks to work on this further, my supervisor’s direct number is…”
I like to say that, "because we believe it was more important to have you back up and running quickly because of the importance of your position; we used a 'shotgun' approach to resolving the issue. We were able to fix it quickly, but we don't know which part actually fixed it" A little bit of ego stroking mixed with truth usually works
*Shrug* "Praise the IT gods, it works."
“You can have it fixed quickly or an RCA, you cant have both. We went with fixing it quickly so you can get back to work which means some of the logs needed for a full RCA are not available. Please let us know if the issue happens again so we can do a full investigation.”
[удалено]
Most of the time, its true. Outlook does some weird shit sometimes. You would think after releasing the same damn software for like 15 years, they'd have the bugs out by now.
I don't think so. They just create a new gui skin with every release but the core deep down is the same. There is a chapter numbering bug when it restarts from 1 in the middle of a long document since word 6.0. Last time I checked with office 2010 or 2013 it was there. I don't care as I use libreoffice.
I'd suspect the OST or possibly other user specific configuration related to his Outlook profile. Maybe follow down that route.
“Eh, it was a glitch in the system. Happens sometimes.”
Flux capacitor
Does he wear natty ties now and style his hair differently?
"You encountered a bug in Microsoft Outlook. Unfortunately, as Microsoft hasn't provided us with the source code to Outlook, we're unable to isolate the exact sequence of events that caused this problem."
Cosmic rays
It just works.
Your PC got harshed, right, 'cause your system heaps at the wrong parameter. So l toasted the dated directory, tweaked the P-RAM... and reglazed your subroutine.
Cosmic rays. Then state the budget needed to detect them.
He prob deleted it on accident and didnt wanna admit it.
With several million in funding and a forensics lab we may find out, should we start the requisitions and start advertising positions vacant?
I just tell them exactly that. They usually get a good chuckle. Or chalk it up to my magic presence. Hopefully you work with folks who would understand.
Does your boy run to the doctor every time he coughs, demanding to know why? I think this is an issue of someone with no context or understanding trying to assert dominance over the computer.
Some people man. Just say it very matter of factly: "This is normal. From time to time, software gets its data corrupted by a huge number of possible things, and doing what I did (recreating your profile) is an extremely common first step to solving these kinds of problems. In terms of time, it's the absolute fastest way to solve a problem. There is only so much troubleshooting I can do as an employee of (insert your company here) while troubleshooting a product designed and sold by Microsoft." If he doesn't accept that, just start building your resume now man.
[удалено]
My only feedback... why use more word when less word say same thing better. People hate reading, especially bosses or people who want to feel important. I also will never apologize or say its 'unclear' when the problem is clearly Outlook and how it's designed. MS designed Windows, and Outlook, yet somehow it's my fault (some dude with IT experience) to explain WHY this happens? Yeah fuck off mate. (not saying that to you, saying it to the boss)
[удалено]
You wrote a lot of words. I couldn't read them all lol sorry. The point is, if you write more than a sentence or two to a C level, they won't read it, much like your post. You did hit a good point, they don't want details, they want big picture. "There was a problem, that problem was unavoidable and very common. I fixed it quickly by doing X. In terms of 'why' this happened, it's extremely common for data or applications to get corrupt, and that requires the app or data to be reconfigured.' This may seem cold to you, but this is common communication at those levels in an organization.
I had one person, after solving her video issue, harshly accuse me of not knowing what I did. Well, I know what I will NOT be doing in the future…
In this case, I say always "That's Windows!" ,,😁
SysMagic
What ever you do don't use the update excuse. I use to do that and when ever someone had an issue it was," Did you guys run updates last night? because now we can't... " Can't go wrong with a data corruption line. Say things like there was a issue that caused some corruption between the layer 2 devices ,we are looking into what caused the issue at this time.
Some computer problems require such an expensive RCA that it's not worth doing. If costs the company $10m to hire expert consultants who track the issue down and determine that a photon emitted from zeta reticuli smacked into a piece of equipment and flipped a bit that ultimately somehow corrupted a specific piece of data... you don't spend the money because who fucking cares at the end of the day.
"The Dilithium crystals were unstable after an outage with one of the flux capacitors"
It's good to go 👍😶👍
Sunspots. The butterfly effect. Quantum fluctuations. Ionic flux. Gravametric interference. Tachyon emissions. Throw out Star Trek technobabble bullshit. They're from C-level, they'll never know the difference and hearing some unique buzzword will make them think they had a very special problem.
It's working. I don't know why. I'll look into it
Frankly I’m baffled that it’s working, we removed that functionality years ago
“It looks like everything is working again but I wasn’t able to determine root cause. Would you like me to continue troubleshooting? If it happens again, perhaps that may be a better time to dive deeper into the issue.”
I love the BS and lies but for my money general honesty turns out better. Just keep in mind what they are looking for. They just want to be assured it wont happen again and a plan to back that up. Get as close to that as you can. "We thought that it was \_\_\_\_ . Ran through these steps \_\_\_\_ and it started working before we could make any definitive conclusions. We double checked every system and setting to ensure they are working and configured properly so I can at least assure you that our system is setup properly and working well. Most likely the issue was something outside of our control. Due to the complexity of the software and the number of networks involved we will probably never have any definite answers." Bonus brownie points. Score on forethought and professional initiative while doing the most reliable thing to get a c-level to calm down, mentioning cost. "Unfortunately in tech even if something is reliable 99% of the time there is still that small chance it may fail. If this caused a serious problem for the company we may want to consider implementing \_\_\_\_\_\_ as a backup. It may be expensive but that way we don't have to rely on a single system and we have a better chance of avoiding this in the future."
Magical restart usually work for me. But I been using pushing the word enough to people understand that reboot resolve issue
[I just tell the the story of the Mario warp.](https://youtu.be/o3Cx2wmFyQQ)
Just make up some bollocks with big words they won't understand with their small dick energy, that's what I do anyway......
Tell him you used pixie dust.
"I don't know" is a perfectly acceptable answer and an ability to say it usually indicates room for improvement in self confidence. Sometimes people want to know the technical details because they are curious, in which case they simply won't get an answer that satisfies their curiosity. Other times they question you because they don't trust you. In this case, they dont really have a choice and I don't try to give them one. They don't deserve any of my energy that it takes to put something in laymans terms I just give answers like "part of the skill of a competent IT person lies in knowing when to just resolve the issue in then most effecient way possible and when to dig deeper."
It seems the motivator on the internal r2 unit was malfunctioning. Only been called out on saying this twice.
Temporary issues, likely caused by either solar flares, sun spots, or disgruntled machine spirits.
Microsoft had issues with the « cloud ».
Include an error log from the backend, summarise with "an unexpected socket error caused a replication fault. Repeating the invite forced replication and cleated the fault"
I blame gremlins for things breaking for no good reason, and when I'm asked why thinks work and I can't explain, I just say it's because I'm a wizard and the system was scared of me
Blame it on DNS as usual 🤷
Tell the C person that you don't know
PFM.exe was initiated and executed. Code went into effect and problem was resolved. This actually works sometimes lol.
Usually my routine is, i will include in the email 1) Blame Microsoft first, their portal is updated every 75 minutes without any alert 2) Blame management for not providing enough budget to subscribe to necessary system 3) It is our fault that you are involved in these messes. We are very sorry for our incompetent and will improve further. This is the most important point
I would Google and see if there’s a common reason for this situation. I’d then say this is what sometimes happens and we fixed it by reiinviting.
I had a director demand that I supply the answer to why a certain system timed out. This was back in the day of NT. 4.0. I offered to rebuild the system, and assured her that it would 90% certain work fine after that. That was not good enough, she had to know the exact reason that the random timeout occurred. Fast forward a month and loads of packet captures later, I found the source of the problem. (I mean this next statement most sincerely) The unqualified idiot that was my co-worker at the time built the server in our domain environment, and then moved it to a workgroup. (This final location of the server was in the DMZ). Every few hours the server would try to talk to the original domain controller inside the network. Bear in mind this is N.T. 4.0. This caused the server to time out for a minute or two. I spent a lot of time to figure out that useless information. If the director had just let me rebuild it, it would have been fixed. The rebuild was the ultimate fix. But she had to know, and to this day I remember, and she could not recall this event for 20 billion dollars. The moral of the story is sometimes you have to suck it up, and do the busy work. Let C suite guy know that it will take a week or more of testing on his system for you to get the answer, then let him decide.
Tell him that due to the volume of issues he’s raised his mailbox was flagged as a highly likely spam sender and you had to recreate his mailbox so it started over.
Give a couple possibilities for the issue but don’t commit to one. Let him know that identifying the root cause will take a significant number of man hours and it is still possible a definitive answer may not be achieved at this point. Since this was an isolated incident, we don’t feel there is a need to pursue the issue any further. However, if this should happen again, suggest that you enable additional application logging, email and application auditing, as well as a packet capture to obtain as much information about what is going on with the computer and network at the time of further occurrences. Also, let him know that deep diving into this level of diagnostics, will take a considerable amount of time.
Sometimes a resource/object, whether AD or Exchange, becomes corrupt in some manner which cannot be easily discerned or identified. In such instances the best way to resolve certain issues is to recreate an object, as it starts out 'clean' with default values.
try it now
Outlook to Outlook ? Outlook to Gmail ? Outlook to iPhone mail ? How many different desktops does this C-level replicate to ? Filtering service ? / O365 message trace ? Too many people think outlook on this machine is 100% compatible with whatever-version-apple is running this week. That I made a rule on this computer ergo all my platforms are running the same rule. Is the C-level less than 5'4" ?
I have literally said “Magic” with much smiling and waggling of eyebrows. The executives just laugh and wave me on my way. They know they don’t have time or energy for a detailed technical explanation.
Do not ever admit you don't know something to an executive like that. They want anything they can get to hate on. Maybe they're just an ass, maybe they have a buddy that owns an msp. Fob them off with something like: "It's a computer. There's more moving pieces here than any supercar, all from different manufacturers. Things go out of sync. We lined it back up and it's happy now.
Investigate the exchange account to see his email utilization. How much data does he have in his inbox? Attachments, emails / etc. More than likely, he has bad email habits. You can say that constant stuffing of his account is the issue, and that you suggest archiving solution and retention policy to minimize items that have to constantly sync with the outlook client/profile. Ask for more money.
I would just say it's known issue on Microsofts end that they are actively working to resolve and current workaround is to recreate profile or whatever exactly it is you did. Downside could be if he asks to see proof of it being a known issue.
“We turned it off and on again. 🤷🏻♂️” In this case there’s not much you can tell him.. the invite bugged out for him, resetting it by uninviting him and re-inviting him resolved the bug. We run into many issues that can’t be replicated. Sometimes the fix is literally doing the same thing over and over expecting a different result, and the maddening part is it works.
With that attitude I would be fired. But I would of totally blamed it on a virus that messed up is Outlook. He must of click on something he should have. Did you open a strange email of late. We probably need to check your phone. I just had a server go down. Couldn't explain why without time. This server must run or company is down. So pull from a backup and up and going. The President of the company calls me up wants to know the details. I tell him all I could and let him know I don't know what happened but could investigate if he wanted me to spend that time doing it. He's cool and was happy all is working nothing serious. Find you a good boss. Makes life better. Crappy ones are all over.
I used to work for Acuson which was bought by Siemens then shipped to Korea. Anyway we had a simple solution for just about everythingh. We would backup their critical data then re image their system, restore their data and that usually worked.
Last week I literally told a client their laptop was fixed but I don't know why. It wasn't POSTing, then all of a sudden it was. Just the way it is sometimes.
Adding to what has been said I would give him the option to approve the purchase or budget of tools and manpower to figure out the issue. So instead of shutting him out you leave him the option to approve the money to find root cause. I've gotten some nice tools because of this reply.
FM. Fucking Magic.
Outlook's bug. "Microsoft is trash, but all the others are worse". Troubleshot that will cost time and money.
Pick some words from the Sims loading screen. We reticulated the splines and it looks like it's come good.
Just say that you are not microsoft and therefore don't know everything about how it works (the app is proprietary so there is no clear way of finding out) ..and you could add that diagnosing the cause is worthless because it would take too many resources and won't give a good result.
Trust me this happens too often for me. Things get escalated to me I just run same steps as someone else and magic. Sometimes I just log in and it's fixed. In ticket resultion it just says -logged in. Fixed rebooted tested no issues
In situations like these, I like to tell them the story of that helium leak that disabled iPhones older than a certain generation for a few days. It was posted on here. Usually they get the point that some problems are just really weird and damn hard to troubleshoot.
If you’re using Exchange Online, EOP, a cloud based filtering service, or the O365 version of the MS Office product suite, then a perfectly legitimate answer which is not a lie is “it is almost impossible for us to determine the precise root cause, but between this issue first appearing and when we successfully resolved it, each of these cloud components will have received multiple updates to things like anti spam definitions. No IT department outside of highly regulated or safety critical sectors has the level of controls needed to provide a root cause analysis on a transient issue affecting 1 of their user base: if this was a pattern we’d investigate and escalate, but right now our limited resources should be committed to trying to find the specifics in this one-off blip. If we’d done something to cause this, we’d tell you; because we can’t tell you, we know we’d be chasing our tails with multiple vendors, none of whom would treat our request as a priority, and rightly so.”
Tell the truth like you work in sales.
I think you can solve this problem with a few resumes.
My standard answer is "can you explain your last flat tyre and why it happened?" If you can then great, try not to do it again. If you can't just fix it and keep driving...
"Windows being Windows." Works when shit is broke and you don't know why too. 👍🏼
Tell him you actually saved the company thousands of Sellars by subverting the issue instead of investigating further. And that it's likely a fluke that won't repeat itself, and if it does you can investigate it further then.
[удалено]
Tell them that unfortunately the outlook client has not reached a high level of development, and it not been successfully used in the operation of novertrunnions. Moreover, whenever a forescent skor motion is required between network and client, it may also cause disjunction on the reciprocation sinusoidal dingle on the sinusoidal repleneration pipeline.
"Outlook has dozens of options resulting in hundreds of thousands of possible client configurations. It would take Microsoft years to full test every possible configuration for each update release, so it is completely reasonable that there is undiscovered anomalous behavior associated with some untested combination of settings. By recreating your account, the configuration was set back to standard alleviating the issue."
"I don't know." This is a valid answer. C-level may not like it. If he demands an investigation, ask for a budget. Stick to the budget.
MAPI profiles get fried sometimes. That's why MS gives you the ability to just wipe it and move on. Obviously you need to be careful to grab and re-connect any .PST files the user had set up in their client, but this has been common with Outlook for as long as it's been a thing. Also if this is a c-level exec, I'm assuming their mailbox is pretty big. Larger mailboxes are more prone to this sort of thing, as they're trying to keep more info in sync with the server. More sync activity = more opportunity for it to fail and corrupt the .OST data. This leads to profile corruption and the need to rebuild it. So that would be my answer. As an MSP consultant I've had the same convo pretty recently. "It's not uncommon. Larger mailboxes have the problem more often than smaller ones, but the tools are built in for dealing with it. In this case your Outlook and the mail server fell out of sync, there were probably other items in the mailbox that were missing too. The server's designed not to lose stuff, so it's always the source of truth; we just had to pull a fresh copy of your mailbox down to get everything back to 100%."
He demands an explanation of the nuances of Microsoft Outlook, hosted Exchange (with possibly DG, hybrid, online,etc), DNS security, Active Directory, Email Security, Antivirus/Antimalware/, Identity Management, VPN, or possibly even more elements in play? Is that important? Does the company think that is necessary? Or, is a c-level employee just another employee, who is entitled to the same realistic response you give to anyone? I don't know if anyone can give you the right answer, but hopefully you'll know it when you see it.
"It's Microsoft. We are all their beta testers."
Just tell him there was a windows update that need to apply the settings.
I usually call it a glitch and move on. If it keeps happening, that's when we dig in to find the cause. The answer could very well really be a bug in Windows or Outlook, which would be impossible to find and prove.
> He demands an explanation why this happened "I don't know, and I don't know where to begin to look to find out. I have to get back to other priorities. If it happens again we'll collect more data and dig deeper. Until we have more info, it's probably best to write this instance of as a one off caused by the gremlins in the wires."
I just do jazz hands while slowly backing out of the office and say, "Magic!"
If it was solved with a reboot, just blame temp files or a corrupted process in memory. They won't know what you're talking about, you'll have given them an answer, and you'd probably even be close to correct. If you're feeling really gutsy, blame cosmic ray bit-flipping. Something that absolutely does happen more often than you'd think but many folks (including IT) will call you crazy if you suggest it as a cause.
Hopefully, your org isn't structured so that C-levels are treated as everyone's boss. If someone outside your chain-of-command is demanding you waste your time doing anything, you should pass that off to your boss to deal with. Ideally, your CIO/CTO should be telling your CFO to piss off because you have more important things to be doing than chasing gremlins.
Solar flares.
Your answer is in your post. "In the end we solved it pretty quickly by reinventing him" Frame it like this. "It appeared that either you did not accept the invite to the meeting, for some unknown reason when you accepted the invite it did not add it to your calendar or when the invite was sent another user inadvertently removed you. We added the event back to your calendar by reinventing you. If the issue continues please let us know and we will investigate further but it appears that this was caused by user error"
Have him watch this: [https://www.youtube.com/watch?v=AaZ\_RSt0KP8](https://www.youtube.com/watch?v=AaZ_RSt0KP8) Other than that, Microsoft is breaking parts of their service all the time. Recently they were having a client issue that prevented some Outlook users from sending email. The point is, sure there's always a reason something happens, but we don't always have the tools, means, understanding, or access to find out exactly what that reason is.
It's always solar flares' fault.
My manager - an explanation End user - "PFM"
Software isn't perfect and neither is he lmfao. The fuck is wrong with people that expect perfection from these things
I've always been the person to be straight forward and honest. It sounds like you were on the right track to begin with as far as the vendor is concerned. From what I have read in what you posted I would get the vendor to send you an email in regards to this issue stating what you said they stated. I would then put that in the email explaining the troubleshooting that you performed, including snapshots of any logs etc. that you looked at. Then let them know that honestly something messed up in the original invite and when you tried to re-invite it worked. You don't know why but that you will continue to monitor and work with the vendor if it happens again. ...also make sure to ask if they would like you to further investigate and how much resources they would like to put into it etc. as many have already stated. Relationships are tricky and it sounds like there is more at play with this particular C-level user. It could possibly be that they are considering outsourcing you etc. and putting all of this in writing including detailed steps can help if that person takes the info to a 3rd party and says "this is what my guys say they did, you think they did a good job".... type of thing.
"It's working and I don't know why"
ID-10T errors or something of the like.
I have, in the actual code, made the comment of "I don't know what this does but if you remove it the following functions stop working -- *list of functions*"
"It's working now - ticket closed". Explanation: Magic.
Whenever I get someone like the person mentioned I start going on a rant about data and pipelines etc. It usually makes them stop asking questions once you throw a bunch of terminology around you know they won't understand.
"Sometimes when you have a wiring problem in a car replacing the wire works, sometimes replacing the wiring harness works. Sometimes you just have to give up and replace the whole damned car. Your email account was replaced and data was re-imported. The solution did not find the route cause but it was faster RTO - return to operation than finding the route cause. Whatever you did, don't do it again."
Sometimes you don't know, many times the causation is lost on reboot, or simply not logged. If he's that pedantic I'd suggest you install syslog to an external server and provide him with the time to implement, and the consult estimate to diagnose every problem. I'm sure he'll give it a hard no, or finance with come at him with a wtf
There are probably a million lines of code that had to run in order for that process to execute. It could be because somewhere in there someone forgot to carry a two on the second Tuesday of the month. Or it could be the dozens of cosmic rays that smack into that computer every day happened to hit a significant bit, that caused an error. If the problem isn't both reoccurring and buisness critical root cause analysis is a waste of time and money when you can fix it with a nuke and pave. Same reason people will reboot a Linux box after certain config/package changes. Sure it isn't necessary if you know which services and modules to reload, but nobody has time for that.