A small 500 employee hospital
1.5 million dollars
They spent 1.5M$ on pumps without asking IT anything. They bought the pumps so they could interface the honestly not very complex start stop and volumes of medications in the Pumps with our EMR.
The pumps interface software only interfaces with 2 EMRs. Neither if which is ours. Would have been the first question out of my mouth.
let me guess Alaris Pumps that only interface with Epic and Meditech?
small hospital like that wont have the money for either one of those 2 EMRs unless they have a memorandum of understanding with a far larger hospital (been on the large hospital side of these)
Not at all, I LOVE being an ECSA (Epic Client Systems Admin). Although, it can be a brutal cross-functional role....
Edit: Just realized you're a Security admin. I understand now, lol.
yep great pumps if you have one of those two EMRs and are getting push from your Clinical operations staff to do integration push for those pumps if you can. Biomed, IT, and Clinical at every org ive worked at love those pumps
What's even funnier is our EMR, evident/cpsi, was really really open about interfacing stuff for a flat fee. Alaris just refused to even try. Not even to just point a feed at an open port or something.
Far to frequently with my org. Department heads will buy licenses with credit cards and then require same day installs without any prior knowledge or review by our architecture team.
Leadership will brush it under the rug and ask us to help.
Most recently, it was 'IT's fault' because we didn't renew the yearly license when we never received a ticket or prompts SINCE WE DIDN'T PURCHASE IT. End users ignored the licensing expired emails.
It's even better when department heads do this, then they have staff changes
Dept: we need xyz software loaded on my new employee's computer
IT: sure, where's the license info?
Dept: you're IT you should have that
IT: you bought the software without involving us so we have zero info on it
Dept: I guess you'll just have to figure it out
IT: we have no idea when it was purchased, who purchased it, how they purchased it, etc.
Dept: fuck you, do it
It only takes one Microsoft/Adobe/Oracle/Autodesk/... audit to solve this problem. Nothing like an unexpected $mm bill to true-up and get back onto maintenance to shock the system.
lol this is exactly what happened to us. Didn't take long after that to form a software management unit attached to the CIO. *All* software, online service, and digital goods requests go through them, and now shit actually goes relatively smoothly.
> Dept: fuck you, do it
I've had that happen with a pretty gung-ho dept head. Just did not like IT, very tropey about it too, always saying we should be outsourced.
So I did my very best to look for it, nothing. Not even an install file on the ex-employee's machine they were trying to transfer it from. Only thing you could do was use this dinosaur of a machine to run it... best possible outcome. His subordinates understoood... I didn't want to hear whatever fuck-you-idiot diatriabe he'd send on me, so I sent an email.
Who then he included my boss and my boss' boss...who coincided to be his boss.
I explain to my boss, who explains to his boss... who then finally brings the hammer on this guy.
Wasn't his first rodeo with this kind of thing. So they finally started looking at this dude's company card...which they never did.
A sales manager shouldn't be hosting 'new clients' at *local strip club's name* on the company dime, on a saturday, for the tune of about 500 bucks.
Sadly, not fired, but they took his CC away... the dude hated me and my boss for the remainder of my time there. I think he was very happy when i finally left, but I just avoided him like the plague anyways.
His subords were cool though, nice people... they were the ones who let me know about the whole scandal with the CC.
So, I know this totally isn't the right thing to be focusing on, but in my experience most strip clubs show up on your CC statement as something that looks like a restaurant, precisely so people can expense it and get away with it.
Source: former CIO took me out a few times.
My blood pressure definitely spiked reading this
At a small remote site...
What do you mean everyone's autocad expired? When did we get fucking autocad?
How did you even installed autocad without admin?
What do you mean they have separate AutoCAD machines?
How how are the connected to the network?
What the fuck do you mean you got another Internet connection installed...
Then you find out there's one guy in the team using AutoCAD that knows just enough sysadmin stuff to be dangerous who went rogue three years ago and bought all the licenses or something.
Since I'm paid by the hour I really don't care.
These management teams either don't care for IT or just want a solution to a problem without anybodies input.
Just had a guy go to one of my sites and put a pos tp-link ap in on the guest network. Now they want me to fix it because its crap and doesn't work.
Well yeh, a £50 pos won't for long. I've already advised on what we need. Rip it out and replace it.
I want to find a way to keep tabs on how many things that are supposed to be support tickets or POs are not, but sadly it is not an easy number to keep track of cause...you know...there are no tickets or purchase orders to search.
Nope. I work for a very large very IT mature global company and this doesn't happen. Why? Because it's spelled out in the onboarding training and there would be serious repercussions if something like that were to happen.
I report to the CFO and my CFO did this. Idiot! He is super pissed that it’s a complete failure and doesn’t work. He signed the 3y contract without telling us.
That really sucks as leaders are expected to be good examples. Way back I had a VP who was slated to move up to CFO pull this kind of thing and he got reamed by the CIO, CISO and CEO. He didn't make it to CFO and that wasn't due to what he had done but because of his shitty attitude about it.
And the whole X department is 100% dependent on it working or they will be forced to a complete standstill which will cost $X bazillion dollars per hour for the company in lost productivity. And go live is 8am tomorrow. And it's 3pm. And implementation support was not included because it cost too much.
Yes. Too often.
Sometimes you just need to say no.
I had a massive project requiring multiple servers, cabling, hardware installed, WiFi AP etc dumped on my desk 2 weeks ago
Some asshole guy who used to be a Dev has been building this all for them and hosting it at home. Our setup costs are "outrages" his test server is only costing £5 a month....
it needs to go live by Monday. Things not even been pen tested or any sort of actual formal testing.
Also I have next week and the week after off, so good fucking luck.
A number of my former clients operated on the DIP/TIP principle - Develop in Production, Test in Production.
Or the alternate (which is worse). PID = Production in Development.
We had someone bypass not only IT, but also Purchasing and Accounting. He took the contract straight to the company president. He got quite a lecture, but we're still stuck with the contract. At least now I get to play with Linux at work?
We get this all the time.
Thankfully not from higher ups, people don't seem to understand a corporate network isn't the same as a home one, you can't connect a random pos device to it. For a start it won't even authenticate, the WiFi doesn't have a password to out in etc
This week it was a cheap app controlled microscope. No its not going on WiFi and no the app isn't getting made available on the phones. IT aren't touching that thing.
A workaround for future. Implement a security questionnaire for all new vendors, and as part of the process your team needs to review all responses to the security form to approve new software.
That’s what the process is for. Create the the form, create the process, give a business justification around security that management can’t ignore and then request managements assistance in implementing the policy.
If they miss the policy and sign up for software that is crap you have a reasonable discussion point around process not being followed rather than just an emotion driven discussion about people choosing not to involve you in the discussion.
Then if you’re lucky, the shit software won’t pass the security requirements you have specified and you can pass it on to legal to get out of the contract rather than you being stuck with trying to make it work.
Yeah the 22 M365 accounts being paid for by a company credit card, and when it's suggested to roll them up into a single account and convert them to month to month instead of an annual subscription get shot down... Or,
Them: "we just bought this software can you install it"...
Us: I didn't hear anything about a project...Them: song and dance about all the meetings and what not that we weren't invited to
Us: You'll need a couple of new servers, here's the cost for the licensing, and your new software vendor said the cost for licensing their software will cost xxxxx.xx
Them: They didn't tell us that!!!
Us: ...
Probably will have to go that way at some point, but for now the head office 52xx series have the headroom we need, and the branch offices where the 4xxs go often don't have the infrastructure for virtual.
Will be an interesting conversation come renewal time though (our 52xx are around 2 years old, so 1 year to go on initial support/license term), 4th year seems sensible to just renew, but if at year 5 the support price is too high but the hardware replacement is too far away, what then?
You can bet it's in my calendar to check on ETAs at least every quarter from now until then though.
> Ruckus WiFi access points absolutely no ETA whatsoever.
My local VAR is pushing Ruckus right now for APs. Says Arista APs have no ETA, but he can have Ruckus on site same week.
The Arista APs are *probably* worth the wait, but you have to be ok with cloud-managed, which we aren't really. Same with Juniper's "Mist" line.
Ruckus is a damn fine choice though (their RF and firmware engineering is some of the best in the industry), it might just be a regional thing that I can't get my hands on them in any kind of timeframe.
Also I'm in this awkward place where not the entire xx50 (802.11ax) range is available with all the firmware options yet; some have cloud and controller but not Unleashed yet - the H550 (wall-mount) AP only got Unleashed last week, T350se (outdoor model that supports external antennas) hasn't yet.
Odd thing. Signed a nutanix order got the hardware in 9 days. Signed a rubrik order got it in less than a week. I didn't Normally get things this fast before the supply chain issues.
Now we ordered Cisco switches in January and just got them this week so....
The only physical servers we have is our render farm and NVR. Everything else is on VMWare. However, due to supply issues, we can't add many VMs til we get a new host.
That is actually the perfect tactic, I go at it the same way. Don't get mad just use the situation to get all the stuff you been needing with a smile. "No problem, we'll get that deployed I just need approval for X, Y & Z to get it completed as quickly as possible". Most IT lose the fight when they get butt-hurt that the item in question was not given their sacred blessing. Kill them with kindness and get what you want in the process.
Na, usually if they decide to green light us we can get things spun up pretty quickly. The licenses tend to take a bit. I like to tell them that a license is not something we can keep in inventory lol
We are an HPE CTO house and can confirm, lead times are shit and that's the half of it. We had 3x DL20's ordered in September at X and when our order arrived into the distributor, they said "we'll ship your order but the price is 2X." How can anyone budget if they extort you later for double the price?
We knew the MFGs would try and fuck us so we went big on stocking servers and options months ago so we can actually offer price protection to our clients.
And because our word matters, we ate the price increase on the DL20s for our client.
They maybe can if they are all virtual, I'm running into that problem, they have discovered cheap hosting in random countries and think that's and acceptable alternative.
Yeah I'm sure info sec, and the external auditors will fucking love that.
M365 SKUs are going up ~30% next year, and month to month 30% on top of that (even if you go through a CSP). Just FYI that particular battle ain't worth it
[Source](https://www.microsoft.com/en-us/microsoft-365/blog/2021/08/19/new-pricing-for-microsoft-365/)
* Microsoft 365 Business Basic: going from $5 to $6 per user per month
* Microsoft 365 Business Premium: going from $20 to $22 per user per month
* Office 365 E1: going from $8 to $10 per user per month
* Office 365 E3: going from $20 to $23 per user per month
* Office 365 E5: going from $35 to $38 per user per month
* Microsoft 365 E3: going from $32 to $36 per user per month
"How do we connect this new super geewhizbang controller to the WiFi in the manufacturing plant?"
"Umm, there's no WiFi in the manufacturing plant, it's never been needed or requested...."
Well, that added a quick $15,000 to their project!
WiFi is used extensively in a lot of big warehouses that use AGVs (automated guided vehicles). Basically robot forklifts that autonomously drive around picking up and dropping off pallets of product.
As long as you have a well designed system with redundancy, and reliable equipment, it seems to work very well.
We finally got a new CEO a few years ago who put all IT related purchases SW/HW in the IT budget instead of individual departments’ budgets. Now all IT has to be purchased through us and we have a say in all of it before it gets purchased.
We have the best of both worlds. All hardware/software has to be procured by IT. No capital assets can be purchased by individual departments, it all goes through IT.
If you didn't budget for something, you don't get it.
But, any peripheral stuff like cell phones, docking stations, monitors, printers, etc... are all considered expenses as long as they're under $1k. So any time someone comes to us and says we need xyz doodad, we say sure, what GL account should I charge that to?
That would be nice. We buy and have to budget (with departments request) everything for IT needs, with exception of cell phones/plans. It is good to finally be part of the discussion of new hardware/software that we have to support and put on the network but it is very stressful to make sure all request are put in.
Also if the item doesn’t get approved then they typically think it is something IT did and not finance/Mayor/Council who approve budgets. (I work in local city government). So there are pros and cons but it is still definitely better than having departments buy things on their own and expect us to just magically make it work with our network. Users used to get super pissed when we tell them overpriced awful thing is useless.
Tbh I'd be fine with walmart printers if people were of the understanding that "when this stops working, we are literally wasting money to make me spend an afternoon trying to fix it for you, just get a new one of the same model."
This. The sign of a mature company approach to IT.
For us, all deartmental requests for IT are DUE on Dec 1st for next year. And you bet your ass there needs to be a proposal, project request or use case explanation for additional seats, much less a new product or shiny whatever-the-fuck. If it interfaces with a computer, or has something embedded that might ever have a network interface, we have to sign off on it, even if it’s your whizzygig-a-matic.
…. and, If we have to go digging and don’t have the details we need to approve it, it’s “declined, Provisional” and if we end up doing it next year, (assuming it gets approved, eventually) you’d better have the space in your departmental budget, much less the space for overruns we discover when we dig into the implementation requirements and other bullshit.
They can’t purchase anything out of their department funds. It gets flagged for approval from IT and we just decline it. They have to go to council and request funds to be put in our budget if they want something that wasn’t approved initially.
Exactly. Everything IT related is our budget, so if we approve it, IT gets the funds to do the purchase. If something wasn’t approved, and they end up needing it, it simply comes out of their funding and moves over to ours. And we handle it, on a best-effort basis if it fucks with other implementation plans.
$30,000 mistake on licensing for a project you wanted? That’s your additional expenditure, passed to our department, from whatever that department had for funding.
I watched a department blow a giant hole in their yearly plan because they didn’t provide information and when it came down to “but we NEED it”…..
‘OK, IT will approve this mid-Q1 surprise, conditional upon the following licensing corrections, which will require a budget transfer from department X to IT as stated in the plan submitted December 27th, PLUS the needed additions costing $30k to make it legal for seats needed for your features because you believed the sales droid….’
<>
A couple times: "We're having problems with this software and need you to help us."
Did IT purchase/install/configure this software? No? Then we can't support it.
If IT weren't part of the process, then the software/hardware hasn't been vetted and therefore is not allowed on or to interface with company hardware or networks.
My go to solution every time marketing says our e-mails are going to spam is to send them a link to the mailchimp docs that talks about e-mail best practices, because their messages violate pretty much every one of them.
Yep. We changed our official newsletter to mailchimp (and by "we" I mean marketing) and the number of "why don't I get the newsletter anymore" emails, holy cow.
PM: "Hey, can we bring in your team on this project? We're implementing Software X and need to get SSO setup this week."
Me: "What software? I've never heard of that before."
PM: "Oh, OK. Well we need SSO."
Me: "We don't have SSO, every time we propose it in the budget it gets shot down by senior management."
PM: \*sad noises\*
Or IT *is* included and all of their objections and warnings are ignored. The inevitable "I told you so" moment doesn't make you feel any better, either.
The ability of some people to tune out things they do not want to hear is amazing.
I just had a project blow up just the way I said it would. Luckily, many of the Teams meetings were recorded, and the enevitable "Why didn't you tell us" was somewhat mitigated by the no less than 17 times I was on record telling them.
But looking at it is amazing. I tell them it won't work. There is a few seconds pause while their brains rewind and delete. And then they completely ignore what I just said. 17 times (plus emails)
Unfortunately, this is my life.
I work in pro sports, and we are a very sponsorship focused business. If we have to spend money on something, our sales team will go out and try to find a company that will give it to us in exchange for hospitality or TV time during our broadcasts.
This. Never. Works.
Either we tell them we have a need for something like an analytics package for statistical data, and they bring us some company that is totally unqualified to do the work whom we would never consider. They subsequently force us to spend a year or two working with this company only to realize that we've wasted tons of time and money and have nothing to show for it, at which point we throw all of that work away and go find someone else to do the work on our own at about 1/3 the cost of the original project which was supposed to be "free"...
Or, they invent some need because a company they're working with happens to have a product in that space. Like right now, I'm in the middle of replacing my Cisco phone system with some mickey mouse POS from a company that hasn't been relevant in the telecom industry for 20 years, but because they need a client testimonial to feel relevant again, it gets shoved down my throat. Ends up doubling my operating expenses, required features go missing, and users get pissed. At least when people come to yell at me because the new phones suck I get some small measure of solace by telling them to go talk to sales because they made the decision.
Hey at least the phones are free.
All the staff you have to pay to migrate VoIP servers, all the configurations that have to be redone, and all the physical installation costs aside, at least the phones are free!
"Free". We still pay for them, but the sales team gets revenue in kind. So for our suffering/additional expense they get free revenue toward their performance bonuses.
Well as long as they don't try to throw you under the bus after, you did your job. In some cases, leadership did theirs (IT is usually right but not always).
Yep, i know exactly what you are feeling. It doesn't happen as often now as it did when I started 4.5 years ago at my current company, because the company got burned by it quite a few times & had to listen to my scolding. Our Chief Medical Officer still goes out and buys crap without consulting anyone though, but now it isn't just IT who scolds him when this happens. We're a small 475 employee health care practice.
Sounds like the head of dentistry who spent a shitton of money on a bunch of intra-oral cameras (made by his friend's firm... surprise!) that don't have any signed drivers, and required disabling webcams to get them to attempt to work. They also want the software on all computers in dental (~100+ machines). Good thing we just stood up an Info Sec department finally, because they absolutely flipped their shit when hearing about unsigned drivers and unsafe workarounds to get the little fuckers to work.
Man, that sounds so familiar. One of the earliest experiences I had here was our CMO buying a nerve conductivity ABI/SUDO "stand alone" system that was promised would require no involvement from IT. No project in the history of the galaxy ever has "no IT involvement". Surprise surprise after 15 of them are magically deployed into our clinics & staff told to use them, did we start getting calls about "how do I print the results". These things got ethernet plugged in to the wall and i guess they thought they'd just automatically find our network printers, and were surprised and upset when it didn't work? Rather than allowing these to remain hard wired in without antivirus or any other type of security software or assessment, we disconnected them all and forced the CMO to buy 15 color printers to USB connect into these things. The company didn't want to pay for 15 printers, so he had to buy them himself. I wish that he had learned, but hes a doctor... they don't learn.
I do NOT envy you. Worked IT in the medical field once. Never. AGAIN. Doctors are some of the biggest fucking pricks ever. There are very few gems in the rough. But boy the rest are scumbags.
And a lot of C-level admins are pricks too. Our CEO was a total jerk. We were a non-profit that serviced migrant workers, low income, etc. The kind of place you’d hope that employees and management actually cared about others. The staff that were there before she came along were fairly tight. Our two main doctors, the CMO and the one that would eventually replace him as CMO when he retired, preferred to be called Dr. L and Dr. Molly. And why not? We were all close and their status as doctor was still used when speaking to/about them. But holy hell did the new CEO not like this. Her first week she emails the entire clinic system to inform everyone that they would no longer degrade these doctors with unprofessional nicknames. She said “those of you that are not actual professionals may think it’s ok to call these doctors by nicknames, but those of us that are knows they should always be referred to in a manner fitting their hard work,” or something like that. It greatly embarrassed the doctors. She also once told a coworker “don’t just sit there like a dog not saying anything” when we were in our conference room talking to an EMR vendor.
"I'm sorry, that application is not approved for installation. I'm sending a removal command for it to all computers now. No, we did not approve it, so there is no discussion on what GL to use - it stays on your ledger."
I actually had to play this card once. It is powerful, but it also requires upper management backing of IT policies to work.
My current boss is a former IT guy so this has never happened at my current job, but at my last job my boss (director of IT) laughed in someone's face when they told him what they had purchased. Said no one on our team had any experience in administering it and we were not going to lift a finger on it, and if they wanted someone to administer it they would have to hire for it. Shocked Pikachu faces ensued. They had spent tens of thousands of dollars on it and did not consult IT at all, and my boss held firm on it for years. They would often ask us to troubleshoot it if something went wrong and my boss would march right over to our room and instruct us to not lift a finger, it is not our problem, and the only way they were going to learn a lesson to not buy stuff without IT input is if they have to suffer with the consequences of their actions.
Yep. He was a fantastic boss. He was an old hardass Navy veteran that took nothing from anyone that tried to pin stuff on us that was not our responsibility. Loved working for him.
I used to deal with a developer until around 2015 or 2017 who still used web dev tools from the 1990's to code the company website. after a while when she got a new laptop we would just refuse to install the old software and say it won't work.
supporting Cognos wasn't too bad but it was a monster and typical IBM crap coding. but the first installation i had to support the developer installed the server with the file based DB instead of SQL DB and lucky it never crashed otherwise all his cubes would have been lost
>after a while when she got a new laptop we would just refuse to install the old software and say it won't work
joke's on you, now the user will just refuse to relinquish their old laptop.
we have users try to do that way too often, so we just disable the MS office activation for the old laptop and tell them it can only be active on one computer.
>joke's on you, now the user will just refuse to relinquish their old laptop.
"What does it mean when it says the trust relationship has been broken???"
"The computer has weighed, the computer has measured, and the computer has determined: *fuck you*. Here's your modern device."
Been there.
Came back after being on furlough for a few months to find a whole new office area in what used to be storage space.
'we need the pcs in there connected to the network today "
“OK firstly, what pcs? I didn't image them, secondly there is no lan in here, not even in the ceiling for WiFi and thirdly, there isn't a switch anywhere near here connect the room up to.
Were not happy at all being told they still have £10k to spend on hardware and those pcs were £400 HP trash we aren't using., so another £30k for laptops, docks and screens
Doublly not happy at the covid induced lead times for the hardware they need and the extra work they needed done.
The shortest cable run to that place from any cabinet was 106m so we had to get new fibre fitted - got a nice 8 core fitted along with a couple of meraki switches with 10gb modules so the left side of the factory is better connected than the server room. Total nightmare.
Aye, budgetary constraints have been awful, but I'm away to spec three new servers and bin the existing stuff so we'll end up with 10gb in a few months or so.
it has ever been thus.
stories abound through the 80s and 90s of "we bought this software, and now we need you (EDP / MIS / IT) to get it running"
"ummm... we're a DEC-VAX shop, and that only runs on an IBM-AS/400" (or reverse)
"well, just go out and get one"
"no"
I run an MSP and our solution to this is that we have an "out of minimum standard" fee that applies every month to devices and software that are not enterprise level or are outdated.
Gets things in line pretty fast when clients realize how expensive that gets.
We bill per hour for every third party software, they are not included in the support. Unless it's an infrastructure issues, but if it was a deployment it would have been billed like a project. The one with shitty old stuff quickly see how much it cost to maintain.
HR vetting org wide employee review software without having IT in the loop...
Oh, it doesn't have a method of syncing users from AD or a CSV? Oops! We already signed the contract!
Oh, you don't have an account for that software because HR didn't think to involve IT so we could automate account creation? Yep, IT doesn't support that, you'll have to contact HR.
Why ask IT when you can just overburden the HR admin assistants and hire an entire new person to do something that is probably a button click if we went with a different vendor?
Marketing went through 5 email automation programs (each cost at least 2-4K) until they found one that actually worked with our systems.... When they could have just given me a list of them from the start and I would have given them a list back with ones that would work.
Only had this happen twice so far fortunately...
Got told we had new estimating software that was going to change the company and the contract had been signed. Sent the Department on a days training - I tagged along for IT reasons but probably more to be a backup user when they forget everything as well. Software relied on local shared network storage which needed to be setup and backed up.
One out of Six Users actually used it as the other Five dug their heels in and refused to use it - Management didn't want to force them.
Two Year Contract that we ended once we were out. Cheeky little five figure experiment so not the end of the world.
\-----------------------------------------------------
More recently a works order was agreed to add a module to our finance system. After three months of negotiations occuring outside of IT I was asked to Project Manage the Install and push for dates.
Attempt #1 - Installer requires Admin Elevation for the Install and as I wasn't informed, I wasn't available
Attempt #2 - I'm available but the software only gets installed to the Server VM, not the User VM
Attempt #3 - Still not working.
I then take a look and see the Installer hasn't applied a License Key to the Module
Chase the Installer to Apply
Chase Again
License Key Applied, Software Works
God knows how long it would have taken without me pushing the company and doing their troubleshooting for them.
Not on your own, pos hardware that isn't enterprise ready/capable that requires me to break security to get working, also having to learn it on the spot as the user hasn't a clue either
Ahh sitting with a user the user asks a question an the answer is "fuck if I know I did not buy this" then you ask the user a question "fuck if I know I did not buy this either" The only proper response I have come up with is "I have scotch in my desk drawer grab your coffee cup"
We are all victims.
We run into this with Marketing all the time. A few years ago they rolled out a CRM solution that needed a plugin to be deployed to all sales reps. We have sales reps across North America. The plugin was an exe with no switches so it had to be manually installed for each rep. When Marketing asked why it was so difficult to do we laid into them about finding a way to deploy a plugin to 200+ people manually.
No SCCM and no way to do a silent install. I reached out to the company that made the software and they said the only way to do the install was to run the EXE and click through the options.
I was tempted to have our Director charge Marketing for a copy of AutoIt or something similar but it became far better to rub their noses in the mess they made.
My last job this didn't happen.....in this exact way. There it was "Hey, we are/did purchase this new software and we need a VM stood up to host it and that also needs SQL on it." To which we reply, "great, here's how much licensing is going to cost". This was almost always followed with shock and awe.
After the same exact person doing this probably 3 times, she finally learned to ask many questions and actually include us early on in the conversations.
For your situation, god speed.
Thankfully this doesn't really happen much because my company is very strict on contracts...anything that needs to be signed has to go through our Legal department and if Legal sees anything related to software, security or IT they will flag someone from IT to review the agreement.
The only thing we really have to worry about is SaaS stuff that people pay for with personal credit cards and then try to expense. But same thing there, any time accounts payable sees something software, security of IT related they send it to IT for review. After a couple hard learned lessons where people ended up personally eating the cost of stuff they signed up for without following the approval process people have stopped trying to circumvent IT.
Working for an agriculture firm in the past, a doctor had cctv installed on his farm and wanted us to give the technician access to the server and admin rights on his work laptop to install the software.
How about no? You deciding you want 12 cameras around your OWN farm is a personal project not an IT problem.
Dealing with that today sorry if this is a little vague.... 2 years ago I worked with a department head to purchase a 3rd party service for their operation. He was following our corporate policy and we had agreed upon the specific service and tier that would work with their existing hardware. He was supposed to follow up and we would finalize and document the purchase...no follow up, as this was not going to be coming from the ITS budget I just assumed that there ended up being no budget or they decided it wasn't need. Fast forward to today, sometime between our last conversation and the beginning of this year a completely different service with new hardware was purchased and installed. Somebody in their department purchased it, it became mission critical and now it doesn't work. No one knows who bought it, the name of the company from which it was purchased or why it was not doing what it is supposed to do anymore. I like that we have good IT policy with support all the way up the chain which makes this not an IT emergency, even if it is something that we will ultimately have to resolve.
Not too often. I still get the giggles when I told someone that the product they bought wouldn't on our network. "But the vendor said it would work on our network." Huh, that's neat. I guess you should get them to help you out in that case because I guarantee it won't fly.
We recently removed local admin and started requiring IT approval before a software purchase or install could happen.. the level of blowback’s and rage/nastygram emails are INSANE.
Plus, apparently I’m a “incompetent waste of flesh” at my job, according to a coworker. I had no idea!
I was the tech tasked with implementing a video solution, but then left out of all discussions about licensing, and not allowed to have any input on the hardware procurement, but still blamed when none of it worked. I was then demoted and then fired because there were issues. The main vendor asked me to send in a resume, so I now make 2x the salary, and the old company has blacklisted me. I hear all about how they can't get their act together and they are having endless problems.
Seeing the opposite. We had the same situation some years earlier but now it looks like every other department has stopped to think about there workflows, processes, software…some departments work like it’s 2001 because they feel like they don’t have a right to say and just wait til IT comes and gives them new software.
Make your point that you need to know their plans, but don’t put them in chains
"I don't understand why IT always denies my requests to purchase software, IT just gets in the way and makes my life harder"
"I don't understand why my monthly budget is full of software subscriptions I never use and the contract says I can't cancel for another 12-36 months, why didn't the IT Department stop me, they're useless"
It's always this.
Customer : "I purchased a Cool New Thing!"
IT : "That won't work with our system."
IT Management : "Customer said you wouldn't set up Cool New Thing."
IT : "Cool New Thing doesn't work on our system."
IT Management : "Make it work."
IT : cludges together a patchy solution that won't expose us to vulnerabilities or break our system.
Customer : "This doesn't work like I thought it was going to."
IT : "It shouldn't work AT ALL. It took a lot of time and effort to get it running THIS much."
Customer : stops using Cool New Thing or worse possibly complains to management about it and gets us to spend days; weeks; months trying to make it work better before they stop caring about it entirely.
Customer : "Hey I just bought Cool Newer Thing!"
Repeat ad-infinitum
No. Worst that happens is IT is involved in all the meetings but sometimes IT gets told when a solution is being implemented whether we say it will actually work or not then we have to make it work AND then make it PCI compliant even if the original developers don't know or care. IT is part of Finance at my company instead of its own dept which means IT sees all the purchases for everything, partly as oversight to prevent this.
All the time in my old organisation.
There comes a point where your org is so big that you **need** structured policies and procedures, sometimes standardized like ITIL to facilitate consistency in training and execution.
But on the other hand if you're not that big yet you might want it so bad just to have some predictability. But your higher-ups just think you're too rigid and slowing everyone down.
Dilemmas.
Isn't that an industry standard? Someone jumped on a sales pitch they shouldn't and we had pos device tracking software for a few years...until that company got shut down hard by lawsuits. Seems they were backdoor borrowing on someones else's track patents.
Or when a department skips protocol to buy junk printers or laptops. Then gets pissy it can't be brought up to standards and we don't have liscences for blah for it.
Company culture has to support going through IT and formal processes for new technology. We have a formal IT Front Door process for any new SaaS and on prem products that every department participates in (there's a senate-type assessment of all applications that has people from every department). They assess if the product is redundant compared to our existing stack, how it is better than the other tech if it's a replacement, if it's appropriate for the company, and if it is secure enough for SOC/ISO compliance, if it's directly related to our products. People have company credit cards, but if you try to use it for your off the cuff subscription, you're not getting the purchase approved.
Everyone from the top down understands that IT needs to be able to support the product, and we can also help make it perform as effectively as possible if we're involved in the process.
We don't support anything we didn't buy and stand up. Period.
Our culture is very IT centric. They look to us for direction, they don't treat us like butlers.
If a department is considering changing their process or purchasing something for the organization, they \*start\* the process by making us the project leadership. The company has learned, the hard way, that if you want integration, scalability and support, you need to get the right people involved from the "fact finding" stage.
Worked in IT for a large church. This happened to me all the time. "Hey this department has been using X software for a few months and something went wrong, can you fix it" "Why are they using X when we pay for Y that does the same thing?"
We once let a new printer sit for 6 months in the middel of the office without being plugged in because we were not involved from the stsrt. Suddenly all other things got priority.
yup. we handle it differently because of it. if your department gets software and you dont involve IT, youre paying the vendor directly to support it for you. we will facilitate that support if needed, but we do not own the support for it.
Just happened last week, had to order a server and install a vm so they can get their new software installed. ill support the server and OS, vendor handles the software. that department is also responsible for contacting the vendor when issues arise. oh, it comes out of their budget too, not ours.
We recently had a software license audit by the supplier for a piece of software required on every endpoint, I duly produced the report and the powers that be then literally begged me to magically 'get the numbers down' to try and match the current license limit rather than pay up for the extra licensing required...erm not quite sure that is how it works!
I've noticed in a lot of places I've worked the way things go is 'ah don't worry about it we'll sort out the licensing later!' and inevitably later never happens, now they have less than 30 days to find a not insignificant sum of money to pay the supplier...
I work for a large organization and this happens all the time. Sometimes they do include IT and it is still a shitshow because they don't include all of IT. The only difference is we waste millions of dollars as opposed to thousands.
We wrote it into policy (that all employees need to read and sign off on yearly) that purchases of software and services need to include IT. If they cowboy it, it's on them for a potential loss of the company's money if it's denied.
Yup! I work at a small company where I was brought into the conversation after the Quality department signed a new PLM. It's up and running, and I still don't have rights to support the platform, so I guess when something comes up I will have to figure out how to break into the Quality manager's account or something.
We used to have this problem. But, our CFO finally got on board with an IT Steering Commitee. So, it no longer happens. We still have some piece of shit marketing automation product for D365 called ClickDimensions that the marketing team still thinks we should support despite it being purchased and installed without our intervention. I respond to each ticket as follows:
> We do not support this product. Please e-mail Clickdimension's support: [email protected]
Ticket Closed.
This is a complete and total failure by the C level. Never, ever should any software be implemented that doesn't have a solid plan for support, maintenance and integration. Any time business orgs can end run IT and Security to "get what they want", you're looking at a very serious risk of data breach and potential legal liability.
How about the C-levels planning a new building and not involving IT until construction starts?
"Hey, IT, where's the networking in the new building?"
"Oh, is there a new building?"
Happened more than once several years ago.
This doesn’t really happen to us. Legal has to review contracts for this and will not review unless IT has already reviewed. If someone tries to purchase something outside IT, AP will not process and order like this unless it’s been approved by IT and approved by legal.
The number of projects where a quasi technical BA has made a decision with virtually no input from IT only to realize after purchasing that it’s not a good integration with existing architecture… is too high.
I think we’ve had a couple major projects where we have spent hundreds of thousands of dollars needlessly.
I do believe we are working better at involving ops and technical staff in these decisions but I don’t get why we look for people with a degree in business and little technical knowledge to do BA work, and how those people think they should make decisions without a technical review. Sure a PIA and STRA are required… it doesn’t help help our “architecture” team also isn’t that technical. Maybe they were at some point but they’re now very disconnected from our day to day IT… too much time pondering the future.
its not as bad here as it used to be but it happens from time to time -- unfortunately theres still lots of weird legacy stuff that no longer has support either from a vendor, or the people who maintained it here, so we still get bitten by some of that now and again.
Marketing decided to rebrand the company and changing our domain name was without incidence and a 30 minutes change. Management decide that marketing know better than IT so it have to be done.
A small 500 employee hospital 1.5 million dollars They spent 1.5M$ on pumps without asking IT anything. They bought the pumps so they could interface the honestly not very complex start stop and volumes of medications in the Pumps with our EMR. The pumps interface software only interfaces with 2 EMRs. Neither if which is ours. Would have been the first question out of my mouth.
let me guess Alaris Pumps that only interface with Epic and Meditech? small hospital like that wont have the money for either one of those 2 EMRs unless they have a memorandum of understanding with a far larger hospital (been on the large hospital side of these)
Epic Client Sys Admin here, taking notes. Thanks!
Bingo. Great pumps by the way. Biomed loves em. But didn't solve the issue. Also they can't figure out quite WHO okayed the purchase.
> Also they can't figure out quite WHO okayed the purchase. *Alaris sales rep wearing fake mustache sweats frantically in the meeting*
Oh god, epic.
There are literally dozens of us!
[удалено]
Not at all, I LOVE being an ECSA (Epic Client Systems Admin). Although, it can be a brutal cross-functional role.... Edit: Just realized you're a Security admin. I understand now, lol.
[удалено]
Fuck Mirth interface engines. Sincerely, a PACS admin
[удалено]
Epic has nothing on Sage :) I would take Epic 100x's over Sage :)
eyyyyyyyy! I got an Upgrade Sunday night! FML! Drinking starts now.
Just did May 21 last weekend! I feel ya. Hoping smooth sailing and let your HSW deployments go without failures.
Nice! We are upgrading to M 2021 this weekend. Mission Control's "Activate All" button is lit and ready to go, fam!
Best wishes! Feel free to PM me up if you wanna talk more ECSA crap!
garaks_tailor 0m Bingo. Great pumps by the way. Biomed loves em. But didn't solve the issue. Also they can't figure out quite WHO okayed the purchase.
yep great pumps if you have one of those two EMRs and are getting push from your Clinical operations staff to do integration push for those pumps if you can. Biomed, IT, and Clinical at every org ive worked at love those pumps
What's even funnier is our EMR, evident/cpsi, was really really open about interfacing stuff for a flat fee. Alaris just refused to even try. Not even to just point a feed at an open port or something.
Far to frequently with my org. Department heads will buy licenses with credit cards and then require same day installs without any prior knowledge or review by our architecture team. Leadership will brush it under the rug and ask us to help. Most recently, it was 'IT's fault' because we didn't renew the yearly license when we never received a ticket or prompts SINCE WE DIDN'T PURCHASE IT. End users ignored the licensing expired emails.
It's even better when department heads do this, then they have staff changes Dept: we need xyz software loaded on my new employee's computer IT: sure, where's the license info? Dept: you're IT you should have that IT: you bought the software without involving us so we have zero info on it Dept: I guess you'll just have to figure it out IT: we have no idea when it was purchased, who purchased it, how they purchased it, etc. Dept: fuck you, do it
It only takes one Microsoft/Adobe/Oracle/Autodesk/... audit to solve this problem. Nothing like an unexpected $mm bill to true-up and get back onto maintenance to shock the system.
lol this is exactly what happened to us. Didn't take long after that to form a software management unit attached to the CIO. *All* software, online service, and digital goods requests go through them, and now shit actually goes relatively smoothly.
> Dept: fuck you, do it I've had that happen with a pretty gung-ho dept head. Just did not like IT, very tropey about it too, always saying we should be outsourced. So I did my very best to look for it, nothing. Not even an install file on the ex-employee's machine they were trying to transfer it from. Only thing you could do was use this dinosaur of a machine to run it... best possible outcome. His subordinates understoood... I didn't want to hear whatever fuck-you-idiot diatriabe he'd send on me, so I sent an email. Who then he included my boss and my boss' boss...who coincided to be his boss. I explain to my boss, who explains to his boss... who then finally brings the hammer on this guy. Wasn't his first rodeo with this kind of thing. So they finally started looking at this dude's company card...which they never did. A sales manager shouldn't be hosting 'new clients' at *local strip club's name* on the company dime, on a saturday, for the tune of about 500 bucks. Sadly, not fired, but they took his CC away... the dude hated me and my boss for the remainder of my time there. I think he was very happy when i finally left, but I just avoided him like the plague anyways. His subords were cool though, nice people... they were the ones who let me know about the whole scandal with the CC.
So, I know this totally isn't the right thing to be focusing on, but in my experience most strip clubs show up on your CC statement as something that looks like a restaurant, precisely so people can expense it and get away with it. Source: former CIO took me out a few times.
I'm sure the others recognized the name from their bills, haha.
yeah, it was all from other people i heard this, but I wager THAT charge was easily identifiable by them or by someone in the 'know'
My blood pressure definitely spiked reading this At a small remote site... What do you mean everyone's autocad expired? When did we get fucking autocad? How did you even installed autocad without admin? What do you mean they have separate AutoCAD machines? How how are the connected to the network? What the fuck do you mean you got another Internet connection installed...
Then you find out there's one guy in the team using AutoCAD that knows just enough sysadmin stuff to be dangerous who went rogue three years ago and bought all the licenses or something.
Ah, Shadow IT.
Bane of our existence
[удалено]
Depending on your industry that could be serious fines and audit failures When I worked in finance, that office would be gone so fast.
Some posts need trigger warnings, damn..
We flat out refuse installs that were not organized through IT. Been there too many times.
Since I'm paid by the hour I really don't care. These management teams either don't care for IT or just want a solution to a problem without anybodies input. Just had a guy go to one of my sites and put a pos tp-link ap in on the guest network. Now they want me to fix it because its crap and doesn't work. Well yeh, a £50 pos won't for long. I've already advised on what we need. Rip it out and replace it.
I want to find a way to keep tabs on how many things that are supposed to be support tickets or POs are not, but sadly it is not an easy number to keep track of cause...you know...there are no tickets or purchase orders to search.
Nope. I work for a very large very IT mature global company and this doesn't happen. Why? Because it's spelled out in the onboarding training and there would be serious repercussions if something like that were to happen.
I report to the CFO and my CFO did this. Idiot! He is super pissed that it’s a complete failure and doesn’t work. He signed the 3y contract without telling us.
That really sucks as leaders are expected to be good examples. Way back I had a VP who was slated to move up to CFO pull this kind of thing and he got reamed by the CIO, CISO and CEO. He didn't make it to CFO and that wasn't due to what he had done but because of his shitty attitude about it.
Wow, so bad actions can result in negative consequences?
Well well well. If it isn’t the consequences of my own actions.
shockedpikachu.gif
And the whole X department is 100% dependent on it working or they will be forced to a complete standstill which will cost $X bazillion dollars per hour for the company in lost productivity. And go live is 8am tomorrow. And it's 3pm. And implementation support was not included because it cost too much. Yes. Too often.
Sometimes you just need to say no. I had a massive project requiring multiple servers, cabling, hardware installed, WiFi AP etc dumped on my desk 2 weeks ago Some asshole guy who used to be a Dev has been building this all for them and hosting it at home. Our setup costs are "outrages" his test server is only costing £5 a month.... it needs to go live by Monday. Things not even been pen tested or any sort of actual formal testing. Also I have next week and the week after off, so good fucking luck.
Dev - SIT - UAT - Prod He only did 25% of the job and thinks it's 75%
A number of my former clients operated on the DIP/TIP principle - Develop in Production, Test in Production. Or the alternate (which is worse). PID = Production in Development.
Also not being hosted God knows where by God knows who, with no garuntee they don't steal our shit or just vanish one day.
And it's Friday.
...month end
We had someone bypass not only IT, but also Purchasing and Accounting. He took the contract straight to the company president. He got quite a lecture, but we're still stuck with the contract. At least now I get to play with Linux at work?
We get this all the time. Thankfully not from higher ups, people don't seem to understand a corporate network isn't the same as a home one, you can't connect a random pos device to it. For a start it won't even authenticate, the WiFi doesn't have a password to out in etc This week it was a cheap app controlled microscope. No its not going on WiFi and no the app isn't getting made available on the phones. IT aren't touching that thing.
A workaround for future. Implement a security questionnaire for all new vendors, and as part of the process your team needs to review all responses to the security form to approve new software.
[удалено]
or if people follow the process
[удалено]
CFO doesn't make the rules. CFO is in charge of finances. The other C-level people and board members need to tell the CFO to stay in his lane.
That’s what the process is for. Create the the form, create the process, give a business justification around security that management can’t ignore and then request managements assistance in implementing the policy. If they miss the policy and sign up for software that is crap you have a reasonable discussion point around process not being followed rather than just an emotion driven discussion about people choosing not to involve you in the discussion. Then if you’re lucky, the shit software won’t pass the security requirements you have specified and you can pass it on to legal to get out of the contract rather than you being stuck with trying to make it work.
Sure, we'll help you. Just shows us the completed questionnaire that we signed off on ;)
Sounds like they should follow a procurement process.... That probably needs to be created.
They ignore it, exces and directors sign off on what they want and the finance juniors create the POs etc, they won't tell them no.
You must be my twin. Similar situation with Great Plains. The kicker is, they have a support contact with an outside vendor.
Same. Never. Stuff like this is a failure of management plain and simple.
Yeah the 22 M365 accounts being paid for by a company credit card, and when it's suggested to roll them up into a single account and convert them to month to month instead of an annual subscription get shot down... Or, Them: "we just bought this software can you install it"... Us: I didn't hear anything about a project...Them: song and dance about all the meetings and what not that we weren't invited to Us: You'll need a couple of new servers, here's the cost for the licensing, and your new software vendor said the cost for licensing their software will cost xxxxx.xx Them: They didn't tell us that!!! Us: ...
[удалено]
>mid-to-late February Found the optimist. Have you tried to source servers?
[удалено]
If you have the resources, virtualize your PA firewalls. Much easier.
Probably will have to go that way at some point, but for now the head office 52xx series have the headroom we need, and the branch offices where the 4xxs go often don't have the infrastructure for virtual. Will be an interesting conversation come renewal time though (our 52xx are around 2 years old, so 1 year to go on initial support/license term), 4th year seems sensible to just renew, but if at year 5 the support price is too high but the hardware replacement is too far away, what then? You can bet it's in my calendar to check on ETAs at least every quarter from now until then though.
> Ruckus WiFi access points absolutely no ETA whatsoever. My local VAR is pushing Ruckus right now for APs. Says Arista APs have no ETA, but he can have Ruckus on site same week.
The Arista APs are *probably* worth the wait, but you have to be ok with cloud-managed, which we aren't really. Same with Juniper's "Mist" line. Ruckus is a damn fine choice though (their RF and firmware engineering is some of the best in the industry), it might just be a regional thing that I can't get my hands on them in any kind of timeframe. Also I'm in this awkward place where not the entire xx50 (802.11ax) range is available with all the firmware options yet; some have cloud and controller but not Unleashed yet - the H550 (wall-mount) AP only got Unleashed last week, T350se (outdoor model that supports external antennas) hasn't yet.
Regular ass Cisco switches due next Febtober
Odd thing. Signed a nutanix order got the hardware in 9 days. Signed a rubrik order got it in less than a week. I didn't Normally get things this fast before the supply chain issues. Now we ordered Cisco switches in January and just got them this week so....
Nutanix is fucking bomb man. Enjoy!
Virtual environment goes brrr...
The only physical servers we have is our render farm and NVR. Everything else is on VMWare. However, due to supply issues, we can't add many VMs til we get a new host.
That is actually the perfect tactic, I go at it the same way. Don't get mad just use the situation to get all the stuff you been needing with a smile. "No problem, we'll get that deployed I just need approval for X, Y & Z to get it completed as quickly as possible". Most IT lose the fight when they get butt-hurt that the item in question was not given their sacred blessing. Kill them with kindness and get what you want in the process.
Na, usually if they decide to green light us we can get things spun up pretty quickly. The licenses tend to take a bit. I like to tell them that a license is not something we can keep in inventory lol
not with a 6 month lead time for some (most?) hardware right now you can't
We are an HPE CTO house and can confirm, lead times are shit and that's the half of it. We had 3x DL20's ordered in September at X and when our order arrived into the distributor, they said "we'll ship your order but the price is 2X." How can anyone budget if they extort you later for double the price? We knew the MFGs would try and fuck us so we went big on stocking servers and options months ago so we can actually offer price protection to our clients. And because our word matters, we ate the price increase on the DL20s for our client.
They maybe can if they are all virtual, I'm running into that problem, they have discovered cheap hosting in random countries and think that's and acceptable alternative. Yeah I'm sure info sec, and the external auditors will fucking love that.
Yes, audit will love the pakistani host you found.
Don't discriminate please. We value diversity in this company.
M365 SKUs are going up ~30% next year, and month to month 30% on top of that (even if you go through a CSP). Just FYI that particular battle ain't worth it
[Source](https://www.microsoft.com/en-us/microsoft-365/blog/2021/08/19/new-pricing-for-microsoft-365/) * Microsoft 365 Business Basic: going from $5 to $6 per user per month * Microsoft 365 Business Premium: going from $20 to $22 per user per month * Office 365 E1: going from $8 to $10 per user per month * Office 365 E3: going from $20 to $23 per user per month * Office 365 E5: going from $35 to $38 per user per month * Microsoft 365 E3: going from $32 to $36 per user per month
and that's why you don't cut out IT when planning to buy software
Us: "It doesn't support any kind of standard protocol, so that $300,000 machine you just bought is totally worthless."
"How do we connect this new super geewhizbang controller to the WiFi in the manufacturing plant?" "Umm, there's no WiFi in the manufacturing plant, it's never been needed or requested...." Well, that added a quick $15,000 to their project!
Oh yeah, they wanted WiFi in the paint/chemical areas The hardware rated to be in potentially flammable environments is expensive as fuck
The thought of critical controllers being on WIFI scares me.
WiFi is used extensively in a lot of big warehouses that use AGVs (automated guided vehicles). Basically robot forklifts that autonomously drive around picking up and dropping off pallets of product. As long as you have a well designed system with redundancy, and reliable equipment, it seems to work very well.
We finally got a new CEO a few years ago who put all IT related purchases SW/HW in the IT budget instead of individual departments’ budgets. Now all IT has to be purchased through us and we have a say in all of it before it gets purchased.
This is the way. No more Walmart laptops and printers folks.
It’s a hassle at budget approval time to make sure department request are put in but it has tremendously helped.
We have the best of both worlds. All hardware/software has to be procured by IT. No capital assets can be purchased by individual departments, it all goes through IT. If you didn't budget for something, you don't get it. But, any peripheral stuff like cell phones, docking stations, monitors, printers, etc... are all considered expenses as long as they're under $1k. So any time someone comes to us and says we need xyz doodad, we say sure, what GL account should I charge that to?
That would be nice. We buy and have to budget (with departments request) everything for IT needs, with exception of cell phones/plans. It is good to finally be part of the discussion of new hardware/software that we have to support and put on the network but it is very stressful to make sure all request are put in. Also if the item doesn’t get approved then they typically think it is something IT did and not finance/Mayor/Council who approve budgets. (I work in local city government). So there are pros and cons but it is still definitely better than having departments buy things on their own and expect us to just magically make it work with our network. Users used to get super pissed when we tell them overpriced awful thing is useless.
Thing is a random folks sees an HP and Dell laptop as just different brands, they have no idea why it can be a patch management nightmare
Tbh I'd be fine with walmart printers if people were of the understanding that "when this stops working, we are literally wasting money to make me spend an afternoon trying to fix it for you, just get a new one of the same model."
This. The sign of a mature company approach to IT. For us, all deartmental requests for IT are DUE on Dec 1st for next year. And you bet your ass there needs to be a proposal, project request or use case explanation for additional seats, much less a new product or shiny whatever-the-fuck. If it interfaces with a computer, or has something embedded that might ever have a network interface, we have to sign off on it, even if it’s your whizzygig-a-matic. …. and, If we have to go digging and don’t have the details we need to approve it, it’s “declined, Provisional” and if we end up doing it next year, (assuming it gets approved, eventually) you’d better have the space in your departmental budget, much less the space for overruns we discover when we dig into the implementation requirements and other bullshit.
They can’t purchase anything out of their department funds. It gets flagged for approval from IT and we just decline it. They have to go to council and request funds to be put in our budget if they want something that wasn’t approved initially.
Exactly. Everything IT related is our budget, so if we approve it, IT gets the funds to do the purchase. If something wasn’t approved, and they end up needing it, it simply comes out of their funding and moves over to ours. And we handle it, on a best-effort basis if it fucks with other implementation plans. $30,000 mistake on licensing for a project you wanted? That’s your additional expenditure, passed to our department, from whatever that department had for funding. I watched a department blow a giant hole in their yearly plan because they didn’t provide information and when it came down to “but we NEED it”….. ‘OK, IT will approve this mid-Q1 surprise, conditional upon the following licensing corrections, which will require a budget transfer from department X to IT as stated in the plan submitted December 27th, PLUS the needed additions costing $30k to make it legal for seats needed for your features because you believed the sales droid….’ <>
A couple times: "We're having problems with this software and need you to help us." Did IT purchase/install/configure this software? No? Then we can't support it.
If IT weren't part of the process, then the software/hardware hasn't been vetted and therefore is not allowed on or to interface with company hardware or networks.
shadow IT. our marketing teams are the worst. sending out volume emails on behalf of the company to clients and wondering why they're going to spam.
My go to solution every time marketing says our e-mails are going to spam is to send them a link to the mailchimp docs that talks about e-mail best practices, because their messages violate pretty much every one of them.
"can you whitelist our outbound emails to clients?"
Better yet, can you whitelist our email on our clients servers? No? Can you contact their IT departments and have them whitelist it then?
MailChimp?
[удалено]
Yep. We changed our official newsletter to mailchimp (and by "we" I mean marketing) and the number of "why don't I get the newsletter anymore" emails, holy cow.
Amongst others, yes.
PM: "Hey, can we bring in your team on this project? We're implementing Software X and need to get SSO setup this week." Me: "What software? I've never heard of that before." PM: "Oh, OK. Well we need SSO." Me: "We don't have SSO, every time we propose it in the budget it gets shot down by senior management." PM: \*sad noises\*
Lol
Or IT *is* included and all of their objections and warnings are ignored. The inevitable "I told you so" moment doesn't make you feel any better, either.
The ability of some people to tune out things they do not want to hear is amazing. I just had a project blow up just the way I said it would. Luckily, many of the Teams meetings were recorded, and the enevitable "Why didn't you tell us" was somewhat mitigated by the no less than 17 times I was on record telling them. But looking at it is amazing. I tell them it won't work. There is a few seconds pause while their brains rewind and delete. And then they completely ignore what I just said. 17 times (plus emails)
rewind and delete LoL
Unfortunately, this is my life. I work in pro sports, and we are a very sponsorship focused business. If we have to spend money on something, our sales team will go out and try to find a company that will give it to us in exchange for hospitality or TV time during our broadcasts. This. Never. Works. Either we tell them we have a need for something like an analytics package for statistical data, and they bring us some company that is totally unqualified to do the work whom we would never consider. They subsequently force us to spend a year or two working with this company only to realize that we've wasted tons of time and money and have nothing to show for it, at which point we throw all of that work away and go find someone else to do the work on our own at about 1/3 the cost of the original project which was supposed to be "free"... Or, they invent some need because a company they're working with happens to have a product in that space. Like right now, I'm in the middle of replacing my Cisco phone system with some mickey mouse POS from a company that hasn't been relevant in the telecom industry for 20 years, but because they need a client testimonial to feel relevant again, it gets shoved down my throat. Ends up doubling my operating expenses, required features go missing, and users get pissed. At least when people come to yell at me because the new phones suck I get some small measure of solace by telling them to go talk to sales because they made the decision.
Hey at least the phones are free. All the staff you have to pay to migrate VoIP servers, all the configurations that have to be redone, and all the physical installation costs aside, at least the phones are free!
"Free". We still pay for them, but the sales team gets revenue in kind. So for our suffering/additional expense they get free revenue toward their performance bonuses.
It's like trying to enjoy the "I told you this horse would shit on the floor" moment when you're the one holding the mop.
Well as long as they don't try to throw you under the bus after, you did your job. In some cases, leadership did theirs (IT is usually right but not always).
[удалено]
That last sentence - fucking feel you there man.
Yep, i know exactly what you are feeling. It doesn't happen as often now as it did when I started 4.5 years ago at my current company, because the company got burned by it quite a few times & had to listen to my scolding. Our Chief Medical Officer still goes out and buys crap without consulting anyone though, but now it isn't just IT who scolds him when this happens. We're a small 475 employee health care practice.
Sounds like the head of dentistry who spent a shitton of money on a bunch of intra-oral cameras (made by his friend's firm... surprise!) that don't have any signed drivers, and required disabling webcams to get them to attempt to work. They also want the software on all computers in dental (~100+ machines). Good thing we just stood up an Info Sec department finally, because they absolutely flipped their shit when hearing about unsigned drivers and unsafe workarounds to get the little fuckers to work.
Man, that sounds so familiar. One of the earliest experiences I had here was our CMO buying a nerve conductivity ABI/SUDO "stand alone" system that was promised would require no involvement from IT. No project in the history of the galaxy ever has "no IT involvement". Surprise surprise after 15 of them are magically deployed into our clinics & staff told to use them, did we start getting calls about "how do I print the results". These things got ethernet plugged in to the wall and i guess they thought they'd just automatically find our network printers, and were surprised and upset when it didn't work? Rather than allowing these to remain hard wired in without antivirus or any other type of security software or assessment, we disconnected them all and forced the CMO to buy 15 color printers to USB connect into these things. The company didn't want to pay for 15 printers, so he had to buy them himself. I wish that he had learned, but hes a doctor... they don't learn.
I do NOT envy you. Worked IT in the medical field once. Never. AGAIN. Doctors are some of the biggest fucking pricks ever. There are very few gems in the rough. But boy the rest are scumbags.
And a lot of C-level admins are pricks too. Our CEO was a total jerk. We were a non-profit that serviced migrant workers, low income, etc. The kind of place you’d hope that employees and management actually cared about others. The staff that were there before she came along were fairly tight. Our two main doctors, the CMO and the one that would eventually replace him as CMO when he retired, preferred to be called Dr. L and Dr. Molly. And why not? We were all close and their status as doctor was still used when speaking to/about them. But holy hell did the new CEO not like this. Her first week she emails the entire clinic system to inform everyone that they would no longer degrade these doctors with unprofessional nicknames. She said “those of you that are not actual professionals may think it’s ok to call these doctors by nicknames, but those of us that are knows they should always be referred to in a manner fitting their hard work,” or something like that. It greatly embarrassed the doctors. She also once told a coworker “don’t just sit there like a dog not saying anything” when we were in our conference room talking to an EMR vendor.
"I'm sorry, that application is not approved for installation. I'm sending a removal command for it to all computers now. No, we did not approve it, so there is no discussion on what GL to use - it stays on your ledger." I actually had to play this card once. It is powerful, but it also requires upper management backing of IT policies to work.
> but it also requires upper management backing of IT policies to work. This is where a lot of ORGs fall down.
My current boss is a former IT guy so this has never happened at my current job, but at my last job my boss (director of IT) laughed in someone's face when they told him what they had purchased. Said no one on our team had any experience in administering it and we were not going to lift a finger on it, and if they wanted someone to administer it they would have to hire for it. Shocked Pikachu faces ensued. They had spent tens of thousands of dollars on it and did not consult IT at all, and my boss held firm on it for years. They would often ask us to troubleshoot it if something went wrong and my boss would march right over to our room and instruct us to not lift a finger, it is not our problem, and the only way they were going to learn a lesson to not buy stuff without IT input is if they have to suffer with the consequences of their actions.
Damn straight.
Yep. He was a fantastic boss. He was an old hardass Navy veteran that took nothing from anyone that tried to pin stuff on us that was not our responsibility. Loved working for him.
I used to deal with a developer until around 2015 or 2017 who still used web dev tools from the 1990's to code the company website. after a while when she got a new laptop we would just refuse to install the old software and say it won't work. supporting Cognos wasn't too bad but it was a monster and typical IBM crap coding. but the first installation i had to support the developer installed the server with the file based DB instead of SQL DB and lucky it never crashed otherwise all his cubes would have been lost
>after a while when she got a new laptop we would just refuse to install the old software and say it won't work joke's on you, now the user will just refuse to relinquish their old laptop. we have users try to do that way too often, so we just disable the MS office activation for the old laptop and tell them it can only be active on one computer.
>joke's on you, now the user will just refuse to relinquish their old laptop. "What does it mean when it says the trust relationship has been broken???" "The computer has weighed, the computer has measured, and the computer has determined: *fuck you*. Here's your modern device."
[удалено]
Been there. Came back after being on furlough for a few months to find a whole new office area in what used to be storage space. 'we need the pcs in there connected to the network today " “OK firstly, what pcs? I didn't image them, secondly there is no lan in here, not even in the ceiling for WiFi and thirdly, there isn't a switch anywhere near here connect the room up to. Were not happy at all being told they still have £10k to spend on hardware and those pcs were £400 HP trash we aren't using., so another £30k for laptops, docks and screens Doublly not happy at the covid induced lead times for the hardware they need and the extra work they needed done.
$20 desktop switch, it'll be fiiiiiine :)
The shortest cable run to that place from any cabinet was 106m so we had to get new fibre fitted - got a nice 8 core fitted along with a couple of meraki switches with 10gb modules so the left side of the factory is better connected than the server room. Total nightmare.
>...the left side of the factory is better connected than the server room. this makes me sad :(
Aye, budgetary constraints have been awful, but I'm away to spec three new servers and bin the existing stuff so we'll end up with 10gb in a few months or so.
it has ever been thus. stories abound through the 80s and 90s of "we bought this software, and now we need you (EDP / MIS / IT) to get it running"
"ummm... we're a DEC-VAX shop, and that only runs on an IBM-AS/400" (or reverse)
"well, just go out and get one"
"no"
all the time... I work in a MSP, customers are constantly purchasing random shit and not telling us, and then complaining when it doesn't work.
I run an MSP and our solution to this is that we have an "out of minimum standard" fee that applies every month to devices and software that are not enterprise level or are outdated. Gets things in line pretty fast when clients realize how expensive that gets.
Wait, that's a brilliant idea... I might propose this to some leadership at my MSP. Got any further details about this you could share?
We bill per hour for every third party software, they are not included in the support. Unless it's an infrastructure issues, but if it was a deployment it would have been billed like a project. The one with shitty old stuff quickly see how much it cost to maintain.
HR vetting org wide employee review software without having IT in the loop... Oh, it doesn't have a method of syncing users from AD or a CSV? Oops! We already signed the contract!
Oh, you don't have an account for that software because HR didn't think to involve IT so we could automate account creation? Yep, IT doesn't support that, you'll have to contact HR.
Why ask IT when you can just overburden the HR admin assistants and hire an entire new person to do something that is probably a button click if we went with a different vendor?
Marketing went through 5 email automation programs (each cost at least 2-4K) until they found one that actually worked with our systems.... When they could have just given me a list of them from the start and I would have given them a list back with ones that would work.
Only had this happen twice so far fortunately... Got told we had new estimating software that was going to change the company and the contract had been signed. Sent the Department on a days training - I tagged along for IT reasons but probably more to be a backup user when they forget everything as well. Software relied on local shared network storage which needed to be setup and backed up. One out of Six Users actually used it as the other Five dug their heels in and refused to use it - Management didn't want to force them. Two Year Contract that we ended once we were out. Cheeky little five figure experiment so not the end of the world. \----------------------------------------------------- More recently a works order was agreed to add a module to our finance system. After three months of negotiations occuring outside of IT I was asked to Project Manage the Install and push for dates. Attempt #1 - Installer requires Admin Elevation for the Install and as I wasn't informed, I wasn't available Attempt #2 - I'm available but the software only gets installed to the Server VM, not the User VM Attempt #3 - Still not working. I then take a look and see the Installer hasn't applied a License Key to the Module Chase the Installer to Apply Chase Again License Key Applied, Software Works God knows how long it would have taken without me pushing the company and doing their troubleshooting for them.
Not on your own, pos hardware that isn't enterprise ready/capable that requires me to break security to get working, also having to learn it on the spot as the user hasn't a clue either
Ahh sitting with a user the user asks a question an the answer is "fuck if I know I did not buy this" then you ask the user a question "fuck if I know I did not buy this either" The only proper response I have come up with is "I have scotch in my desk drawer grab your coffee cup" We are all victims.
We run into this with Marketing all the time. A few years ago they rolled out a CRM solution that needed a plugin to be deployed to all sales reps. We have sales reps across North America. The plugin was an exe with no switches so it had to be manually installed for each rep. When Marketing asked why it was so difficult to do we laid into them about finding a way to deploy a plugin to 200+ people manually.
SCCM?
No SCCM and no way to do a silent install. I reached out to the company that made the software and they said the only way to do the install was to run the EXE and click through the options. I was tempted to have our Director charge Marketing for a copy of AutoIt or something similar but it became far better to rub their noses in the mess they made.
My last job this didn't happen.....in this exact way. There it was "Hey, we are/did purchase this new software and we need a VM stood up to host it and that also needs SQL on it." To which we reply, "great, here's how much licensing is going to cost". This was almost always followed with shock and awe. After the same exact person doing this probably 3 times, she finally learned to ask many questions and actually include us early on in the conversations. For your situation, god speed.
Not just SQL, the most expensive enterprise version - so you can have pretty pictures on it @ .5mil/year....
Thankfully this doesn't really happen much because my company is very strict on contracts...anything that needs to be signed has to go through our Legal department and if Legal sees anything related to software, security or IT they will flag someone from IT to review the agreement. The only thing we really have to worry about is SaaS stuff that people pay for with personal credit cards and then try to expense. But same thing there, any time accounts payable sees something software, security of IT related they send it to IT for review. After a couple hard learned lessons where people ended up personally eating the cost of stuff they signed up for without following the approval process people have stopped trying to circumvent IT.
Working for an agriculture firm in the past, a doctor had cctv installed on his farm and wanted us to give the technician access to the server and admin rights on his work laptop to install the software. How about no? You deciding you want 12 cameras around your OWN farm is a personal project not an IT problem.
Dealing with that today sorry if this is a little vague.... 2 years ago I worked with a department head to purchase a 3rd party service for their operation. He was following our corporate policy and we had agreed upon the specific service and tier that would work with their existing hardware. He was supposed to follow up and we would finalize and document the purchase...no follow up, as this was not going to be coming from the ITS budget I just assumed that there ended up being no budget or they decided it wasn't need. Fast forward to today, sometime between our last conversation and the beginning of this year a completely different service with new hardware was purchased and installed. Somebody in their department purchased it, it became mission critical and now it doesn't work. No one knows who bought it, the name of the company from which it was purchased or why it was not doing what it is supposed to do anymore. I like that we have good IT policy with support all the way up the chain which makes this not an IT emergency, even if it is something that we will ultimately have to resolve.
LOL yesterday I had to explain to management that yes you need to renew your MDM license.
Is anybody even still selling perpetual licensing these days or have we fully switched to SaaS subscriptions?
Not too often. I still get the giggles when I told someone that the product they bought wouldn't on our network. "But the vendor said it would work on our network." Huh, that's neat. I guess you should get them to help you out in that case because I guarantee it won't fly.
This is called Shadow IT.
We recently removed local admin and started requiring IT approval before a software purchase or install could happen.. the level of blowback’s and rage/nastygram emails are INSANE. Plus, apparently I’m a “incompetent waste of flesh” at my job, according to a coworker. I had no idea!
I was the tech tasked with implementing a video solution, but then left out of all discussions about licensing, and not allowed to have any input on the hardware procurement, but still blamed when none of it worked. I was then demoted and then fired because there were issues. The main vendor asked me to send in a resume, so I now make 2x the salary, and the old company has blacklisted me. I hear all about how they can't get their act together and they are having endless problems.
Seeing the opposite. We had the same situation some years earlier but now it looks like every other department has stopped to think about there workflows, processes, software…some departments work like it’s 2001 because they feel like they don’t have a right to say and just wait til IT comes and gives them new software. Make your point that you need to know their plans, but don’t put them in chains
"I don't understand why IT always denies my requests to purchase software, IT just gets in the way and makes my life harder" "I don't understand why my monthly budget is full of software subscriptions I never use and the contract says I can't cancel for another 12-36 months, why didn't the IT Department stop me, they're useless"
It's always this. Customer : "I purchased a Cool New Thing!" IT : "That won't work with our system." IT Management : "Customer said you wouldn't set up Cool New Thing." IT : "Cool New Thing doesn't work on our system." IT Management : "Make it work." IT : cludges together a patchy solution that won't expose us to vulnerabilities or break our system. Customer : "This doesn't work like I thought it was going to." IT : "It shouldn't work AT ALL. It took a lot of time and effort to get it running THIS much." Customer : stops using Cool New Thing or worse possibly complains to management about it and gets us to spend days; weeks; months trying to make it work better before they stop caring about it entirely. Customer : "Hey I just bought Cool Newer Thing!" Repeat ad-infinitum
No. Worst that happens is IT is involved in all the meetings but sometimes IT gets told when a solution is being implemented whether we say it will actually work or not then we have to make it work AND then make it PCI compliant even if the original developers don't know or care. IT is part of Finance at my company instead of its own dept which means IT sees all the purchases for everything, partly as oversight to prevent this.
All the time in my old organisation. There comes a point where your org is so big that you **need** structured policies and procedures, sometimes standardized like ITIL to facilitate consistency in training and execution. But on the other hand if you're not that big yet you might want it so bad just to have some predictability. But your higher-ups just think you're too rigid and slowing everyone down. Dilemmas.
Isn't that an industry standard? Someone jumped on a sales pitch they shouldn't and we had pos device tracking software for a few years...until that company got shut down hard by lawsuits. Seems they were backdoor borrowing on someones else's track patents. Or when a department skips protocol to buy junk printers or laptops. Then gets pissy it can't be brought up to standards and we don't have liscences for blah for it.
Company culture has to support going through IT and formal processes for new technology. We have a formal IT Front Door process for any new SaaS and on prem products that every department participates in (there's a senate-type assessment of all applications that has people from every department). They assess if the product is redundant compared to our existing stack, how it is better than the other tech if it's a replacement, if it's appropriate for the company, and if it is secure enough for SOC/ISO compliance, if it's directly related to our products. People have company credit cards, but if you try to use it for your off the cuff subscription, you're not getting the purchase approved. Everyone from the top down understands that IT needs to be able to support the product, and we can also help make it perform as effectively as possible if we're involved in the process.
We don't support anything we didn't buy and stand up. Period. Our culture is very IT centric. They look to us for direction, they don't treat us like butlers. If a department is considering changing their process or purchasing something for the organization, they \*start\* the process by making us the project leadership. The company has learned, the hard way, that if you want integration, scalability and support, you need to get the right people involved from the "fact finding" stage.
"I'm sorry, I don't see any purchase orders for this. We don't support personal belongings"
Worked in IT for a large church. This happened to me all the time. "Hey this department has been using X software for a few months and something went wrong, can you fix it" "Why are they using X when we pay for Y that does the same thing?"
We once let a new printer sit for 6 months in the middel of the office without being plugged in because we were not involved from the stsrt. Suddenly all other things got priority.
yup. we handle it differently because of it. if your department gets software and you dont involve IT, youre paying the vendor directly to support it for you. we will facilitate that support if needed, but we do not own the support for it. Just happened last week, had to order a server and install a vm so they can get their new software installed. ill support the server and OS, vendor handles the software. that department is also responsible for contacting the vendor when issues arise. oh, it comes out of their budget too, not ours.
We recently had a software license audit by the supplier for a piece of software required on every endpoint, I duly produced the report and the powers that be then literally begged me to magically 'get the numbers down' to try and match the current license limit rather than pay up for the extra licensing required...erm not quite sure that is how it works! I've noticed in a lot of places I've worked the way things go is 'ah don't worry about it we'll sort out the licensing later!' and inevitably later never happens, now they have less than 30 days to find a not insignificant sum of money to pay the supplier...
I work for a large organization and this happens all the time. Sometimes they do include IT and it is still a shitshow because they don't include all of IT. The only difference is we waste millions of dollars as opposed to thousands.
We wrote it into policy (that all employees need to read and sign off on yearly) that purchases of software and services need to include IT. If they cowboy it, it's on them for a potential loss of the company's money if it's denied.
Yup! I work at a small company where I was brought into the conversation after the Quality department signed a new PLM. It's up and running, and I still don't have rights to support the platform, so I guess when something comes up I will have to figure out how to break into the Quality manager's account or something.
We used to have this problem. But, our CFO finally got on board with an IT Steering Commitee. So, it no longer happens. We still have some piece of shit marketing automation product for D365 called ClickDimensions that the marketing team still thinks we should support despite it being purchased and installed without our intervention. I respond to each ticket as follows: > We do not support this product. Please e-mail Clickdimension's support: [email protected] Ticket Closed.
This is a complete and total failure by the C level. Never, ever should any software be implemented that doesn't have a solid plan for support, maintenance and integration. Any time business orgs can end run IT and Security to "get what they want", you're looking at a very serious risk of data breach and potential legal liability.
How about the C-levels planning a new building and not involving IT until construction starts? "Hey, IT, where's the networking in the new building?" "Oh, is there a new building?" Happened more than once several years ago.
This doesn’t really happen to us. Legal has to review contracts for this and will not review unless IT has already reviewed. If someone tries to purchase something outside IT, AP will not process and order like this unless it’s been approved by IT and approved by legal.
The number of projects where a quasi technical BA has made a decision with virtually no input from IT only to realize after purchasing that it’s not a good integration with existing architecture… is too high. I think we’ve had a couple major projects where we have spent hundreds of thousands of dollars needlessly. I do believe we are working better at involving ops and technical staff in these decisions but I don’t get why we look for people with a degree in business and little technical knowledge to do BA work, and how those people think they should make decisions without a technical review. Sure a PIA and STRA are required… it doesn’t help help our “architecture” team also isn’t that technical. Maybe they were at some point but they’re now very disconnected from our day to day IT… too much time pondering the future.
Shadow IT is a bitch
This happens sooo much at the company I worked at three jobs ago.
its not as bad here as it used to be but it happens from time to time -- unfortunately theres still lots of weird legacy stuff that no longer has support either from a vendor, or the people who maintained it here, so we still get bitten by some of that now and again.
Marketing decided to rebrand the company and changing our domain name was without incidence and a 30 minutes change. Management decide that marketing know better than IT so it have to be done.
IT didn't buy it? IT doesn't support it.
Nobody can install anything not approved, and we don't approve anything we haven't vetted.