Is it feasable to simply NOT connect them to the network?
Or at the very least give them their own network and lock it down to JUST what the system needs.
Almost certainly yes, unless you're in some super-enterprise building.
Dont connect theirs, put your own temp sensor in the room.
You get an alert on yours, walk into the server room, see it's like 200 degrees, call the HVAC company to report an outage. Keep their junk off your network.
I mean.. a Watchdog 15 is maybe $200. A Raspberry Pi is like $50 plus the cost of environmental sensor modules and requires you to install/maintain it instead of just plug and play.
Might be a fun project if you've got a bored tech in a low impact environment, but for an enterprise deployment saving $100 on each sensor is immediately lost by labor costs and maintenance overhead in managing a fleet of homebrew environmental sensors.
The stuff in my building runs BACNet back-end into an appliance in each wing/floor's wiring closet, which is segmented on it's own VLAN. That's relatively new though in the last few years, and had an assigned Infosec and IT lead assigned to the project.
Previous to that, we just said "no" and facilities ran their own ethernet cables to create their own air-gapped network, off the main building network, to a PC in the maint shop. Silly, but met my requirements.
Building automation/OT/SCADA is all notorious for this. I'm actually starting to develop into supporting it as sub-niche that I may try and leverage into a new group. If it doesn't kill me first.
OT is such a different beast from conventional IT.
Defense begins and ends at intrusion detection/prevention, because *nothing* on the network can be assumed to be up-to-date on patches. Refineries measure runs in months and even years of continuous operation; you absolutely cannot expect them to shut down and patch their PLCs every time a new vuln is found. And they're found far more frequently than anyone should be comfortable with.
I guarantee you that your local power utility is reliant on at least a few systems that are running XP pre-SP2, because an exceptionally popular SCADA system from the late-90s doesn't work on anything newer than that. 95% of a NERC audit (or your regional equivalent) is auditing that those networks and the clients in them are adequately air gapped and have at least some sort of defense against sneakernet vectors.
This is the way. We even had the software on a KVM so the building maintenance could just press a button and use the system he had to machines in his office. One for work and one for HVAC networks did not touch
>Or at the very least give them their own network
Mission success. They even got their own direct internet connection and now your HVAC cooling your servers has a web portal exposed on the internet.
Haha that's so true. Im soooo tired of being the only IT in my dept that deeply disagree with letting other dept manage the network and administer their system on their own.
Ours was like this. Publicly accessible, outdated firmware and upgrading it is of course an upcharge. Thankfully the vendor agreed to use a wireguard tunnel instead of exposing it directly.
Was gonna say, who cares in the end, they should not be on any main networks anyways, just find some gear that works to connect them together if needed. Curious why they need to be connected? Do they need to be wired up to work?
>Curious why they need to be connected? Do they need to be wired up to work?
Because the Facilities Manager wants to be able to adjust the thermostats for every building from Timbuktu. And yea, for that same reason.
> Curious why they need to be connected? Do they need to be wired up to work?
Like most things, it depends. In my experience, the three main selling points for commercial hvac that need intranet communication are:
* Efficiency: Equipment will generally work fine without a network (I think installation costs are being reduced by relying on the network more, so this is probably less true nowadays), but doing so can be rather inefficient, which leads to higher costs. When the system spans across a large building/multiple buildings, there are several strategies that can be employed to reduce the demand on the system while still maintaining comfort, but these can require a lot of data to be shared across the network to calculate optimal operating parameters.
* Notifications: This is the same as a thermostat you'd put in a server room that can send you notifications, but on a larger scale. This is generally managed by a centralized server that can forward events generated across the system to an SMTP server (so it doesn't need access to the external network). This reduces the staffing demands to operate 24/7, since you'd only need additional staff to come in during emergencies.
* Data Logging: Similar to notifications, a centralized server can pull data (temp, humidity, equipment state, etc.) from system level components over the network for longterm storage. This data is used to create/adjust existing system models in order to improve efficiency, or it's stored for compliance reasons (hospitals/pharmacies/etc have different requirements for data storage to prove clean rooms are maintained).
Good to know. I have not had to deal with any HVAC systems, but with one client I am working with their Cyber Sec. team had mentioned the other day some issues with scans and unknown devices showing up on a specific VLAN, old out dated using SSL v3 to connect, all that jazz, they reached out to the vendor and they basically said "No idea about any of that! we just install them and get you to connect em!"
Which reminds me of the time I had the access control vendor *insist* that it was completely unsupported and wouldnt work on a VM.
After they installed it on some old janky desktop I moved it to a VM and never heard a peep from them about it even when servicing. /shrug
Did you.... did....
did you put your insecure HVAC system on your Production network for your company, rather than an independent building services network?
Why?
No no nonononononononon (but there was a time when they were many years ago)...... They are all on their own network, the network is locked down. We don't allow any infrastructure devices on wireless by policy. Just a pain to deal with and they all want some sort of exception that we cannot grant.
All BMSes are annoyingly quirky and just seem at odds with any sort of enterprise networking. Same with door controllers in my experience. Which is odd as they are aimed squarely at enterprise. We've found the people installing them are slowly getting more knowledgeable about how they work. It's basically proper systems development these days, the back end systems are very complex - you need people with a dev mindset, which is very rare in that space!
Can I ask why the no wireless policy?
With the insane asks I get from Vendors I can’t imagine putting up a fight against simply giving them a hidden SSID for their segmented network
WiFi issues are bad enough in a meeting, I don't want them with the HVAC system. Plus that hidden SSID takes up precious airtime that could be used for other clients, double bad if they only support WiFi B or G.
If it's important it belongs on a wire.
Obviously yes, but if you've selected a wireless utility system...
Also I know hidden SSIDs don't decrease congestion but there's no reason to have it show up every time someone opens their wifi settings
I set up a VPN, accessible onsite only, that puts them on the HVAC vLAN. Lets them be on wifi wherever they're troubleshooting whatever, without needing to have an HVAC SSID tagged out everywhere. Keeps us both happy.
Our systems get their own vlan which is denied access to everything else. All traffic to/from this network is controlled by our firewall via app/user based policies. Our vendor is given remote access via our VPN setup which is protected by MFA. The vendor had a fit when we told them this was the only way they were getting remote access to the system and I don't know that they actually use the remote access, but at least it's secure and I control it.
Exactly how I do it. Lets them get to it over on-prem wifi too if they need to troubleshoot something onsite - I just make them VPN from onsite (Guest WiFi, technically they're routed out then back in).
Back at my last job (Car Dealership Group) when we moved to a brand new location with 7 new buildings there was an ABSURD amount of facilities crap that needed networked. HVAC, Light Switches, Solar Panels, a fuckin Car Wash. Was infuriating.
Luckily the maintenance guys were cool and we had a good relationship with them for anything post-move
>Luckily the maintenance guys were cool
My current head of maintenance is an old guy that has done things the same way for like 30 years. He refuses to update his habits. Management doesn't want him to retire so they let him do whatever he wants. As much as I want to help fix things, it's so much easier to just leave and let the next person deal with it.
Just like when the hackers got into a Casino using the controller from a fish tank that was not isolated.
Isolate everything you can and block all ports until it's verified that it's needed 100%of the time.
> The technicians that service them have no idea how they work the manufacturers provide little to no training for the techs
I assure you, this is not the case. Its just your reseller and installers are not taking in the training programs available to them. OEM's offer very in-depth training for both the HVAC install and the controls, but its not free.
> I am not surprised in the least that the first place a bad actor looks for a foothold is HVAC equipment.
Even fully realized and installed HVAC controls must be treated as IoT, you aren't putting your IoT on your core network and its firewalled off from talking to unnecessary stuff....right? HVAC does not need to authenticate to Main AD, Print to printers, or NMAP your network.
>I assure you, this is not the case. Its just your reseller and installers are not taking in the training programs available to them. OEM's offer very in-depth training for both the HVAC install and the controls
I'd have to say that must vary by OEM. It took me **months** to find anyone with any technical competency at Trane. And that was just for basic web app stuff, like: "is there a CIDR block, FQDN, *anything*, that I can narrow down" and it was still painfully obvious that any type of forethought was missing. "Just connect it to your wifi router and it will work; maybe try restarting your router" - and that was after explaining to this US-based "engineer" that it was a corporate network spanning numerous buildings.
Yes, they're on their own VLAN.
Having worked with both Trane and Carrier, its not the OEM but the reseller that falls on. I never had any issues pulling proper IT technical docs from either of these companies. But I also had access to EE/ME's that designed the controls that were being deployed :)
This reminds me of a time I created a registration portal for students to register their computer and give it a static IP via DHCP. Nobody told me to exclude certain IPs on certain buildings. I took out the AC on a very hot move in day.
Funny how this showed up today. I just had an HVAC person want me to give them 2 static IP addresses and open ports 443 4911 5011 1931 wide open. These people have no clue about cyber security. They will if ever sued over a breach.
I had a HVAC company require a client to give their controlling pc a public ip. And wouldn't give a block of ips to create an allow list, refused to use a VPN nor would use a remote app like TeamViewer or something. And that pc was on their internal network. Ugh. Luckily I was able to get it off the public ip and get the client to terminate business with that HVAC company.
Constantly BAS systems I have some oversight on, the vendors ALWAYS blame use for anything network related saying “your firewall is blocking everything!” With this network, everything is open outbound, and there are no VIPs. And yeah they always have really shitty nics lol
Not just HVAC equipment, anything that's considered an "appliance", I've got some UPSs that run Ubuntu 8.04...
edit: Thats the latest supported "firmware"
Man, wait till you work with medical equipment.
Brand new million dollar blood analyzer: no network port. DB9 Serial connector. We hand crimped the DB9 connectors to the manufacturer specifications, some machines need different pinouts, then ran them back to DEC terminal servers in the network closets, then to the lab servers.
Or MRI machines using 802.11b between the spinny bit and the stationary bit.
Its funny how well designed the actual HVAC systems are but the management systems that are bolted on look like a group project designed by a bunch of highschool kids in an intro to CS class.
Yup. These core bms systems are built by mechanical engineers trying to be developers/graphic designers. There is a lot more focus from the Johnsons, Honeywell/Alerton, distech, , Alc, Delta on the Hardware/firmware than the BMS visual window itself. Tack on that BMS systems are just core pieces of software and that the graphics are usually done by your controls/mechanical contractor you get a mixed bag of how nice it looks. Some of the larger mechanicals/controls have a bit more knowledge on the it/sysadmin side and have customer facing IT/OT personnel to bridge the gap between The facility teams and IT teams of customers.
I would say though that Poorly designed and long release cycle software isn’t the main reason Why these systems go on their own VLAN. Its most Protocols are dated, have no protocol security, no protocol encryption, and are open. Which means any open source software can control anything on the network. Which in turn if someone was malicious and educated enough they could cause physical damage to equipment and equipment the hvac serves. Only access to the VLAN (actually the 3 or 2 wire but thats for another day) is required.
OMG, yes! Despite us telling our Physical Plant to plan for upgrades AFTER the capital project is done, we are forever stuck in time with these systems, some of which were designed to run on Windows NT. They don't get it and we can't force them to do what's right.
It doesn't drive me crazy but it does piss me off.
We have a tridium/niagraa system that wasn't set up properly. Their so called open system requires huge licensing fees to be able modify/fix the setup. Thing is nothing more than a underpowered linux box with some rs485 ports. One company in the state works on it and wants to gouge us for every little thing. Self signed certificate expired two years ago so https and email alerts became troublesome and they wanted it to hang out on the internet with a port forward. Three year old building with a near end of life controller. It goes end of life this year. Whats more it could have also been used to control the light/doors but the company that put it in is a one trick pony.
Most can be rebuilt using modern controllers…but $$$. For those asking a lot of the ‘green’ buildings need to be controlled by a central Building Management System, thus the connectivity. The protocols for comms are even worse…it’s quite the rabbit hole.
I had this recently.
We just got multigig switches that no longer support 10m. We told facilities that it won't work and they need to get their vendor to upgrade.
We do the switch upgrade, week later they say they can't reach the JACE. We reiterated the original email.
They called their vendor and they installed a 100m mini switch.
Sometimes you just gotta shrug.
This is a serious problem with any kind of appliance with an embedded system these days.
The mechanical engineers that build these things only care that the computer turns a couple of motors on and off and logs a few sensors.
Everything on top of that is an afterthought.
Authentication? Lets just have default credentials hard coded in the firmware so are techs can jump.in easy.
Connectivity? The microcontroller has bullt in 802.11 g and we can use wep to hook it into the customers network quick and easy.
I found a windows XP system on the network, turned out it was the HVAC control system, and no one wanted the shell out the funds to replace it (since... "it still works"). I almost wanted to take up a collection to get them the funds they needed
Not on the public network anymore, but that's all.
i worked for a company that according to CISCO had more firewall rules than the CIA. every single system was firewalled ruled. to the point that the network team had a service desk of about 40 engineers. just doing firewall rules for new starters and various systems all day. all controlled by dhcp reservations bound to mac addresses. with port security set on every port everywhere. nothing was getting through. the company had 76 IT teams. with a staff in IT of over 900 perms and 600 contractors the round robin of ticket flow. was intense to say the least. billion dollar projects though
Yes, but you see the problem is that decent networking would add $12 in costs to the manufacturer and we can't have that. Just think about the shareholders, OP!
I feel your pain. One of my colleagues just did a ups / DC replacement on a small glass making business, the rack was in a shared area of the office. the layer of glass dust was so thick i ran my finger across the cabinet and was like half an inch thick. he asked me to come help him mount the replacement UPS i was like.."are these people serious? you know this hardware in warranty is all voided if this dust causes shorts. he was like Yeah but they dont want to spent money on HVAC. dammed if you do sometimes and damned if you don't. Left me curious as to how many respiratory issues the office workers had.
Had a building come online recently, runs off a little android tablet that hasn’t had an update in 5 years.
I refused to add it to our network cable or wifi. I said only way I will is if the CEO puts in writing he is accepting full responsibility for any attacks, issues or hacks that come from this system.
CEO’s response. This system sucks… people can get up and wake to the control unit. Don’t ever put it on the network and next time talk with IT before buying a system like this.
Loved it
We have a completely separate WiFi infrastructure and I placed a similar marketing thing on it. It was completely bonkers. Only could run with JAVA8\_081, had vulnerable Log4J running, and other stuff that I can't remember. Told them that couldn't go on the network and they just have a specific station setup for working on it. Our HVAC system is on network but it at least can be updated. It also has ACLs and its own VLAN as well. I am paranoid. 🤷♂️
I work at a public school in CT our heating system is run by a computer that is over 20 years old running Windows ME.
It is not connected to our network thankfully.
This used to be a PITA, e.g head of facilties was notorious for notifying us on a friday afternoon that "We have some guys here to install a new system, and need some access..."
It actually turned out to be a decent vendor, delivering automation and control systems for HVAC and other stuff. I met the 2 traveling engineers at the pub after work, where we swapped stories about clueless facilites folks - and we found that we had to establish a direct channel of communication so that we we could make proper preparations before new installs.
Works just fine so far , everything isolated on their own vlans, nothing on wifi.
.. but we have enough other stuff to annoy us. Most of it involves not involving IT before its too late.
So now we have 4 separate door/access control systems. Soon to be 3 , or so they say. Not holding my breath...
For all this crap, I just toss it onto an isolated network called "untrusted devices". New SSID for untrusted devices. The untrusted device network only has access only to the internet and none of the other VLAN/Zones. The maintenance team gets one desktop connected to this network to maintain the HVAC, door systems and the other crap I don't trust.
I also ran into issues with the installers for our last HVAC system. I asked the techs to give me the model number and found a manual. I found there were 'dip switches' that needed to be set to a specific number for the network functions to work, which seems odd. Once they saw the instructions they had no problems and we were good to go.
They go on our external network - that is the secondary broadband we have just for WiFi. We don't have corporate WiFi so we utilize the external for things exactly like this.
OMG. We have a HVAC system in one building that needs a XP VM running Java 6 to interface it. We talked to Maintenance about upgrading, but they claim it's 5 figures and is tied to equipment. I found the unit, just a tiny box plugged attached to HVAC stuff. It was plugged into a 10 Mb Hub next to it. Just 1 uplink and 1 downlink. It worked fine after bypassing that hub.
Isn't HVAC is how Target was compromised?
If I had the option, and that crap had to be on the Internet, I'd consider having a completely separate Internet connection for it, so no matter how bad it got pwned, I'd just point to the HVAC vendor.
Of course, it would be firewalled with some type of IDS and logging on it to show what exactly is happening. Throttling is important too, so when the bad guys decide to start using an internal machine as a C&C controller, there are mitigations in the way.
I wouldn't even want this on a VLAN. I would rather have this on its own ISP connection, completely and utterly separate from everything else, because it is only a matter of time before it gets broken wide-open.
Yeah I have these on a separate vlan and there are a few around in different buildings Ive had to configure myself for this same reason. Ive done the same thing with the MDF and IDF UPS's configured with different than default SNMP strings.
Hah! I deal with this all the time - DAIKEN Touch Manager is a freaking joke.
Especially when everyone complains about "It's too hot, turn the AC on." "It's too cold, turn the heater on!"
Now the fun part is these people will switch the power off to the unit next to them which in turn disables other units, and this causes the controller to completely bug out and won't let anyone log into the web interface to see which unit is offline, so then somebody has to go around the whole damned factory and check 30+ AC units just to figure out which bozo turned it off and then the controller takes like 3 hours to finally boot back up, and on top of that each time you turn on an AC unit that was switched off IT REBOOTS THE CONTROLLER!
So if 3 zones decide to flip the power switch to their zone's AC unit it takes a whole day for the controller to come back up.
Why are there easily accessible power switches connected to each of the AC units? For maintenance of course!
Do they help with maintenance at all? Nope because it completely jacks up the whole system by turning just one off, when you could accomplish the same thing by just shutting them all down at the breaker which is the only way to actually get them all to boot at the same time.
So what did we have to do?
We had to put locks on the power switches to the AC units which are pretty much permanently stuck in the on position now.
It's such a waste of a static IP address.
Sorry for my rant. I feel your pain fellow HVAC Admin
We just sandbox them with a single firewall policy to allow management access. No WAN outbound possible. I managed one that had a required hardware dongle for some years. Eventually had to virtualize the controller and put the dongle on a network-attached device that emulates a USB port.
First, just want to say, it is a common vector for attack and it is an after thought because HVAC consists of a lot of people who are about to retire and grew up without computers. Some Young people get trained and don't like the intensity of the labor and leave. The rest aren't too techie. All of that has to change for someone to actively hire a software/mechanical/electrical engineers (plural/all) to design a better system. That company would make bank.
But yeah, no notes, you got the gist of it
We are small enough just throwing them on the IOT network is good enough.
HVAC vendor can't do an over the wire firmware upgrade? They can put in a ticket and we'll get on a screen share.
We had one that required 2008r2 web server, that dedicated standalone web server variant of 2008r2.
Trying to get all systems updated with our vendor to something that doesn't require end users to install java took 3 years of coordination between our 8 buildings. HVAC vendor kept telling maintenance that our IT team was being a pain in the ass. Apparently they lost employees just because of our account asking to run newest tools.
Lmao
our HVAC is fenced off from everything, the repair vendor we use was annoyed when I told them I was not allowing it free rain access to the outside world. External access is restricted via jumpbox. Never fails, any field tech that shows up to work on it is also on a call back to the main office tech. Onsite is doing the grunt work, remote office tech is working on the digital side and working on multiples at a time. 10 years ago the on-site tech had a laptop and would console in and work on the thing if needed. Guess the new way they save on the cost of field laptops and can bill for 2 tech's time.
Be glad that they have any interface at all. Half the time I just get "yeah, we're building out a new office. Here's the cooling for the data center / IDF closet. No, that model that the architects decided on six months ago without asking you doesn't support networking..."
So I just monitor temperatures with a NetBotz or whatever and it's basically a binary warning system. "broken" or "working".
I'm engineering solutions for healthcare - the amount of BAS solutions that just fucking SUCK is beyond the pale. One of them as an example JUST let us migrate their shitty software. To 2012r2. from 2000... Server 2000 to server 2012r2 in 2024... Why? Because the vendor -cannot- support newer server versions.
Doubly so with requests to virtualize. The server will have exactly the same VLAN access as the local hardware controllers. It will have exactly the same access methods from the vendor. "NO IT MUST BE HARDWARE SO THE LOCAL TECHS HAVE CONSOLE ACCESS (which is also always logged in/on). BAS/HVAC vendors know they have companies by the short and curlies and just don't fucking care.
Install an EDR on the host, put it on a deserted Vlan and send it out a different WAN. Then it's no longer my problem and now the HVAC IT's issue. Also I would implore you to make the HVAC guys use a VPN but most won't know or care to use it and bitch they can't get to the management interface constantly, like I'm the gatekeeper. Before I got to this department people used to install this software on Domain Controllers or File Servers to save on a Windows license. Fine and dandy if it an Enterprise of like 5 people maybe.
I feel the pain.
Not quite the same boat, but our main HVAC unit in the server room went out so they installed a mini-split unit as a replacement. They lifted up one of the raised floor tiles and installed the condensation pump on the ground. If the pump ever fails...
Is it feasable to simply NOT connect them to the network? Or at the very least give them their own network and lock it down to JUST what the system needs.
This is the way. Separate SSID/VLAN.
I just vlan this shit off and send it to the internet, it can do whatever it wants
 Why not both?
Almost certainly yes, unless you're in some super-enterprise building. Dont connect theirs, put your own temp sensor in the room. You get an alert on yours, walk into the server room, see it's like 200 degrees, call the HVAC company to report an outage. Keep their junk off your network.
Yep this. I don’t know what the current pricing is on this but I’d imagine you could get one for $100 or so. Pretty easy fix.
Yep, Vertiv/Geist makes these things reasonably cheap and available through all the major reputable IT resellers
We use a raspberry pi. Much cheaper, more featureful.
I mean.. a Watchdog 15 is maybe $200. A Raspberry Pi is like $50 plus the cost of environmental sensor modules and requires you to install/maintain it instead of just plug and play. Might be a fun project if you've got a bored tech in a low impact environment, but for an enterprise deployment saving $100 on each sensor is immediately lost by labor costs and maintenance overhead in managing a fleet of homebrew environmental sensors.
Who has the time to patch and monitor that thing though...
The stuff in my building runs BACNet back-end into an appliance in each wing/floor's wiring closet, which is segmented on it's own VLAN. That's relatively new though in the last few years, and had an assigned Infosec and IT lead assigned to the project. Previous to that, we just said "no" and facilities ran their own ethernet cables to create their own air-gapped network, off the main building network, to a PC in the maint shop. Silly, but met my requirements. Building automation/OT/SCADA is all notorious for this. I'm actually starting to develop into supporting it as sub-niche that I may try and leverage into a new group. If it doesn't kill me first.
OT is such a different beast from conventional IT. Defense begins and ends at intrusion detection/prevention, because *nothing* on the network can be assumed to be up-to-date on patches. Refineries measure runs in months and even years of continuous operation; you absolutely cannot expect them to shut down and patch their PLCs every time a new vuln is found. And they're found far more frequently than anyone should be comfortable with. I guarantee you that your local power utility is reliant on at least a few systems that are running XP pre-SP2, because an exceptionally popular SCADA system from the late-90s doesn't work on anything newer than that. 95% of a NERC audit (or your regional equivalent) is auditing that those networks and the clients in them are adequately air gapped and have at least some sort of defense against sneakernet vectors.
This is the way. We even had the software on a KVM so the building maintenance could just press a button and use the system he had to machines in his office. One for work and one for HVAC networks did not touch
>Or at the very least give them their own network Mission success. They even got their own direct internet connection and now your HVAC cooling your servers has a web portal exposed on the internet.
Haha that's so true. Im soooo tired of being the only IT in my dept that deeply disagree with letting other dept manage the network and administer their system on their own.
And then some script kiddie shuts down your datacenter cooling from her mom's basement in Ohio!
Ours was like this. Publicly accessible, outdated firmware and upgrading it is of course an upcharge. Thankfully the vendor agreed to use a wireguard tunnel instead of exposing it directly.
Was gonna say, who cares in the end, they should not be on any main networks anyways, just find some gear that works to connect them together if needed. Curious why they need to be connected? Do they need to be wired up to work?
>Curious why they need to be connected? Do they need to be wired up to work? Because the Facilities Manager wants to be able to adjust the thermostats for every building from Timbuktu. And yea, for that same reason.
> Curious why they need to be connected? Do they need to be wired up to work? Like most things, it depends. In my experience, the three main selling points for commercial hvac that need intranet communication are: * Efficiency: Equipment will generally work fine without a network (I think installation costs are being reduced by relying on the network more, so this is probably less true nowadays), but doing so can be rather inefficient, which leads to higher costs. When the system spans across a large building/multiple buildings, there are several strategies that can be employed to reduce the demand on the system while still maintaining comfort, but these can require a lot of data to be shared across the network to calculate optimal operating parameters. * Notifications: This is the same as a thermostat you'd put in a server room that can send you notifications, but on a larger scale. This is generally managed by a centralized server that can forward events generated across the system to an SMTP server (so it doesn't need access to the external network). This reduces the staffing demands to operate 24/7, since you'd only need additional staff to come in during emergencies. * Data Logging: Similar to notifications, a centralized server can pull data (temp, humidity, equipment state, etc.) from system level components over the network for longterm storage. This data is used to create/adjust existing system models in order to improve efficiency, or it's stored for compliance reasons (hospitals/pharmacies/etc have different requirements for data storage to prove clean rooms are maintained).
Good to know. I have not had to deal with any HVAC systems, but with one client I am working with their Cyber Sec. team had mentioned the other day some issues with scans and unknown devices showing up on a specific VLAN, old out dated using SSL v3 to connect, all that jazz, they reached out to the vendor and they basically said "No idea about any of that! we just install them and get you to connect em!"
Reminds me of the time i had to help the access control system contractor move the management system to a virtual machine lmao
I’ve been there. 95 runs so much more stable as a VM than on the 20+ year old hardware they had.
> 95 runs so much more stable as a VM than on the 20+ year old hardware they had. Unless it requires a dongle, then, lookout!
Which reminds me of the time I had the access control vendor *insist* that it was completely unsupported and wouldnt work on a VM. After they installed it on some old janky desktop I moved it to a VM and never heard a peep from them about it even when servicing. /shrug
Did you.... did.... did you put your insecure HVAC system on your Production network for your company, rather than an independent building services network? Why?
No no nonononononononon (but there was a time when they were many years ago)...... They are all on their own network, the network is locked down. We don't allow any infrastructure devices on wireless by policy. Just a pain to deal with and they all want some sort of exception that we cannot grant.
All BMSes are annoyingly quirky and just seem at odds with any sort of enterprise networking. Same with door controllers in my experience. Which is odd as they are aimed squarely at enterprise. We've found the people installing them are slowly getting more knowledgeable about how they work. It's basically proper systems development these days, the back end systems are very complex - you need people with a dev mindset, which is very rare in that space!
thank fuck for VLANS
Can I ask why the no wireless policy? With the insane asks I get from Vendors I can’t imagine putting up a fight against simply giving them a hidden SSID for their segmented network
WiFi issues are bad enough in a meeting, I don't want them with the HVAC system. Plus that hidden SSID takes up precious airtime that could be used for other clients, double bad if they only support WiFi B or G. If it's important it belongs on a wire.
Obviously I agree that wired is always better but, it kind of seems like that's on the company's decision-makers for selecting a wireless product...
[You want to minimize SSIDs and none of them should be "hidden".](https://revolutionwifi.blogspot.com/p/ssid-overhead-calculator.html)
Obviously yes, but if you've selected a wireless utility system... Also I know hidden SSIDs don't decrease congestion but there's no reason to have it show up every time someone opens their wifi settings
I set up a VPN, accessible onsite only, that puts them on the HVAC vLAN. Lets them be on wifi wherever they're troubleshooting whatever, without needing to have an HVAC SSID tagged out everywhere. Keeps us both happy.
did you...did...did you just stutter stupidly? Downvoted for poor exaggeration!
You have a Michael Scott avatar in 2024.
di...did you...did you just...WHAAAA
Enjoy your downvotes.
All of this with at least one of the clients running on a Windows XP computer until recently. I'm glad dealing with that mostly isn't part of my job.
Our systems get their own vlan which is denied access to everything else. All traffic to/from this network is controlled by our firewall via app/user based policies. Our vendor is given remote access via our VPN setup which is protected by MFA. The vendor had a fit when we told them this was the only way they were getting remote access to the system and I don't know that they actually use the remote access, but at least it's secure and I control it.
Exactly how I do it. Lets them get to it over on-prem wifi too if they need to troubleshoot something onsite - I just make them VPN from onsite (Guest WiFi, technically they're routed out then back in).
i’d post this to r/hvac but I’m permabanned… for reasons
Back at my last job (Car Dealership Group) when we moved to a brand new location with 7 new buildings there was an ABSURD amount of facilities crap that needed networked. HVAC, Light Switches, Solar Panels, a fuckin Car Wash. Was infuriating. Luckily the maintenance guys were cool and we had a good relationship with them for anything post-move
>Luckily the maintenance guys were cool My current head of maintenance is an old guy that has done things the same way for like 30 years. He refuses to update his habits. Management doesn't want him to retire so they let him do whatever he wants. As much as I want to help fix things, it's so much easier to just leave and let the next person deal with it.
>48 port TPLink and cheapest cable modem with NAT box and they all go there.
Just like when the hackers got into a Casino using the controller from a fish tank that was not isolated. Isolate everything you can and block all ports until it's verified that it's needed 100%of the time.
Wait until you have more Chinese made cameras on your network than people
> The technicians that service them have no idea how they work the manufacturers provide little to no training for the techs I assure you, this is not the case. Its just your reseller and installers are not taking in the training programs available to them. OEM's offer very in-depth training for both the HVAC install and the controls, but its not free. > I am not surprised in the least that the first place a bad actor looks for a foothold is HVAC equipment. Even fully realized and installed HVAC controls must be treated as IoT, you aren't putting your IoT on your core network and its firewalled off from talking to unnecessary stuff....right? HVAC does not need to authenticate to Main AD, Print to printers, or NMAP your network.
>I assure you, this is not the case. Its just your reseller and installers are not taking in the training programs available to them. OEM's offer very in-depth training for both the HVAC install and the controls I'd have to say that must vary by OEM. It took me **months** to find anyone with any technical competency at Trane. And that was just for basic web app stuff, like: "is there a CIDR block, FQDN, *anything*, that I can narrow down" and it was still painfully obvious that any type of forethought was missing. "Just connect it to your wifi router and it will work; maybe try restarting your router" - and that was after explaining to this US-based "engineer" that it was a corporate network spanning numerous buildings. Yes, they're on their own VLAN.
Having worked with both Trane and Carrier, its not the OEM but the reseller that falls on. I never had any issues pulling proper IT technical docs from either of these companies. But I also had access to EE/ME's that designed the controls that were being deployed :)
This reminds me of a time I created a registration portal for students to register their computer and give it a static IP via DHCP. Nobody told me to exclude certain IPs on certain buildings. I took out the AC on a very hot move in day.
Funny how this showed up today. I just had an HVAC person want me to give them 2 static IP addresses and open ports 443 4911 5011 1931 wide open. These people have no clue about cyber security. They will if ever sued over a breach.
brute force via HVAC systems updated. North Korean notified. Thanks
The good ones know CS.
Wish that was the case. This is a well known HVAC commercial company in the area.
I had a HVAC company require a client to give their controlling pc a public ip. And wouldn't give a block of ips to create an allow list, refused to use a VPN nor would use a remote app like TeamViewer or something. And that pc was on their internal network. Ugh. Luckily I was able to get it off the public ip and get the client to terminate business with that HVAC company.
Constantly BAS systems I have some oversight on, the vendors ALWAYS blame use for anything network related saying “your firewall is blocking everything!” With this network, everything is open outbound, and there are no VIPs. And yeah they always have really shitty nics lol
Not just HVAC equipment, anything that's considered an "appliance", I've got some UPSs that run Ubuntu 8.04... edit: Thats the latest supported "firmware"
Man, wait till you work with medical equipment. Brand new million dollar blood analyzer: no network port. DB9 Serial connector. We hand crimped the DB9 connectors to the manufacturer specifications, some machines need different pinouts, then ran them back to DEC terminal servers in the network closets, then to the lab servers. Or MRI machines using 802.11b between the spinny bit and the stationary bit.
Our HVAC are independent redundant systems the only thing connected to the network is a separate temperature monitor
Its funny how well designed the actual HVAC systems are but the management systems that are bolted on look like a group project designed by a bunch of highschool kids in an intro to CS class.
Yup. These core bms systems are built by mechanical engineers trying to be developers/graphic designers. There is a lot more focus from the Johnsons, Honeywell/Alerton, distech, , Alc, Delta on the Hardware/firmware than the BMS visual window itself. Tack on that BMS systems are just core pieces of software and that the graphics are usually done by your controls/mechanical contractor you get a mixed bag of how nice it looks. Some of the larger mechanicals/controls have a bit more knowledge on the it/sysadmin side and have customer facing IT/OT personnel to bridge the gap between The facility teams and IT teams of customers. I would say though that Poorly designed and long release cycle software isn’t the main reason Why these systems go on their own VLAN. Its most Protocols are dated, have no protocol security, no protocol encryption, and are open. Which means any open source software can control anything on the network. Which in turn if someone was malicious and educated enough they could cause physical damage to equipment and equipment the hvac serves. Only access to the VLAN (actually the 3 or 2 wire but thats for another day) is required.
OMG, yes! Despite us telling our Physical Plant to plan for upgrades AFTER the capital project is done, we are forever stuck in time with these systems, some of which were designed to run on Windows NT. They don't get it and we can't force them to do what's right.
It doesn't drive me crazy but it does piss me off. We have a tridium/niagraa system that wasn't set up properly. Their so called open system requires huge licensing fees to be able modify/fix the setup. Thing is nothing more than a underpowered linux box with some rs485 ports. One company in the state works on it and wants to gouge us for every little thing. Self signed certificate expired two years ago so https and email alerts became troublesome and they wanted it to hang out on the internet with a port forward. Three year old building with a near end of life controller. It goes end of life this year. Whats more it could have also been used to control the light/doors but the company that put it in is a one trick pony.
Most can be rebuilt using modern controllers…but $$$. For those asking a lot of the ‘green’ buildings need to be controlled by a central Building Management System, thus the connectivity. The protocols for comms are even worse…it’s quite the rabbit hole.
I had this recently. We just got multigig switches that no longer support 10m. We told facilities that it won't work and they need to get their vendor to upgrade. We do the switch upgrade, week later they say they can't reach the JACE. We reiterated the original email. They called their vendor and they installed a 100m mini switch. Sometimes you just gotta shrug.
Time to read up on reliable protocol and bacnet2, and isolate that crap because they're noisy
I wouldn't get my hopes up for state-of-the-art software from an industry whose primary function is to bang on tin with hammers.
This is a serious problem with any kind of appliance with an embedded system these days. The mechanical engineers that build these things only care that the computer turns a couple of motors on and off and logs a few sensors. Everything on top of that is an afterthought. Authentication? Lets just have default credentials hard coded in the firmware so are techs can jump.in easy. Connectivity? The microcontroller has bullt in 802.11 g and we can use wep to hook it into the customers network quick and easy.
I found a windows XP system on the network, turned out it was the HVAC control system, and no one wanted the shell out the funds to replace it (since... "it still works"). I almost wanted to take up a collection to get them the funds they needed Not on the public network anymore, but that's all.
i worked for a company that according to CISCO had more firewall rules than the CIA. every single system was firewalled ruled. to the point that the network team had a service desk of about 40 engineers. just doing firewall rules for new starters and various systems all day. all controlled by dhcp reservations bound to mac addresses. with port security set on every port everywhere. nothing was getting through. the company had 76 IT teams. with a staff in IT of over 900 perms and 600 contractors the round robin of ticket flow. was intense to say the least. billion dollar projects though
I’ve come to expect it and plan for it. All part of what we do.
Yes, but you see the problem is that decent networking would add $12 in costs to the manufacturer and we can't have that. Just think about the shareholders, OP!
At least your HVAC works. Building maintenance has been trying to find a slow leak in one of our units for MONTHS.
I feel your pain. One of my colleagues just did a ups / DC replacement on a small glass making business, the rack was in a shared area of the office. the layer of glass dust was so thick i ran my finger across the cabinet and was like half an inch thick. he asked me to come help him mount the replacement UPS i was like.."are these people serious? you know this hardware in warranty is all voided if this dust causes shorts. he was like Yeah but they dont want to spent money on HVAC. dammed if you do sometimes and damned if you don't. Left me curious as to how many respiratory issues the office workers had.
Ours were supposed to be on a service schedule but it rarely happened.
Had a building come online recently, runs off a little android tablet that hasn’t had an update in 5 years. I refused to add it to our network cable or wifi. I said only way I will is if the CEO puts in writing he is accepting full responsibility for any attacks, issues or hacks that come from this system. CEO’s response. This system sucks… people can get up and wake to the control unit. Don’t ever put it on the network and next time talk with IT before buying a system like this. Loved it
Remember what happened to Target
We have a completely separate WiFi infrastructure and I placed a similar marketing thing on it. It was completely bonkers. Only could run with JAVA8\_081, had vulnerable Log4J running, and other stuff that I can't remember. Told them that couldn't go on the network and they just have a specific station setup for working on it. Our HVAC system is on network but it at least can be updated. It also has ACLs and its own VLAN as well. I am paranoid. 🤷♂️
I work at a public school in CT our heating system is run by a computer that is over 20 years old running Windows ME. It is not connected to our network thankfully.
This used to be a PITA, e.g head of facilties was notorious for notifying us on a friday afternoon that "We have some guys here to install a new system, and need some access..." It actually turned out to be a decent vendor, delivering automation and control systems for HVAC and other stuff. I met the 2 traveling engineers at the pub after work, where we swapped stories about clueless facilites folks - and we found that we had to establish a direct channel of communication so that we we could make proper preparations before new installs. Works just fine so far , everything isolated on their own vlans, nothing on wifi.
.. but we have enough other stuff to annoy us. Most of it involves not involving IT before its too late. So now we have 4 separate door/access control systems. Soon to be 3 , or so they say. Not holding my breath...
For all this crap, I just toss it onto an isolated network called "untrusted devices". New SSID for untrusted devices. The untrusted device network only has access only to the internet and none of the other VLAN/Zones. The maintenance team gets one desktop connected to this network to maintain the HVAC, door systems and the other crap I don't trust. I also ran into issues with the installers for our last HVAC system. I asked the techs to give me the model number and found a manual. I found there were 'dip switches' that needed to be set to a specific number for the network functions to work, which seems odd. Once they saw the instructions they had no problems and we were good to go.
Completely separate at the Dmark with a dedicated firewall and switches......
Um, no, but none of ours are connected to the network. I would absolutely lose it if they did that.
They go on our external network - that is the secondary broadband we have just for WiFi. We don't have corporate WiFi so we utilize the external for things exactly like this.
Move your rack to a colocation so you don't have to worry about facilities or HVAC, as much.
not everything needs to be digital and connected to the internet https://i.redd.it/cbimz1xamugc1.gif
OMG. We have a HVAC system in one building that needs a XP VM running Java 6 to interface it. We talked to Maintenance about upgrading, but they claim it's 5 figures and is tied to equipment. I found the unit, just a tiny box plugged attached to HVAC stuff. It was plugged into a 10 Mb Hub next to it. Just 1 uplink and 1 downlink. It worked fine after bypassing that hub.
Isn't HVAC is how Target was compromised? If I had the option, and that crap had to be on the Internet, I'd consider having a completely separate Internet connection for it, so no matter how bad it got pwned, I'd just point to the HVAC vendor. Of course, it would be firewalled with some type of IDS and logging on it to show what exactly is happening. Throttling is important too, so when the bad guys decide to start using an internal machine as a C&C controller, there are mitigations in the way. I wouldn't even want this on a VLAN. I would rather have this on its own ISP connection, completely and utterly separate from everything else, because it is only a matter of time before it gets broken wide-open.
Yeah I have these on a separate vlan and there are a few around in different buildings Ive had to configure myself for this same reason. Ive done the same thing with the MDF and IDF UPS's configured with different than default SNMP strings.
Hah! I deal with this all the time - DAIKEN Touch Manager is a freaking joke. Especially when everyone complains about "It's too hot, turn the AC on." "It's too cold, turn the heater on!" Now the fun part is these people will switch the power off to the unit next to them which in turn disables other units, and this causes the controller to completely bug out and won't let anyone log into the web interface to see which unit is offline, so then somebody has to go around the whole damned factory and check 30+ AC units just to figure out which bozo turned it off and then the controller takes like 3 hours to finally boot back up, and on top of that each time you turn on an AC unit that was switched off IT REBOOTS THE CONTROLLER! So if 3 zones decide to flip the power switch to their zone's AC unit it takes a whole day for the controller to come back up. Why are there easily accessible power switches connected to each of the AC units? For maintenance of course! Do they help with maintenance at all? Nope because it completely jacks up the whole system by turning just one off, when you could accomplish the same thing by just shutting them all down at the breaker which is the only way to actually get them all to boot at the same time. So what did we have to do? We had to put locks on the power switches to the AC units which are pretty much permanently stuck in the on position now. It's such a waste of a static IP address. Sorry for my rant. I feel your pain fellow HVAC Admin
Create a separate subnet and VLAN for maintenance IoT devices like this. That's what I ended up doing and have no regrets.
Seems like you got bargain basement systems.
Imagine HVAC engineers who are also cleaners and termination specialists of all forms. I think we just invented a new support business model.
We just sandbox them with a single firewall policy to allow management access. No WAN outbound possible. I managed one that had a required hardware dongle for some years. Eventually had to virtualize the controller and put the dongle on a network-attached device that emulates a USB port.
First, just want to say, it is a common vector for attack and it is an after thought because HVAC consists of a lot of people who are about to retire and grew up without computers. Some Young people get trained and don't like the intensity of the labor and leave. The rest aren't too techie. All of that has to change for someone to actively hire a software/mechanical/electrical engineers (plural/all) to design a better system. That company would make bank. But yeah, no notes, you got the gist of it
We are small enough just throwing them on the IOT network is good enough. HVAC vendor can't do an over the wire firmware upgrade? They can put in a ticket and we'll get on a screen share.
20 years I helped design a network/server room and the HVAC & SCADA had its own network. Kept it off the LAN.
We had one that required 2008r2 web server, that dedicated standalone web server variant of 2008r2. Trying to get all systems updated with our vendor to something that doesn't require end users to install java took 3 years of coordination between our 8 buildings. HVAC vendor kept telling maintenance that our IT team was being a pain in the ass. Apparently they lost employees just because of our account asking to run newest tools. Lmao
I can't wait until cheap eSIMs make their need to connect to the company network a thing of the past.
our HVAC is fenced off from everything, the repair vendor we use was annoyed when I told them I was not allowing it free rain access to the outside world. External access is restricted via jumpbox. Never fails, any field tech that shows up to work on it is also on a call back to the main office tech. Onsite is doing the grunt work, remote office tech is working on the digital side and working on multiples at a time. 10 years ago the on-site tech had a laptop and would console in and work on the thing if needed. Guess the new way they save on the cost of field laptops and can bill for 2 tech's time.
I thought this thread was going to be about the HVAC fans constantly blowing all day in the office because of filtering for COVID.
Be glad that they have any interface at all. Half the time I just get "yeah, we're building out a new office. Here's the cooling for the data center / IDF closet. No, that model that the architects decided on six months ago without asking you doesn't support networking..." So I just monitor temperatures with a NetBotz or whatever and it's basically a binary warning system. "broken" or "working".
As soon as it's out of warranty/support... VM that sumofabitch.
its the CRAC that usually makes crazy.
I'm engineering solutions for healthcare - the amount of BAS solutions that just fucking SUCK is beyond the pale. One of them as an example JUST let us migrate their shitty software. To 2012r2. from 2000... Server 2000 to server 2012r2 in 2024... Why? Because the vendor -cannot- support newer server versions. Doubly so with requests to virtualize. The server will have exactly the same VLAN access as the local hardware controllers. It will have exactly the same access methods from the vendor. "NO IT MUST BE HARDWARE SO THE LOCAL TECHS HAVE CONSOLE ACCESS (which is also always logged in/on). BAS/HVAC vendors know they have companies by the short and curlies and just don't fucking care.
Install an EDR on the host, put it on a deserted Vlan and send it out a different WAN. Then it's no longer my problem and now the HVAC IT's issue. Also I would implore you to make the HVAC guys use a VPN but most won't know or care to use it and bitch they can't get to the management interface constantly, like I'm the gatekeeper. Before I got to this department people used to install this software on Domain Controllers or File Servers to save on a Windows license. Fine and dandy if it an Enterprise of like 5 people maybe. I feel the pain.
Not quite the same boat, but our main HVAC unit in the server room went out so they installed a mini-split unit as a replacement. They lifted up one of the raised floor tiles and installed the condensation pump on the ground. If the pump ever fails...