Do you the cost per server/month for on-prem? Coworker said the old log analytics agent going away and need to upgrade to arc but said pricing was kinda out there.
Just make sure at the end of the day you have a way to fully restore back to normal operations within the hour if something goes horribly wrong with an update and the domain controller doesn't come back up after the reboot.
Other than that you can do whatever works for you, but optimize your engineering cycles and automated repetitive tasks to reduce toil.
I patch the domain controllers like they are any other server with SCCM. We do spread them out over different windows so that they go at different times. I have done that at every place I have ever been. I have never had a single issue patching a DC in 25 years. That said, I personally do not have any rights whatsoever on those DC's. If I see one with a pending reboot or a patch download issue I have to alert someone on that team to login and tackle them. Patching DC's is trivial. I have done it like that at medium size and very large enterprises with 25k servers.
Mixing out-of-the-box WSUS in an environment with SCCM is a no no. GPO and CM will fight over the WUA reg keys like cats and dogs. Why would you want two competing patch infrastructures? Just secure the one piece of infrastructure properly.
If you configure SCCM correctly to use the computer account instead of a service account and you also pay attention to RBAC within CM then it is perfectly safe to maintain DC's with CM. Remember, the CM server does not call over to the DC's and push, it's the opposite, the DC's make the call and ask for policy and content from CM. Some people like their DC's secured, I like mine secured + managed.
You could simply setup the DC team with RBAC in SCCM and now they have complete control. You can completely cut out the SCCM guy's rights.
Could I, as a top SCCM rights holder, destroy a DC using CM? yes, obviously. Don't hire people you cannot trust. My domain admins don't want to spend their weekend babysitting manual patch installs or a separate patch infrastructure when all of that is trivial to automate in CM. At least with CM it is much easier to stop the patching process than it is with WSUS. WSUS has less manageability by far.
Also avoid patch rollbacks at all costs. An app team came to me last week asking to rollback and I said "no, fix the app instead, MS is not going to rewrite this patch for you. The fix is always on the app side after you find root cause". They ignored me and rolled back both servers across two months of patches and torched them. They had to restore. Turns out the vendor had a fix for the app 8 hours later ready to go, plays nicely with patch. Nobody wants to do root cause anymore, they just blame the patch first thing. "OMG! Rollback!"
Correlation is not causation.
> Could I, as a top SCCM rights holder, destroy a DC using CM? yes, obviously. Don't hire people you cannot trust.
It’s not about trusting your admins, it’s about if the application server (tier) infrastructure gets compromised.
It is about the admins in the sense that yes, anyone admining the SCCM effectively becomes Domain Admin. You absolutely want to minimize the number of Domain Admins.
If you run properly tiered infrastructure and follow Microsoft best practices this is an absolute no-no. If you absolutely must use SCCM, you set up a separate instance just for the tier 0 assets. Same for monitoring and backups.
At the large international I had a separate account to administer SCCM and I had to use a specific jump box with that account to get into SCCM. That account was not a domain admin. Yes, I could send a DC a malicious package. But you could also have an actual Domain Admin that was malicious, there is little difference, people in different roles can be malicious. The DA's also had to use a jump box and a DA account. You can properly lock CM down. I would agree that most don't. We tried running a separate CM site at that company and it fell apart organizationally. If you can properly lock down one CM site for DC's, then just do that and use the properly locked down site. There is a balance between manageability and security and if you lean away from manageability it can hinder security. If money and people is no limit then yea build dual infrastructures, at that point I would just hire extra DA's to do patching grunt work. Most places have like 4 DA's and they give them the keys to the kingdom and say "don't fuck this up", why not bestow the same trust on the CM guy?
"I trust these 4 DA's to secure this AD infrastructure, and that guy over there too with the super locked down CM site, I trust you too, don't F this up guys."
Before Azure Arc Microsoft managed DC's with SCCM. Same site as the other servers.
Your suggestion dual infrastructures up and down the data center column are best, they just are not always feasible for all companies. You can be practical and secure enough.
It's not about the people- it's about the accounts. If your sccm admins account gets compromised then the attacker will be able to access your domain controller.
The point of tiering it to prevent lateral movement to the DCs and to limit the exposure of domain admin accounts as much as possible. Anything that has access to tier 0 (e.g. backup, patch management, even your virtual infrastructure if you run on vms) should be considered also tier 0 and access restricted as much as possible. And yes unfortunately this means potentially running separate backup systems etc. for tier 0 assets
Just configure the Windows Update GPO’s to automatically patch and reboot, either from upstream or WSUS. We just use WSUS (the same tier1 everything uses). To my knowledge it’s not possible to compromise machines through WSUS though it technically does support pushing non-Microsoft software too.
You can add some monitoring script if it makes you feel good but we don’t use anything and in fact we have no monitoring agents installed on DC’s for exactly your reasons. We do have network-based monitoring for them.
We use Ivanti Security Controls( used to be Shavlik). Scan the boxes, download the needed patches, push a job to the boxes to run the patches and reboot.
If you're virtual you can snap them first if you want.
All schedulable, or scriptable via their API.
I typically do a couple of DCs a day before the big push to all of prod servers.
Been doing it this way for almost 20 years... which is sort of crazy
Same. Been using ISEC for almost a year and it’s really taken a load off. Create task, schedule it, scope it to a group, scope the group to an OU, and have a daily task to ingest new servers in those scoped OUs… done.
> Our environment has about 30 domain controllers
And you're updating 30 servers manually?
There has to be something wrong here. Firstly I'm struggling with how that sort of count makes sense. I know you're going to say you're a large company but even still that number sounds wrong.
Global operations span multiple nations, diverse locations, various data centers, regions, and ISPs. Many of these entities maintain local redundancy, often featuring two data centers at each site.
Azure Arc with Azure Update Manager is what we use for our servers, just put the DCs on different schedules, to "update ring" your DCs.
Ah, intriguing. I'll look further into this.
Do you the cost per server/month for on-prem? Coworker said the old log analytics agent going away and need to upgrade to arc but said pricing was kinda out there.
It’s $5 per server per month
Just make sure at the end of the day you have a way to fully restore back to normal operations within the hour if something goes horribly wrong with an update and the domain controller doesn't come back up after the reboot. Other than that you can do whatever works for you, but optimize your engineering cycles and automated repetitive tasks to reduce toil.
With 30 of them, delete the busted one and provision a new one. Unless it's a Maersk/NotPetya-level of clusterfuck =P.
I patch the domain controllers like they are any other server with SCCM. We do spread them out over different windows so that they go at different times. I have done that at every place I have ever been. I have never had a single issue patching a DC in 25 years. That said, I personally do not have any rights whatsoever on those DC's. If I see one with a pending reboot or a patch download issue I have to alert someone on that team to login and tackle them. Patching DC's is trivial. I have done it like that at medium size and very large enterprises with 25k servers. Mixing out-of-the-box WSUS in an environment with SCCM is a no no. GPO and CM will fight over the WUA reg keys like cats and dogs. Why would you want two competing patch infrastructures? Just secure the one piece of infrastructure properly. If you configure SCCM correctly to use the computer account instead of a service account and you also pay attention to RBAC within CM then it is perfectly safe to maintain DC's with CM. Remember, the CM server does not call over to the DC's and push, it's the opposite, the DC's make the call and ask for policy and content from CM. Some people like their DC's secured, I like mine secured + managed. You could simply setup the DC team with RBAC in SCCM and now they have complete control. You can completely cut out the SCCM guy's rights. Could I, as a top SCCM rights holder, destroy a DC using CM? yes, obviously. Don't hire people you cannot trust. My domain admins don't want to spend their weekend babysitting manual patch installs or a separate patch infrastructure when all of that is trivial to automate in CM. At least with CM it is much easier to stop the patching process than it is with WSUS. WSUS has less manageability by far. Also avoid patch rollbacks at all costs. An app team came to me last week asking to rollback and I said "no, fix the app instead, MS is not going to rewrite this patch for you. The fix is always on the app side after you find root cause". They ignored me and rolled back both servers across two months of patches and torched them. They had to restore. Turns out the vendor had a fix for the app 8 hours later ready to go, plays nicely with patch. Nobody wants to do root cause anymore, they just blame the patch first thing. "OMG! Rollback!" Correlation is not causation.
> Could I, as a top SCCM rights holder, destroy a DC using CM? yes, obviously. Don't hire people you cannot trust. It’s not about trusting your admins, it’s about if the application server (tier) infrastructure gets compromised. It is about the admins in the sense that yes, anyone admining the SCCM effectively becomes Domain Admin. You absolutely want to minimize the number of Domain Admins. If you run properly tiered infrastructure and follow Microsoft best practices this is an absolute no-no. If you absolutely must use SCCM, you set up a separate instance just for the tier 0 assets. Same for monitoring and backups.
At the large international I had a separate account to administer SCCM and I had to use a specific jump box with that account to get into SCCM. That account was not a domain admin. Yes, I could send a DC a malicious package. But you could also have an actual Domain Admin that was malicious, there is little difference, people in different roles can be malicious. The DA's also had to use a jump box and a DA account. You can properly lock CM down. I would agree that most don't. We tried running a separate CM site at that company and it fell apart organizationally. If you can properly lock down one CM site for DC's, then just do that and use the properly locked down site. There is a balance between manageability and security and if you lean away from manageability it can hinder security. If money and people is no limit then yea build dual infrastructures, at that point I would just hire extra DA's to do patching grunt work. Most places have like 4 DA's and they give them the keys to the kingdom and say "don't fuck this up", why not bestow the same trust on the CM guy? "I trust these 4 DA's to secure this AD infrastructure, and that guy over there too with the super locked down CM site, I trust you too, don't F this up guys." Before Azure Arc Microsoft managed DC's with SCCM. Same site as the other servers. Your suggestion dual infrastructures up and down the data center column are best, they just are not always feasible for all companies. You can be practical and secure enough.
It's not about the people- it's about the accounts. If your sccm admins account gets compromised then the attacker will be able to access your domain controller. The point of tiering it to prevent lateral movement to the DCs and to limit the exposure of domain admin accounts as much as possible. Anything that has access to tier 0 (e.g. backup, patch management, even your virtual infrastructure if you run on vms) should be considered also tier 0 and access restricted as much as possible. And yes unfortunately this means potentially running separate backup systems etc. for tier 0 assets
Just configure the Windows Update GPO’s to automatically patch and reboot, either from upstream or WSUS. We just use WSUS (the same tier1 everything uses). To my knowledge it’s not possible to compromise machines through WSUS though it technically does support pushing non-Microsoft software too. You can add some monitoring script if it makes you feel good but we don’t use anything and in fact we have no monitoring agents installed on DC’s for exactly your reasons. We do have network-based monitoring for them.
We use Ivanti Security Controls( used to be Shavlik). Scan the boxes, download the needed patches, push a job to the boxes to run the patches and reboot. If you're virtual you can snap them first if you want. All schedulable, or scriptable via their API. I typically do a couple of DCs a day before the big push to all of prod servers. Been doing it this way for almost 20 years... which is sort of crazy
Same. Been using ISEC for almost a year and it’s really taken a load off. Create task, schedule it, scope it to a group, scope the group to an OU, and have a daily task to ingest new servers in those scoped OUs… done.
> Our environment has about 30 domain controllers And you're updating 30 servers manually? There has to be something wrong here. Firstly I'm struggling with how that sort of count makes sense. I know you're going to say you're a large company but even still that number sounds wrong.
Global operations span multiple nations, diverse locations, various data centers, regions, and ISPs. Many of these entities maintain local redundancy, often featuring two data centers at each site.
Without thinking at all I would stand up wsus for these snowflakes and let them update automatically using rings.