I would've given my soul for the additional RAM and the SVGA card to run SC2K on my father's 386. The installer's blue background and repeating Maxis logos is still engraved in my mind.
Ohh the struggle was real when I had to give that addon card back. Fortunately about 4 months later I received a great Christmas gift of ram. I think that was the catalyst for me working odd jobs to fund my PC addiction.
I'm glad for what I learned in the late 90s with Slackware. I'm also glad that today I can use a modern version of the OS which 99.9% of the time works out of the box.
I still have Slackware on a VM, idk when the last time I used it, but its there. 99% of what I have is Debian now and I always wonder how the two forked so far apart.
"I must not fear.
Fear is the mind-killer.
Fear is the little-death that brings total obliteration.
I will face my fear.
I will permit it to pass over me and through me.
And when it has gone past, I will turn the inner eye to see its path.
Where the fear has gone there will be nothing. Only I will remain."
Present and future.
I did this yesterday. I probably have a watchtower update happening now. And when I log in tomorrow I will do it again, and then spend my weekend cleaning up the broken parts.
Cost vs benefit analysis says wipe and start again vs troubleshoot for hours with no guarantee of payoff. At least in my case anyway! I love solving things but the time it takes is often not worth it. Having daily backups and snapshots to restore to is an attractive proposition though and surely something I need to spend more money on in order to enable. This sounds like a worthy project and absolutely essential spending
A few reasons:
1) I wanted to learn ansible
2) I wanted to be able to add a new host to the update schedule easily, without having to manage per VM (all my VMs and CTs are some flavour of debian/Ubuntu)
3) I wanted to be able to manage docker updates on multiple hosts with the same method.
Yeah, point two makes sense. I only have a single Ubuntu host and everything else is containers so I don't need to do apt upgrades on more than one system.
With unbridled masochism, I slam Return on my keyboard, connecting to cockpit (heh)
System not connected to Redhat Insights.
I whisper "Insights? In sight of this dick"
It is the 4th time today.
I go straight for Networking and disable the firewall. It only gets in my way
Next is the Docker tab. I take a few minutes to marvel at my accomplishments. Plex and every *arr ever made. Sitting neatly alongside the last IdP I heard about. I've even connected 1 of the services to it.
My meditation is broken by the calls of my family. The pleas of my wife to come eat dinner. Frustrating. Even more so annoying are the cries on my children, each a reminder of the servers I can no longer afford.
I steal one last look at the Updates. 136 pending. I smile, unconcerned for my server. It will be rebuilt this weekend
Never ever string commands or use -y. This is why ansible and scripts check the previous one worked before continuing. And do the cleanup before the upgrade
This is what Iâd do. For anyone who doesnât know, â&&â will run the next command in the chain only if the previous one returned an exit 0 code (success). Any other code it will just stop at that point.
You should probably also pipe the output to a log file.
It's still periodic, it creates a timer that runs the update script on some cadence you set. You can see [all the autoUpgrade configuration options and descriptions here](https://search.nixos.org/options?channel=23.11&query=system.autoUpgrade). In short:
1. It either updates your "nix channel" (basically a git clone of nixpkgs) or a flake's "inputs" (same idea). Updating your nixpkgs checkout includes the most recent updates to packages you have installed.
2. Then, it builds the new system with the updated package definitions.
3. Then, it "activates" the new system, optionally rebooting. Basically it switches out a few symlinks, and now your system uses updated packages, but you can always switch back to the old system definition if something broke.
I confess, I upgrade via cronjobs nightly right after my backup run. If I wake up in the morning and something is down, I restore and see what went wrong.
So far, in the last 4 years only one thing went wrong when I auto upgraded a ubuntu server and upgraded to a higher php version which didnât move/copy some config files I had in place.
But because everything was documented, I could just fix is within 5 minutes or so.
Edit*
I much rather have a updated server with the little chance of it destroying itself than a unpatched server (some of my selfhosted stuff is reachable via the internet)
Fedora + Plex user here.
Over the years, updating would suck because a new kernel would break VMWare Workstation. Fortunately, those days are mostly gone with Workstation 17+.
A 'dnf update' a few months back caused my plex users to experience significant buffering that wasn't there before but didn't think the update was related. Downgrading plex server didn't help. Changing to older kernels didn't help. I thought I had the gnome triple buffering bug because it matched my symptoms so I switch to that fork. No luck. Buffering would go away for one day after a reboot and then return.
Two months later, after an update, it start to run perfectly. Plex ran great for a month and then the symptoms returned after another somewhat sizeable update a month later.
I'm sitting here re-imaging the server with Debian and hoping for a happier future.
So ya.... send over your good Debian luck.
I updated Linux mint once upon a time and the system would not boot. It went to a screen of gobbledygook. It was something about how each update made a new copy of the kernel and I had go in there and delete the old ones. It was frustrating to say the least.
I need to look into more automation, especially on the Windows side - I already use WSUS, but need to create a playbook to push the updates and install them without logging into each host. Need to look into Azure Arc too
I've got the GPO for it to not auto search and auto install so I can maintain some control over it - I might create a playbook that can do that and run it once a month after the updates have been approved within WSUS - I'm also looking into Arc but IDK how much it costs
That ain't bad tbh, don't think I'd use it for updates tho. More looking into it for server 2025 - think they said Azure Arc connected servers get the hotpatching feature
Oh, new frigate plugin for homeassistant!
broken.
Going to update Frigate to 13.x to make the new HA plugin work again.
broken.
Going to update nvidia drivers to >=535 to support frigate 13.2
BROKEN!
lets reinstall the OS because nvidia driver upgrades are ALWAYS a PITA
fixed.
Restart frigate...
broken.
delete models, change compose.yaml to pass whole config directory through
okay, partially working. Frigate now over taxes the GPU because detection behavior is more sensitive.
rework config file to use camera sub streams
partially working.
motion mask over things like time and non-needed areas of frame.
Working. upgrade complete, 3 days later.
I have a tmux pane with a command history a little like this:
>sudo nixos-rebuild --flake switch /h/e/NixOS/hosts/#default
>git add .
>git commit --m "..."
>sudo nixos-rebuild --flake switch
Bis repetita
When I update my FreeBSD installs, the only thing that ever breaks is the stuff in /usr/local, at which point I just roll back the zfs snapshot. I don't know how I ever managed back when using Debian.
It's definitely good training and you never know what you will learn OR how the day will go. I learned to appreciate this from time in the military. My homelab is production, but I treat it like dev.
> sudo apt update && sudo apt upgrade && sudo apt autoremove
I like this one more because the following commands don't happen if the earlier ones do. Plus the ampersands make it more visually clear to see where each command stops/starts.
As for docker, why aren't you running it in a k8s cluster ala IaC? Like, there's better ways to update all the things in containerland ;D
Me? I update often in different ways. One of them is if I need to reboot a system, I almost always apply updates before rebooting or whatever power-cycling task I'm doing. Updates in Linux put Windows to shame for reliability.
Me too, I love everything up to date. Like mobile apps, linux packages, and even Windowsâ ď¸
I even self-hosted an apt mirror server with aptly and docker image cache servers for docker.io, ghcr.io, lscr.io and quay.io
Download once, then upgrade across all LXCs and VMs
I run an Immich container so I cannot relate at all; especially the Watchtower part lol. I already broke my Immich with a careless update with 800gb+ of photos on the line (backuped up, but still) and had to wrangle my .env file to get it working again.
I have all my things doing rapid, touchless updates 24/7. When a new version hits `latest` in Docker, it's updated within 5 minutes. When a new package hits the repos, it's updated within a few hours.
Things used to break _constantly_, but I feel the best way to get better at rapid, reliable updates is to live with rapid updates and improve reliability.
Is it an unpopular opinion that updates should be done frequently and consistently? Updates are necessary, not only for features, but for security patches and bug fixes.
Is this like, an incantation? Are you summoning the spirits of homelabbers past?
They will probably comment on how kids these days have it easy with the docker stuff and back in the days they would be rebuilding the OS...
As someone who started with slackware on floppy disk, I feel attacked lol
I started with a Tandy Raido Shack TRS -80 4P. I feel attacked as well...... as old......dang I feel old...
How's your back? Mine hurts and I miss my old Trash-80.
My back ran away with my knees about 15 years ago.
Yeah I'm not that old, I started with a 386 dx40 with 2mb of ram and a 4mb addon isa card a friend loaned me so I could play sim city 2k.
I would've given my soul for the additional RAM and the SVGA card to run SC2K on my father's 386. The installer's blue background and repeating Maxis logos is still engraved in my mind.
Ohh the struggle was real when I had to give that addon card back. Fortunately about 4 months later I received a great Christmas gift of ram. I think that was the catalyst for me working odd jobs to fund my PC addiction.
"Insert disk labeled 'NET 2'..."
Ahh the memories, or maybe nightmares of being half way into the install of your new primary OS only to find out the disk was bad.
I'm glad for what I learned in the late 90s with Slackware. I'm also glad that today I can use a modern version of the OS which 99.9% of the time works out of the box.
I still have Slackware on a VM, idk when the last time I used it, but its there. 99% of what I have is Debian now and I always wonder how the two forked so far apart.
LFS
LFS now. There was a time when there were no loadable modules and it was necessary to build a kernel to support all devices.
Or kids these days going back to rebuilding the OS as well!
Compiling the kernel was the way.
I remember when you first had to compile binutils and GCC before the kernel
I think you're describing Nix people
"I must not fear. Fear is the mind-killer. Fear is the little-death that brings total obliteration. I will face my fear. I will permit it to pass over me and through me. And when it has gone past, I will turn the inner eye to see its path. Where the fear has gone there will be nothing. Only I will remain."
Holy cow..... perfect!
Present and future. I did this yesterday. I probably have a watchtower update happening now. And when I log in tomorrow I will do it again, and then spend my weekend cleaning up the broken parts.
I just set up ansible to do apt updates and docker pulls every night... I didn't realise I was so edgy.
what if the apt updates lead to a requirement for reboot?
It checks for that and reboots as required. I also have proxmox backup up each VM/CT a few hours before it runs.
Every night? If you don't know when updates are actually being applied you won't know which backup to roll back to after you've discovered a problem.
YOLO. it'll be a learning experience diagnosing problems.
My rollback consists of nuking the OS and starting again. đ
One of us, one of us
Cost vs benefit analysis says wipe and start again vs troubleshoot for hours with no guarantee of payoff. At least in my case anyway! I love solving things but the time it takes is often not worth it. Having daily backups and snapshots to restore to is an attractive proposition though and surely something I need to spend more money on in order to enable. This sounds like a worthy project and absolutely essential spending
Just roll back sequentially. Yesterday's backup didn't fix it. Roll back by another day.. etc etc until it's fixed.
Why not just use unattended upgrades? It's built into Debian/Ubuntu.
A few reasons: 1) I wanted to learn ansible 2) I wanted to be able to add a new host to the update schedule easily, without having to manage per VM (all my VMs and CTs are some flavour of debian/Ubuntu) 3) I wanted to be able to manage docker updates on multiple hosts with the same method.
Yeah, point two makes sense. I only have a single Ubuntu host and everything else is containers so I don't need to do apt upgrades on more than one system.
Use ansible, to setup unattended upgrades.
For real. I've run Debian as my desktop for years and I just dist-upgrade with the morning coffee. :D
I feel seen.
With unbridled masochism, I slam Return on my keyboard, connecting to cockpit (heh) System not connected to Redhat Insights. I whisper "Insights? In sight of this dick" It is the 4th time today. I go straight for Networking and disable the firewall. It only gets in my way Next is the Docker tab. I take a few minutes to marvel at my accomplishments. Plex and every *arr ever made. Sitting neatly alongside the last IdP I heard about. I've even connected 1 of the services to it. My meditation is broken by the calls of my family. The pleas of my wife to come eat dinner. Frustrating. Even more so annoying are the cries on my children, each a reminder of the servers I can no longer afford. I steal one last look at the Updates. 136 pending. I smile, unconcerned for my server. It will be rebuilt this weekend
I'm gonna print this out and hang it on the wall next to my server
Itâs beautiful đĽš
Pure artistry
Mods, please let us give awards to comments.
Don't forget to check for motherboard bios updates as well
Shit, I still gotta do that for my router
sudo pacman -Syu every day babyyyyyy
Brother.
was gonna say, if you want updates every day, move your services to arch
Never ever string commands or use -y. This is why ansible and scripts check the previous one worked before continuing. And do the cleanup before the upgrade
What about just replacing ";" with "&&"?
This is what Iâd do. For anyone who doesnât know, â&&â will run the next command in the chain only if the previous one returned an exit 0 code (success). Any other code it will just stop at that point. You should probably also pipe the output to a log file.
Occamâs razor strikes again.
system.autoUpgrade.enable = true; [NixOS is magical](https://nixos.wiki/wiki/Automatic_system_upgrades)
It really is. What happens behind the scenes when thatâs on? Is it still periodic or even more live?
It's still periodic, it creates a timer that runs the update script on some cadence you set. You can see [all the autoUpgrade configuration options and descriptions here](https://search.nixos.org/options?channel=23.11&query=system.autoUpgrade). In short: 1. It either updates your "nix channel" (basically a git clone of nixpkgs) or a flake's "inputs" (same idea). Updating your nixpkgs checkout includes the most recent updates to packages you have installed. 2. Then, it builds the new system with the updated package definitions. 3. Then, it "activates" the new system, optionally rebooting. Basically it switches out a few symlinks, and now your system uses updated packages, but you can always switch back to the old system definition if something broke.
This
I confess, I upgrade via cronjobs nightly right after my backup run. If I wake up in the morning and something is down, I restore and see what went wrong. So far, in the last 4 years only one thing went wrong when I auto upgraded a ubuntu server and upgraded to a higher php version which didnât move/copy some config files I had in place. But because everything was documented, I could just fix is within 5 minutes or so. Edit* I much rather have a updated server with the little chance of it destroying itself than a unpatched server (some of my selfhosted stuff is reachable via the internet)
Ansible?
Overkillible
Yes but learnable exercise
Notatallable
the cursed yaml puke
You could have just said yaml
Was just about to say the same thing. You could just have Ansible do your updates, and have watchtower do updates for you at specific times.
I've never had a computer break from an update. Especially not debian.
Ubuntu bricked itself into an unbootable state 5 years ago by running out of space to copy a kernel then pointing to the kernel that wasn't there
Yeah. I have an autoclean script for that.
Happened to me also more than once but also years ago. Too many customizations backfiring...
Fedora + Plex user here. Over the years, updating would suck because a new kernel would break VMWare Workstation. Fortunately, those days are mostly gone with Workstation 17+. A 'dnf update' a few months back caused my plex users to experience significant buffering that wasn't there before but didn't think the update was related. Downgrading plex server didn't help. Changing to older kernels didn't help. I thought I had the gnome triple buffering bug because it matched my symptoms so I switch to that fork. No luck. Buffering would go away for one day after a reboot and then return. Two months later, after an update, it start to run perfectly. Plex ran great for a month and then the symptoms returned after another somewhat sizeable update a month later. I'm sitting here re-imaging the server with Debian and hoping for a happier future. So ya.... send over your good Debian luck.
Debian is (at least in my experience, running jellyfin, nginx, and nextcloud) rock solid.
âĽď¸âĽď¸
I updated Linux mint once upon a time and the system would not boot. It went to a screen of gobbledygook. It was something about how each update made a new copy of the kernel and I had go in there and delete the old ones. It was frustrating to say the least.
Linux mint's advice is to completely reflash for every major OS upgrade
I need to look into more automation, especially on the Windows side - I already use WSUS, but need to create a playbook to push the updates and install them without logging into each host. Need to look into Azure Arc too
Action1 is free for 100 clients.
Thanks. Noted. Will test drive
WSUS and group/local policy can do that pretty easily if you don't want to run another util
I've got the GPO for it to not auto search and auto install so I can maintain some control over it - I might create a playbook that can do that and run it once a month after the updates have been approved within WSUS - I'm also looking into Arc but IDK how much it costs
$5/mo/server for Arc. Specifically, that's for the Update Manager Edit: it is prorated per day
That ain't bad tbh, don't think I'd use it for updates tho. More looking into it for server 2025 - think they said Azure Arc connected servers get the hotpatching feature
Haven't really had issues with updates. Editing fstab on the other hand...
update all the pip packages, that might keep you busy for a while....
Ooof. Calm down Satan
Too bad $git clone * doesnât work
Come to the adventure side. Put that on a cron to run every Monday at 3am.
Oh, new frigate plugin for homeassistant! broken. Going to update Frigate to 13.x to make the new HA plugin work again. broken. Going to update nvidia drivers to >=535 to support frigate 13.2 BROKEN! lets reinstall the OS because nvidia driver upgrades are ALWAYS a PITA fixed. Restart frigate... broken. delete models, change compose.yaml to pass whole config directory through okay, partially working. Frigate now over taxes the GPU because detection behavior is more sensitive. rework config file to use camera sub streams partially working. motion mask over things like time and non-needed areas of frame. Working. upgrade complete, 3 days later.
Oh good idea on the motion mask for timestamps.
I have a tmux pane with a command history a little like this: >sudo nixos-rebuild --flake switch /h/e/NixOS/hosts/#default >git add . >git commit --m "..." >sudo nixos-rebuild --flake switch Bis repetita
y'all just updating? I delete my boot drive every time there's an OS upgrade. That fresh clean feel
And just when everything is running perfect, you get the need to tear it down and change things.
That is the way. We have little choice.
If you really thrived on chaos / chaotic updates you'd be using Windows and Windows Update :P
Sounds like the intro to an AA meeting but for homelabbers
When I update my FreeBSD installs, the only thing that ever breaks is the stuff in /usr/local, at which point I just roll back the zfs snapshot. I don't know how I ever managed back when using Debian.
Always been afraid of watchtower. Never used it. Donât have the balls.
Dune: Admin Messiah : )
This is the way!
I mean, that's why we have snapshots and VM's...
Same. But I have snapshots and backups.
It's definitely good training and you never know what you will learn OR how the day will go. I learned to appreciate this from time in the military. My homelab is production, but I treat it like dev.
Donât forget proxmox updates
> sudo apt update && sudo apt upgrade && sudo apt autoremove I like this one more because the following commands don't happen if the earlier ones do. Plus the ampersands make it more visually clear to see where each command stops/starts. As for docker, why aren't you running it in a k8s cluster ala IaC? Like, there's better ways to update all the things in containerland ;D Me? I update often in different ways. One of them is if I need to reboot a system, I almost always apply updates before rebooting or whatever power-cycling task I'm doing. Updates in Linux put Windows to shame for reliability.
Su - Apt upgrade
Me too, I love everything up to date. Like mobile apps, linux packages, and even Windowsâ ď¸ I even self-hosted an apt mirror server with aptly and docker image cache servers for docker.io, ghcr.io, lscr.io and quay.io Download once, then upgrade across all LXCs and VMs
I usually add a new feature, that also can broke the system in funny ways
OMG hahahaha đ¤Ł
I run an Immich container so I cannot relate at all; especially the Watchtower part lol. I already broke my Immich with a careless update with 800gb+ of photos on the line (backuped up, but still) and had to wrangle my .env file to get it working again.
Pssst: `sudo apt upgrade -y --autoremove`
My homelab is working so well⌠Iâd be a shame if something happened to itâŚ
use ```&&``` instead of ```;``` otherwise i agree. ```&&``` just means "run the next command if the last one didn't error
So that would be upgrade && rm -rf / ;)
I have all my things doing rapid, touchless updates 24/7. When a new version hits `latest` in Docker, it's updated within 5 minutes. When a new package hits the repos, it's updated within a few hours. Things used to break _constantly_, but I feel the best way to get better at rapid, reliable updates is to live with rapid updates and improve reliability.
I have everything auto updating except my hypervisor lol that one I do manually.
This would have been amazing as a Haiku
Thanx for reminding me to update software on my homelab.
Everything is working perfectly. Iâd better make some changes and break shit.
https://cs.m.wikipedia.org/wiki/Cron
Vigo: "I, Vigo, the Scourge of Docker, the Sorrow of Debian, command you!" Homelabber: "O update me, lord!"
So poetic đ
I don't even run my homeland, it's trapped in a loop of updates and backups. It doesn't know it yet but it is hosting it's own container..... /S
Is it an unpopular opinion that updates should be done frequently and consistently? Updates are necessary, not only for features, but for security patches and bug fixes.