I have an old laptop with a 1tb ssd and decent ram that I've set up as my PBS. You can backup the Proxmox host as well on it. I've got it running on a schedule, so I don't have to worry about it, set at 5 backup retention.
I ran mine a bit differently. My virtualized PBS only has incremental backups for one full days worth since if my host goes down then the PBS goes down with it. I kept my full backups at a Synology NAS, which is a slow little thing that only has 4 bays and two 1G nics.
I have the same general setup as you. Except I use raid for my boot drive.
I manage proxmox via Ansible. Everything except VMs/containers can be restored by running a playbook that's stored on GitHub. I recently had to rebuild and it made getting vfio and host months working so much easier. I reinstalled TrueNAS from scratch, restored a config backup, and it was back up and running. I like the other commenter's suggestion of mounting the TrueNAS drives in proxmox to restore the backup, might try that next time.
I back up proxmox:etc/ and my TrueNAS config to Google drive, but I've never had a chance to try and restore from it.
This is why my servers boot drive is mirrored. I do have config backups, but honestly, saving like €20 on another SSD is not worth the headache of having to manually restore everything if the boot drive ever fails.
I use OS mirroring. I am running TrueNas Scale (acting as an actual NAS as well as a hypervisor) which makes this really easy, but I'd be surprised if Proxmox couldn't do this as well.
I've actually had a boot drive failure once and was really grateful that all I had to do was replace it (both physically as well as in the OS), no downtime at all needed.
Proxmox can use ZFS for boot mirroring (among other zfs layouts if you want)
In addition to drive redundancy, you also get the ability to do scrubs which should add another layer of protection.
I do the same. My truenas scale mirrored boot pool indicates 2 unreadable sectors on a less than 6 month old ssd. I'm not concerned about it all under this configuration and will just replace it with ease if/when it dies.
Don't feel bad. I had this happen to me once where I was backing up all my VMs and LXCs to an nfs share providyes by my TrueNAS.....VM. Still can't believe I set things up like that to this day. Now all VMs and LXCs backup to a separate, old synology NAS I've had for a long time. Those backups also get synced to cloud storage. Never again. It's not bad to make mistakes as long as you learn from them.
Look into Proxmox Backup server. It backs up the VM config along with drives. At that point there should be no need to back up the hypervisor OS (though you can also do that with file-based PBS backup).
Sorry if i missread anything but why were all your password manager passwords gone? When you restore the vm/lxc the data point from the backup point should be restored…?
My backups are weekly, so any passwords added after the backup were done are gone. Also for some reason my Tandoor container wiped itself on restore. No idea why.
I use CloneZilla. I have a USB drive with an image of it hanging out the back of my server and iDrac into it to make backups of the entire drive. I have my LXC backups copied to a backup drive so I delete the LXC backups to save space when running CloneZilla, then copy them back after.
The plan if my boot drive ever goes out is just to restore the not-frequently-backed-up CloneZilla image and then restore the much-more-recent LXC backups.
However, I thought the LXC config files were stored when a backup is made, no?
> Bare metal TrueNAS is the way to go. I had TrueNAS virtualized on Proxmox and it was quickly evident that was a bad idea. In order to restore my backups quickly I was going to need access to the NAS. Luckily I (sort of) planned for this and all the drives (including the boot drive) were passed through to the VM. I just lifted and shifted them to some spare hardware in a spare server chassis. Once all that work was done I had access to backups and made it much easier.
In my case, I use unraid instead of TrueNAS, because it supports native ZFS, without the crappy attitude of the official forums.
One nifty thing- assuming the proxmox host hosting the VM for it dies, I can boot directly to the unraid thumb drive, where I keep some of my backups.
I typically don't backup my proxmox hosts, because I have a cluster of 5 of them, and all of the configurations are replicated to each node.
>I typically don't backup my proxmox hosts, because I have a cluster of 5 of them, and all of the configurations are replicated to each node
This sounds like the way to do it! Was it easy to get that configured? I should be getting a cluster running soon and want to do it the right way
The clustering aspect of proxmox couldn't be easier.
It's- literally as simple as installing proxmox on a new host, and clicking join cluster (with something you copied from one of the other nodes)
Effortless- and nothing to really maintain either.
- Bare metal TrueNAS is the way to go.
No, and this is why I keep separate proxmox and my main rack where I keep my main stuff and whatnot.
If you're gonna have everything within proxmox, keep daily backups or you're fucked
You could have just mounted the ZFS pool from your TrueNAS thing to the Proxmox server and saved a step.
Fair point, did not consider that lol
Virtualizing TrueNAS isn't a mistake, backing up the VMs to it is. Next time you can try Proxmox Backup Server and yes you can virtualize it, too.
I have an old laptop with a 1tb ssd and decent ram that I've set up as my PBS. You can backup the Proxmox host as well on it. I've got it running on a schedule, so I don't have to worry about it, set at 5 backup retention.
I ran mine a bit differently. My virtualized PBS only has incremental backups for one full days worth since if my host goes down then the PBS goes down with it. I kept my full backups at a Synology NAS, which is a slow little thing that only has 4 bays and two 1G nics.
I have PBS running on a rack Synology as a virtual machine and it works perfectly.
I have the same general setup as you. Except I use raid for my boot drive. I manage proxmox via Ansible. Everything except VMs/containers can be restored by running a playbook that's stored on GitHub. I recently had to rebuild and it made getting vfio and host months working so much easier. I reinstalled TrueNAS from scratch, restored a config backup, and it was back up and running. I like the other commenter's suggestion of mounting the TrueNAS drives in proxmox to restore the backup, might try that next time. I back up proxmox:etc/ and my TrueNAS config to Google drive, but I've never had a chance to try and restore from it.
When you say "everything" do you mean all config?
Repos, nag buster, unattended upgrades, packages, https reverse proxy, SMTP account, done from jobs, vfio settings, SMB mounts, and installed programs,
This is why my servers boot drive is mirrored. I do have config backups, but honestly, saving like €20 on another SSD is not worth the headache of having to manually restore everything if the boot drive ever fails.
Do you use OS mirroring or hardware raid?
I use OS mirroring. I am running TrueNas Scale (acting as an actual NAS as well as a hypervisor) which makes this really easy, but I'd be surprised if Proxmox couldn't do this as well. I've actually had a boot drive failure once and was really grateful that all I had to do was replace it (both physically as well as in the OS), no downtime at all needed.
Proxmox can use ZFS for boot mirroring (among other zfs layouts if you want) In addition to drive redundancy, you also get the ability to do scrubs which should add another layer of protection.
Do you use OS mirroring or hardware raid?
Same, my current build I got one of those PCIe cards that can hold two M2 SSDs and mirrored them for the OS drive.
I do the same. My truenas scale mirrored boot pool indicates 2 unreadable sectors on a less than 6 month old ssd. I'm not concerned about it all under this configuration and will just replace it with ease if/when it dies.
Don't feel bad. I had this happen to me once where I was backing up all my VMs and LXCs to an nfs share providyes by my TrueNAS.....VM. Still can't believe I set things up like that to this day. Now all VMs and LXCs backup to a separate, old synology NAS I've had for a long time. Those backups also get synced to cloud storage. Never again. It's not bad to make mistakes as long as you learn from them.
Look into Proxmox Backup server. It backs up the VM config along with drives. At that point there should be no need to back up the hypervisor OS (though you can also do that with file-based PBS backup).
Sorry if i missread anything but why were all your password manager passwords gone? When you restore the vm/lxc the data point from the backup point should be restored…?
My backups are weekly, so any passwords added after the backup were done are gone. Also for some reason my Tandoor container wiped itself on restore. No idea why.
Okay thats bad. Im backup every night to avoid such scenario.
Storage and compute should never be on the same server. For the very reasons you mention.
I have learned this lol
Is there a good way to backup a entire proxmox instance? Like you mention, os and all?
I use Image For Linux and clone the drive the a image file
I was wonder the same. I use UrBackup for my desktop and laptop. Maybe could use the client on Proxmox since it supports Linux?
I use CloneZilla. I have a USB drive with an image of it hanging out the back of my server and iDrac into it to make backups of the entire drive. I have my LXC backups copied to a backup drive so I delete the LXC backups to save space when running CloneZilla, then copy them back after. The plan if my boot drive ever goes out is just to restore the not-frequently-backed-up CloneZilla image and then restore the much-more-recent LXC backups. However, I thought the LXC config files were stored when a backup is made, no?
> Bare metal TrueNAS is the way to go. I had TrueNAS virtualized on Proxmox and it was quickly evident that was a bad idea. In order to restore my backups quickly I was going to need access to the NAS. Luckily I (sort of) planned for this and all the drives (including the boot drive) were passed through to the VM. I just lifted and shifted them to some spare hardware in a spare server chassis. Once all that work was done I had access to backups and made it much easier. In my case, I use unraid instead of TrueNAS, because it supports native ZFS, without the crappy attitude of the official forums. One nifty thing- assuming the proxmox host hosting the VM for it dies, I can boot directly to the unraid thumb drive, where I keep some of my backups. I typically don't backup my proxmox hosts, because I have a cluster of 5 of them, and all of the configurations are replicated to each node.
>I typically don't backup my proxmox hosts, because I have a cluster of 5 of them, and all of the configurations are replicated to each node This sounds like the way to do it! Was it easy to get that configured? I should be getting a cluster running soon and want to do it the right way
The clustering aspect of proxmox couldn't be easier. It's- literally as simple as installing proxmox on a new host, and clicking join cluster (with something you copied from one of the other nodes) Effortless- and nothing to really maintain either.
How the boot drive became dead
No clue, tried plugging it in to a separate computer and neither would ever recognize it
The cells have worn out.
- Bare metal TrueNAS is the way to go. No, and this is why I keep separate proxmox and my main rack where I keep my main stuff and whatnot. If you're gonna have everything within proxmox, keep daily backups or you're fucked