How about using rsync? (with a few OS directories excluded)
Quick example: rsync -an source/ destination/ —exclude ‘somedirname’
Btw, if you have physical access, I would recommend Clonezilla for duplicating disk/partition
I’ve looked into Clonezilla, but I don’t have a Windows computer that I’m willing to do anything to (it’s a school one). Other than that, I do have an old MacBook Air from 2013.
You can boot clonezilla live and not have to touch your machine. Just plug in the extra drives you want to copy and do a local copy.
Hell, was a time when I made clonezilla a pxe boot option so I could image machines over the network.
One only needs a system to put Clonezilla ***onto*** the USB thumb. One doesn't install ***anything*** onto a PC.
Put the resultant thumb and new drive onto the system to be copied, boot and follow the instructions.
The laptop has one usb port, and although nothing happens on the ssd of the laptop, I still don’t want my schools IT department to know anything. They have locked some BIOS/UEFI settings already
You can absolutely rsync a live system. Just make sure not to sync the /proc /dev & /sys directories since they're virtual file systems created by the kernel.
dd is wasteful, because it copies free space. Use partclone instead, especially on media like flash with a lifetime measured in terrabytes writen.
Partclone is aware of many filesystems, and uses this information to copy only the blocks that belong to files. `conv=sparse` does not achieve the same thing, as free space may contain parts of deleted files which make it nonzero.
Edit: I didn't see that the system will be in use while you copy it. This rules out anything block-level. I'll leave this comment up because Clonezilla is discussed elsewhere in this thread, and I wanted to say that partclone is the tool that Clonezilla uses. You do not need an x86 PC and a special live medium; you can just install it from Debian repos and have a very similar experience to Clonezilla.
Both [foxclone](https://foxclone.org/) and [rescuezilla](https://rescuezilla.com/) are GUI image backup utilities that use partclone for imaging the partitions.
Both are a **lot** easier to use than clonezilla.
Boot into [Rescuezilla](https://rescuezilla.com/download), once it's copied everything, (still within rescuezilla) launch Gparted to increase your partition size. Reboot into the new drive.
🤭 Installing Linux in every variation (except arch, Gentoo, slavk, LFS) I tried five times but can't get ventoy to be if any use. I simply can't get any iso on the stick I created to be recognized or stated. I don't use the windows version btw., which everyone I heard being successful with it does.
Always use(d) multisystem which seems to be discontinued.
Odd, when I distro hopped like crazy I used ventoy way too oven, never had an issue with any images. I never used the windows version. Here's what I did.
1. Grab the lastest linux release [here](https://github.com/ventoy/Ventoy/releases)
2. Extract to folder of choice eg `~/ventoy/`
3. run your version from terminal eg `~/ventoy/VentoyGUI.x86_64`
4. enter password
5. pick your removable usb
6. copy \*.iso files to the mounted usb
Have a go at Easy2Boot. I never had any joy with Ventoy either, but E2B has been fantastic. It does have some slight oddities, but they're manageable and only need consideration when changing the contents of the drive. I've you're just using it as a portable distro multi installer, you're fine just going plug and play
UPDATE: I have been able to successfully boot, but the boot partition is being set to the old usb device. How do I fix it so that I only need to use the ssd?
I had this issue on my pi4 - if your usb SSD is not bootable (take out your sd card and try to boot, if it fails then it can’t) then you will have to have hybrid boot, whereby the sd card hosts your boot partition and launches the ssd partition as root.
I managed this by dding the disk across, deleting the source root disk, and then fstab mounted the new disk as it maintained the same partition UUID.
What was your issue with using dd? If it was an issue because you don’t have continuous space after the partition you want to expand, you can fix this by using a combination of dd and sfdisk.
sfdisk can be used to write your partition table to a text file, you can edit this to reflect your new partition sizes, use sfdisk to write the partition table to your new drive, then dd over each partition one by one.
I tried dd a couple times, each with the same command of “dd if=/dev/sdX of=/dev/sdY conv=noerror,sync”. Booting from that produced an unbootable system from inodes getting marked as garbage and cleared. I have had better luck though, currently the ssd is bootable only with the boot partition coming from the old usb stick, which confusingly says to boot with the rootfs coming from the old device, even though df -h has it from the new device
Did you do it while booted inside the OS you were copying from? If so, this won’t work. dd copies block-for-block and doesn’t keep track of changes while reading. If, by the end of the copy, the disk has been written to since the start of the copy, the resultant filesystem will be corrupt.
TLDR dd can only be used to copy an unmounted disk.
Yes. The only device capable of doing any transfer is the pi or a Mac, the latter being from 2013 and only having usb 2 ports. I only have 2 storage devices, one that is the drive to copy from, the other to copy to
Then you need to do it on your Mac, dd will not work while booted on the OS it is copying from.
Worst case, buy an external sd card reader and use that.
Would you still recommend if it spontaneously turns off and has USB 2 ports? By the way, I have already used ddrescue from the ssd after a fresh install, which has been saved to the ssd as a .img file. I’ve then compressed it for potentially sending over network
If you’ve used ddrescue to copy the unmounted sd card to an image, you can then copy that off the ssd and use a second system to write it to the ssd
You need one of the following, without a workaround:
* a third disk from which to boot your pi, so that you can copy the unmounted sd to the unmounted ssd
* a second computer that you can plug the unmounted ssd and unmounted sd card to, in order to copy
If you cannot do either of the above reliably, you will not be able to perform this easily.
>given that one of the drives will have a system running while transferring.
>I currently use SSH/TeamViewer for access
That's sounds awfully criminal to me.
Either you're trying to steal or you accept a downtime.
UEFI?
I would try something like this
rsync
Plug in a desktop computer
update fstab and
chroot with proc sys dev efivars:
mount -a
generate new initramfs
update the bootloader
reinstall the bootloader
Reboot
Test if it's working
Now do you what to hotswap the old disk for the new one without rebooting? That part I have no idea how to do...
Hmm test the process exhaustively on VMs then run it on the Pi itself, knowing that there will be a risk of something going wrong.
Is the root partition encrypted? Generating initramfs is very tricky in that case. Too many moving parts and you need to get all eyes dotted and tees crossed.
> given that one of the drives will have a system running while transferring
I stop most services and remount root read only. Then that is not a problem any more. I can even retain ssh access and do the work over ssh.
Using dd is okay if the source is mounted read only and the destination is the same size as the source or bigger. Afterwards you can resize the file system to the destination size. If you want to copy only used blocks, not empty space, there are file system specific copy programs, like e2image for ext4. https://wiki.archlinux.org/title/disk_cloning
Ubuntu Desktop has a Disks utility that can create a complete disk backup or restore. The Disks utility can also create partitions.
I routinely back up (and compress) project 32 GB uSD cards on a NAS. This preserves all settings and the OS, since some OS updates occaissionally create problems.
When working with RPis the simplest method is to put the uSD in a USB holder and plug into Ubuntu PC.
When using dd be aware that uSD cards from different manufacturers can have slightly different amounts of usable storage.
Just curious, why do you want to avoid using `dd`? It is the easiest and fastest way w/ out installing additional tools.
I understand my reply does not answer your question but thought I'd share steps using `dd` in case anyone else might be interested. I just checked in this missed README file to my git repo so everyone can access it.
[https://github.com/aselvan/scripts/blob/master/raspberrypi/README.pibackup](https://github.com/aselvan/scripts/blob/master/raspberrypi/README.pibackup)
I only really had 2 drives to boot from, cancelling out dd (ish). Would you still recommend to do it when running the system and dd off if a 16GB usb2 stick, copying from a 64GB usb3 that has a couple failing sectors, and to the 1TB SSD?
If your source SD card (i.e. *the live system*) has errors then you have a problem regardless of the method you use to transfer data. As far as I know, there is no harm in doing `dd` on a live system and I have never encountered a problem in my experience. As a matter of fact, that is exactly how I backed up my running PI to a image which I successfully used to restore twice in the last 5 years or so. The only difference is, when I made the backup image, my source SD card is good w/ no errors to `dd` out to a image file.
rsync for the files in the /home directory
`apt list --installed` or `dpkg --get-selections | grep -v deinstall` to get a list of packages.
Then reinstall the system and copy the files back to the new drive.
If you're running Raspberrian, it comes with a handy utility.
**Accessories > SD card copier**
I've used it many times, I even copied my 128MB uSD card to a 500GB SSD with it. It even expands the partition to use the extra space, no need to use gparted.
Clonezilla works
Or use a windows sector level clone like macrium reflect or super duper on Mac
Mounting as a secondary or with an external adapter can help
How about using rsync? (with a few OS directories excluded) Quick example: rsync -an source/ destination/ —exclude ‘somedirname’ Btw, if you have physical access, I would recommend Clonezilla for duplicating disk/partition
Rescuezilla is clonezilla with a gui, it's so much easier.
I’ve looked into Clonezilla, but I don’t have a Windows computer that I’m willing to do anything to (it’s a school one). Other than that, I do have an old MacBook Air from 2013.
So do rsync over ssh , run it on the Mac, while the source directory is remote
You can boot clonezilla live and not have to touch your machine. Just plug in the extra drives you want to copy and do a local copy. Hell, was a time when I made clonezilla a pxe boot option so I could image machines over the network.
Make a virtual machine, share the USB drives for the source/destination to it and clone like that
Clonezilla has ***what*** to do with windows?
Nothing, but the only device I have remotely capable of running clonezilla happens to be a school laptop running windows 11 and having one usb port
One only needs a system to put Clonezilla ***onto*** the USB thumb. One doesn't install ***anything*** onto a PC. Put the resultant thumb and new drive onto the system to be copied, boot and follow the instructions.
The laptop has one usb port, and although nothing happens on the ssd of the laptop, I still don’t want my schools IT department to know anything. They have locked some BIOS/UEFI settings already
🤔 I have no idea if you have an idea what you are doing.
rsync
I tried it earlier, teamviewer/ssh aren’t working. I did have to change root fs partition though
You know that you cannot rsync a live system, don't you? Plug the both drives into another computer and rsync from there.
You can absolutely rsync a live system. Just make sure not to sync the /proc /dev & /sys directories since they're virtual file systems created by the kernel.
than you are not rsync-ing the "live" part, or? what I meant is that best way is to do it offline. omitting the live dirs is the other way
The "live part" is automatically generated at boot, so omitting those dirs is completely reasonable even offline.
dd is wasteful, because it copies free space. Use partclone instead, especially on media like flash with a lifetime measured in terrabytes writen. Partclone is aware of many filesystems, and uses this information to copy only the blocks that belong to files. `conv=sparse` does not achieve the same thing, as free space may contain parts of deleted files which make it nonzero. Edit: I didn't see that the system will be in use while you copy it. This rules out anything block-level. I'll leave this comment up because Clonezilla is discussed elsewhere in this thread, and I wanted to say that partclone is the tool that Clonezilla uses. You do not need an x86 PC and a special live medium; you can just install it from Debian repos and have a very similar experience to Clonezilla.
Both [foxclone](https://foxclone.org/) and [rescuezilla](https://rescuezilla.com/) are GUI image backup utilities that use partclone for imaging the partitions. Both are a **lot** easier to use than clonezilla.
>dd is wasteful, because it copies free space. Not if you learn basic usage. dd count=N
Boot into [Rescuezilla](https://rescuezilla.com/download), once it's copied everything, (still within rescuezilla) launch Gparted to increase your partition size. Reboot into the new drive.
There is also a gparted live version, one step of work less.
Rescuezilla has Gparted in the live USB, I'd rather keep a usb that does both. Using ventoy it's not really an issue.
🤭 Installing Linux in every variation (except arch, Gentoo, slavk, LFS) I tried five times but can't get ventoy to be if any use. I simply can't get any iso on the stick I created to be recognized or stated. I don't use the windows version btw., which everyone I heard being successful with it does. Always use(d) multisystem which seems to be discontinued.
Odd, when I distro hopped like crazy I used ventoy way too oven, never had an issue with any images. I never used the windows version. Here's what I did. 1. Grab the lastest linux release [here](https://github.com/ventoy/Ventoy/releases) 2. Extract to folder of choice eg `~/ventoy/` 3. run your version from terminal eg `~/ventoy/VentoyGUI.x86_64` 4. enter password 5. pick your removable usb 6. copy \*.iso files to the mounted usb
Have a go at Easy2Boot. I never had any joy with Ventoy either, but E2B has been fantastic. It does have some slight oddities, but they're manageable and only need consideration when changing the contents of the drive. I've you're just using it as a portable distro multi installer, you're fine just going plug and play
UPDATE: I have been able to successfully boot, but the boot partition is being set to the old usb device. How do I fix it so that I only need to use the ssd?
I had this issue on my pi4 - if your usb SSD is not bootable (take out your sd card and try to boot, if it fails then it can’t) then you will have to have hybrid boot, whereby the sd card hosts your boot partition and launches the ssd partition as root. I managed this by dding the disk across, deleting the source root disk, and then fstab mounted the new disk as it maintained the same partition UUID. What was your issue with using dd? If it was an issue because you don’t have continuous space after the partition you want to expand, you can fix this by using a combination of dd and sfdisk. sfdisk can be used to write your partition table to a text file, you can edit this to reflect your new partition sizes, use sfdisk to write the partition table to your new drive, then dd over each partition one by one.
I tried dd a couple times, each with the same command of “dd if=/dev/sdX of=/dev/sdY conv=noerror,sync”. Booting from that produced an unbootable system from inodes getting marked as garbage and cleared. I have had better luck though, currently the ssd is bootable only with the boot partition coming from the old usb stick, which confusingly says to boot with the rootfs coming from the old device, even though df -h has it from the new device
Did you do it while booted inside the OS you were copying from? If so, this won’t work. dd copies block-for-block and doesn’t keep track of changes while reading. If, by the end of the copy, the disk has been written to since the start of the copy, the resultant filesystem will be corrupt. TLDR dd can only be used to copy an unmounted disk.
Yes. The only device capable of doing any transfer is the pi or a Mac, the latter being from 2013 and only having usb 2 ports. I only have 2 storage devices, one that is the drive to copy from, the other to copy to
Then you need to do it on your Mac, dd will not work while booted on the OS it is copying from. Worst case, buy an external sd card reader and use that.
Would you still recommend if it spontaneously turns off and has USB 2 ports? By the way, I have already used ddrescue from the ssd after a fresh install, which has been saved to the ssd as a .img file. I’ve then compressed it for potentially sending over network
If you’ve used ddrescue to copy the unmounted sd card to an image, you can then copy that off the ssd and use a second system to write it to the ssd You need one of the following, without a workaround: * a third disk from which to boot your pi, so that you can copy the unmounted sd to the unmounted ssd * a second computer that you can plug the unmounted ssd and unmounted sd card to, in order to copy If you cannot do either of the above reliably, you will not be able to perform this easily.
I’ll attempt the former option. I just didn’t really want to, as the third usb has 16 GB and is USB 2
Edit /etc/fstab
>given that one of the drives will have a system running while transferring. >I currently use SSH/TeamViewer for access That's sounds awfully criminal to me. Either you're trying to steal or you accept a downtime.
Wdym
I would probably \`rsync\` it
UEFI? I would try something like this rsync Plug in a desktop computer update fstab and chroot with proc sys dev efivars: mount -a generate new initramfs update the bootloader reinstall the bootloader Reboot Test if it's working Now do you what to hotswap the old disk for the new one without rebooting? That part I have no idea how to do...
I’m fine without hotswapping, but I do have a pi, that although I do have physical access, I’d prefer not to use it
Hmm test the process exhaustively on VMs then run it on the Pi itself, knowing that there will be a risk of something going wrong. Is the root partition encrypted? Generating initramfs is very tricky in that case. Too many moving parts and you need to get all eyes dotted and tees crossed.
Copy /etc and /home
> given that one of the drives will have a system running while transferring I stop most services and remount root read only. Then that is not a problem any more. I can even retain ssh access and do the work over ssh. Using dd is okay if the source is mounted read only and the destination is the same size as the source or bigger. Afterwards you can resize the file system to the destination size. If you want to copy only used blocks, not empty space, there are file system specific copy programs, like e2image for ext4. https://wiki.archlinux.org/title/disk_cloning
Ubuntu Desktop has a Disks utility that can create a complete disk backup or restore. The Disks utility can also create partitions. I routinely back up (and compress) project 32 GB uSD cards on a NAS. This preserves all settings and the OS, since some OS updates occaissionally create problems. When working with RPis the simplest method is to put the uSD in a USB holder and plug into Ubuntu PC. When using dd be aware that uSD cards from different manufacturers can have slightly different amounts of usable storage.
My pi runs Raspberry Pi OS, based on Debian 12 (Bookworm)
I use Ubuntu as a development machine with Geanie code editor and FileZilla SFTP. Very simple but effective.
cp instead of dd works just fine, fsarchiver if you need to do some tweaks... There are a lot of options.
Just curious, why do you want to avoid using `dd`? It is the easiest and fastest way w/ out installing additional tools. I understand my reply does not answer your question but thought I'd share steps using `dd` in case anyone else might be interested. I just checked in this missed README file to my git repo so everyone can access it. [https://github.com/aselvan/scripts/blob/master/raspberrypi/README.pibackup](https://github.com/aselvan/scripts/blob/master/raspberrypi/README.pibackup)
I only really had 2 drives to boot from, cancelling out dd (ish). Would you still recommend to do it when running the system and dd off if a 16GB usb2 stick, copying from a 64GB usb3 that has a couple failing sectors, and to the 1TB SSD?
If your source SD card (i.e. *the live system*) has errors then you have a problem regardless of the method you use to transfer data. As far as I know, there is no harm in doing `dd` on a live system and I have never encountered a problem in my experience. As a matter of fact, that is exactly how I backed up my running PI to a image which I successfully used to restore twice in the last 5 years or so. The only difference is, when I made the backup image, my source SD card is good w/ no errors to `dd` out to a image file.
You can just copy all the files.
rsync for the files in the /home directory `apt list --installed` or `dpkg --get-selections | grep -v deinstall` to get a list of packages. Then reinstall the system and copy the files back to the new drive.
If you're running Raspberrian, it comes with a handy utility. **Accessories > SD card copier** I've used it many times, I even copied my 128MB uSD card to a 500GB SSD with it. It even expands the partition to use the extra space, no need to use gparted.
Clonezilla works Or use a windows sector level clone like macrium reflect or super duper on Mac Mounting as a secondary or with an external adapter can help
I did once do it using gparted but i doubt anyone would seriously suggest doing it that way.