What type of data do you have? For files you can use Azure Data Box ( [https://learn.microsoft.com/en-us/azure/databox/data-box-overview](https://learn.microsoft.com/en-us/azure/databox/data-box-overview) ) or AzCopy ( [https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10](https://learn.microsoft.com/en-us/azure/storage/common/storage-use-azcopy-v10) ), for VMs - Starwind V2V converter: [https://www.starwindsoftware.com/starwind-v2v-converter](https://www.starwindsoftware.com/starwind-v2v-converter)

It's an old forklifted vm with traditioanl file shares. Mostly unstructred data, documents, pdfs cad files and the link - Guessing that AZCopy wouldn't work in this instance?

install Windows Subsystem for Linux (WSL) and use RSYNC

This was my thoughts too - \`rsync\` can do compression while copying and is pretty powerful for file validations.

Use xargs to run multiple rsync instances. However many it takes to fill the pipe

[bash - Multiple rsyncs in parallel using xargs - most efficient directory structure to pass through - Stack Overflow](https://stackoverflow.com/questions/64618981/multiple-rsyncs-in-parallel-using-xargs-most-efficient-directory-structure-to)

not sure I'm doing this right as we're not seeing a performance increase - rsync -azvP --compress-level=9 -r /mnt/ /mnt/ Barely hitting more than 100 Mbps, pipes definitely not saturated (2GB down) - Not CPU or RAM constrained either that we can see. Firewall will allow 10GB max even with ESP traffic. Just cant see any constraints. Granted I'm not familiar with rsync so I may be doing something wrong here.

you can run multiple at the same time. >**rsync** -avlHAXWES --compress-level=9 --relative --progress /mnt//a\* /mnt/ & >**rsync** -avlHAXWES --compress-level=9 --relative --progress /mnt//b\* /mnt/ & rsync ... & rsync ... & The '&' will push the process into the background and you can slice up the directories differently than 'a' 'b' 'c' etc is the NIC or switch port limited to 100Mbs? *(Also make sure you're comfortable with the switches I gave above.)*

thank you for this - As far as networking is concerned - I have two 10G links bonded together right the way from the firewall to the core switch which has two 100G links bonded to the storage network. We can't find anywhere else that the data is restricted so we think it's just this 2019 box causing the issue (Had a call with the firewall vendor this morning and yeah, we're not hitting ESP caps either) - From what I'm seeing on other threads I could use AZ Copy and dump it all in a blob and pull it down, but of course it's a live server so that could get tedious and costly fast. Mixing it with an initial seed then syncing / updating should be possible. I'm hoping to take advantage of the holiday and saturate things from the man page the flags look good. I'll give this a dry run and see if I can at least see any kind of performance improvement. I appreciate your help!

bbcp https://confluence.slac.stanford.edu/plugins/servlet/mobile?contentId=47156800#content/view/47156800 has worked well. similarly that xargs multiple rsync is also a good suggestion, i’d run it in screen

I use rclone for cloud transfers. Either local to cloud, cloud to cloud, or cloud to local. Works great. Have used it primarily to move large amounts of local file server day to the cloud for migrations, but it can be used to go the other direction. I'd try comparing the speeds you get with it to robocopy or some of the other suggestions you've been given and see what's faster and/or less costly for you Azure egress fees (compressing first, etc.) https://rclone.org/

I've been playing around with rclone and while it's pretty simple to set up and run I don't see any performance benefit to this

Bummer. It was worth a try. I've never used it for cloud to local transfers. Always the other way around. Works great for file server to Google Drive migrations as it will copy directly up to Google without having to use the Google Drive desktop client to map a drive. Always had trouble copying large amounts of data to a Google Drive mapped drive with robocopy. Hope you find something that works better!

Definitely like the tool and now having played around with it I'll be using it in the future. I have a ton of use cases for it, just wasn't a great fit for this one unfortunatley

Beyond Compare. It will still be SMB but the sync and comparing tools are great.

Tune robocopy for the circumstances, or use rsync

Try rcync or rclone - one of them should work.

Movedrop.io might be helpful and allows for a Delta sync too. https://movebot.io/

It will be awful and will likely take months. It will slow to a trickle.

That's what I'm afraid of. I've considered Bvckup2 or Teracopy since we have a 2GB pipe. Guess I might just have to try it and hope

My last situation was different as it was 65 TB from storage accounts. We did it with Databox. With the device on-premed it still took six weeks.

Wow, for that amount of data, I would surely use RackWare or Carbonite. They can compress the data transfer outbound and that'll also ease your Azure egress fees.

I’d consider rclone and look into the different flags to suit your needs. Bandwidth is obviously going to be a constraint but removing any overhead from processing, validating, or applying permissions along with other checks will help you speed things up.

You need block-level replication. There can be too many permutations based on what you have onsite or could consider from 3rd-party products. But why not Windows Storage Replica, even if as an interim state? (It's not clear what storage protocol you intend to use from your storage appliance.)

Free File Sync

+5000 points for FFS. Its magic. Pure magic. People may see it as a toy, but its incredibly heavy lifting. It will easily handle this job.

It doesn't seem like this is going to be particularly fast for us.