*FASTEST* copy of a few million small files around 5.5TB

njcwotx · Guru Joined: 25 Feb 2005 Posts: 587 Location: Texas

I did a little research and the consensus is all over the board on this.

I have to transfer file from an SMB share mounted on a windows server using ntfs to a linux host on ext4. Its millions of small files and about 5.5TB total. Even on a 10Gig network, its copying about 20G/hr. I estimate over 10days to copy.

to start, we mounted the share on the linux host and I think dev is using rsync to move the files.

Doing a little research and I see some suggestions of using tar via ssh ( I have done this in past but dont remember how fast it was). Other suggest using netcat or scp.

10+ days is a long time to run a copy. I wonder if anyone has crossed this bridge before

_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.

chithanh · Posted: Fri Sep 30, 2016 11:17 pm Post subject:

I think SMB is the bottleneck here.
Can't you run rsync on the Windows server directly?

scp is definitely slower than rsync if you have small files.

njcwotx · Guru Joined: 25 Feb 2005 Posts: 587 Location: Texas

Dev did a mount -t cifs on Linux. Then a local rsync. So u think an rsync from Windows direct is faster?
_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.

Hu · Moderator Joined: 06 Mar 2007 Posts: 21635

mount -t cifs means you are using SMB/CIFS to transport the files from the Windows machine to the Linux kernel, which then exposes them to rsync as a regular filesystem. If you run an rsync daemon on the Windows machine, you can use the rsync protocol, which may be more efficient.

chithanh · Posted: Sat Oct 01, 2016 8:52 am Post subject:

You could also run the rsync daemon on the Linux host and an rsync client on the Windows machine, which might be easier to set up.

toralf · Posted: Sat Oct 01, 2016 11:15 am Post subject:

The fastest method IMO is something like

frostschutz · Advocate Joined: 22 Feb 2005 Posts: 2977 Location: Germany

sometimes its faster to just carry the hard disks over :lol:

sending the raw filesystem image over could be faster than tar. it depends.

seriously though, rsync is not the worst solution by far. it might lack speed but if there is any problem during the transfer, rsync resumes easily where the others do not.

Naib · Posted: Sat Oct 01, 2016 11:32 am Post subject:

szatox · Advocate Joined: 27 Aug 2013 Posts: 3137

toralf · Posted: Sun Oct 02, 2016 9:07 pm Post subject:

njcwotx · Guru Joined: 25 Feb 2005 Posts: 587 Location: Texas

Well, I got back to work, found an older windows vm with same type of file set and tested rsync from windows and the copy moved very fast from this box to linux. Didnt try the reverse on test though.

Showed this to boss and dev guy. They stopped their copy and we did the rsync from prod windows host only to find out it still crawled terribly copying that same set of files. Does not appear to be any faster, although looking at a network perf graph, it does appear to be smoother on transfer, but overall we calculate its going to take a very long time.

these are scanned images, millions of small kilobyte sized files and a few larger ones scattered in there. immense folder tree. Some things just take forever.
_________________
Drinking from the fountain of knowldege.
Sometimes sipping.
Sometimes gulping.
Always thirsting.

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

Doesn't tar have a maximum file size? You might have to segment the archive? Note I'm not talking about making a tar file on the source system and then sending it over.

If you have a 10gbps secure (non-Internet) network then I think ssh is unnecessary. It adds encryption which some cpus aren't so fast at.

But +1 on something to combine all the small files into one or more big files, preferably compressed, and then sending the big file.

Edit: Gnu tar has a maximum file size of 8gb on the compressed side.

frostschutz · Advocate Joined: 22 Feb 2005 Posts: 2977 Location: Germany

russK · l33t Joined: 27 Jun 2006 Posts: 665

Are these machines in close physical proximity? I just wonder if sneakernet with a USB 3.0 drive would be faster in the long run.

Regards

Akkara · Posted: Sat Oct 15, 2016 5:48 am Post subject:

Naib · Posted: Sat Oct 15, 2016 8:39 am Post subject:

1clue · Advocate Joined: 05 Feb 2006 Posts: 2569

@frostschutz, I recently tried to make an archive of large database backups and ran into a size limitation, going onto an xfs filesystem. Not sure exactly now what limit I hit, but there is one.

@russk, sneakernet is clearly better if that's an option.

@Akkara, The 10gbps connection depends on actual throughput. 10gbe is highly sensitive to tuning and hardware. Worst case some 10gbps ethernet connections get not much more than 3-4 gbps. That said, not all hardware is equal for disk access or compression.

For example, I have an i7 920 and an atom c2758. The atom has hardware support for encryption and compression. The atom is faster than the i7 with respect to encryption and compression, maintaining ~2.5gbps compressing or encrypting, and more than 2gbps for both combined. When doing both encryption and compression, the i7 struggles along at 1gbps. For things like compilation, the i7 beats the pants off of the atom, like you would expect.